Remove an LVM volume group from the kernel

I FINALLY found it!

The problem: You are messing around with loopback files and volume groups. You've removed the loopback, but the VG stays in the kernel's internal list. vgremove is of course NOT how you get rid of it.

The sane way of doing this is:

vgchange -a n VolGroup00
kpartx -d /dev/nbd0
qemu-nbd -d /dev/nbd0
vgscan

But maybe you killed qemu-nbd by mistake? Or maybe your partition is long gone. Then you need to use dmsetup to remove all traces:

# ls -l /dev/mapper/
total 0
lrwxrwxrwx. 1 root root      7 Mar 18 22:08 GEORGE2-root -> ../dm-0
lrwxrwxrwx. 1 root root      7 Mar 18 22:08 GEORGE2-swap -> ../dm-1
lrwxrwxrwx. 1 root root      7 Mar 19 15:44 Test02-LogVol00 -> ../dm-4
lrwxrwxrwx. 1 root root      7 Mar 19 15:44 Test02-LogVol01 -> ../dm-5
crw-rw----. 1 root root 10, 58 Mar 18 22:08 control
# dmsetup info /dev/dm-4
Name:              Test02-LogVol00
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-zo1BvaMXr6TS1knhxoyjhtItHEaIVH4wGJz2s2w8w24za3486Aa9ur0igGMxpLf7
# dmsetup remove /dev/dm-4
# dmsetup remove /dev/dm-5
# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "GEORGE2" using metadata type lvm2

And there was much rejoicing.

I am INVINCIBLE!

Nothing is beyond me. When it comes to computers, I am all conquering.

That actually might be an exaggeration. But I just pulled off a stunt that really impressed me.

I'm moving all my systems from CentOS 5 to CentOS 6. (Why so soon? Shut up) In the process, I need to move my VMs from VMware Server 1 (Seriously? Shut up) to KVM (libvirt specificly). For the most part, I'm actually starting up whole new VMs and reconfiging them. But I still might want to look at my old data, fetch old files, and what not. This means being able to read VMware's vmdk files. This is "easy":

modprobe nbd max_part=8
qemu-nbd -r --connect=/dev/nbd0 /vmware/files/sda.vmdk
kpartx -a /dev/nbd0
vgscan
vgchange -a y VolGroup00
mount -o ro /dev/mapper/VolGroup00-LogVol00 /mnt/files

There are 3 complications to this:

First, you can't have multiple VGs with the same name active at once. Work around is to only mount one at a time. You can renamed VGs with vgrename but that's a job for another day.

Next off, I chose to have my vmdk split into multiple 2GB files. This makes copying them around so much more fun. But qemu only understands monolithic files, so you need vmware-vdiskmanager to convert them. Specificly

vmware-vdiskmanager -r /vmware/files/sda.vmdk -t 0 /vmware/files/single.vmdk

Lastly (and this is the main point of this post) CENTOS 6 DOESN'T SHIP WITH NBD! After WTFing about as hard as I could, I googled around for one. Someone must have needed nbd at some point, surely. The only solution I found was to recompile the kernel from scratch. Which is stupid. As a work around, I used the kernel-lt from elrepo. But the real solution would be a kmod. I thought doing a kmod would be hard, so I set aside a few hours. Turns out, it's really easy and I got it right on the first try.

tl;dr - rpm -ivh kmod-nbd-0.0-1.el6.x86_64.rpm

I based my kmod on kmod-jfs from elrepo.

1. Install the kmod-jfs SRPM;
2. Copy jfs-kmod.spec to nbd-kmod.spec;
3. Copy kmodtool-jfs-el6.sh to kmodtool-nbd-el6.sh
4. Edit nbd-kmod.spec. You have to change kmod_name and the %changelog section. You might also want to change kversion to your current kernel (uname -r). If not, you need to add --define "kversion $(uname -r)" when running rpmbuild;
5. Create nbd-0.0.tar.bz2;
6. Build, install and test the new module.

   rpmbuild -ba nbd-kmod.spec
   rpm -ivh ~/rpmbuild/RPMS/x86_64/kmod-nbd-0.0-1.el6.x86_64.rpm
   modprobe nbd
   ls -l /dev/nbd*

7. FLAWLESS VICTORY!

The hard part (of course) is that I wasn't sure what to put in nbd-0.0.tar.bz2. The contents of jfs-0.0.tar.bz2 just look like the files from drivers/jfs in the kernel tree with Kconfig and Makefile added on. So I pull down the kernel SRPM, did a rpmbuild -bp on that (just commend out all the BuildRequires that give you grief. You aren't doing a full build.) Then I poked around for nbd in ~/rpmbuild/BUILD/vanilla-2.6.32-431.5.1.el6/. Turns out there's only nbd.c and nbd.h. So that goes in the pot. I copied over the Makefile from jfs, modifying it slightly because jfs is spread over multiple source files. Kconfig looked like kernel configuration vars. I just copied BLK_DEV_NBD out of vanilla-2.6.32-431.5.1.el6/drivers/block/Kconfig.

This entire process took roughly 1 hours. It worked on the first try. Of course, all the magic is in kmodtool-nbd-el6.sh. But I was expecting a lot of pain. Instead it worked on the first try. I was so surprised I did modprobe -r nbd ; ls -l /dev/nbd* just to make sure I wasn't getting a false positive.

All the little houses falling over just right

To boot CentOS 5, all the following needs to be set up and running BEFORE you reboot your computer.

1. GRUB installed on the boot drive (aka sda, ata1, sata1);
2. a /boot that grub can find and read. In particular, it must be small and at the front of the boot drive;
3. a kernel+initrd that contains the drivers to find and mount the final root filesystem (aka /); this can involve swearing, repeated applications of mkinitrd. At worse, you can pull initrd apart with gzip+cpio, add the drivers, modify init and repack it. Note that this is not a shell script but a nash script. CentOS 6 has moved to dash;
4. a / that has a working init and all that entails (obviously);
5. an /etc/fstab that won't fail out because fsck -A can't find a drive or partition. If you've moved / to another partition or to LVM, then you need to update fstab. You can set the last 2 columns to 0 if you don't want fsck to check that mount.

How not to spend a friday night.

I cascade of stupidity caused me to drive to Montreal and back on a Friday night. In the dark. In heavy rain. If you know me, this isn't my idea of fun. While I am slightly to blame, most of the blame is elsewhere.

First, power outage at a client's in St-Constant at 13h00 (roughly). DAMN YOU HYDRO. Though really, black-outs are par for the course. Client has a UPS though. BUT the USB cable went missing 6 years ago, so no way for the computer to turn itself off cleanly. DAMN YOU APC. Why not just put a USB-B port on the back of your damn UPSes instead of having using a secret-sacred RJ45 with 10 pins that costs way to much. Oh, yeah, that would be why. And DAMN YOU JEAN-PHILLIPP, a UPS without apcupsd (or equiv) is less then useful.

So anyway, battery eventually drains, BAM! Hard shutdown. Power comes back at some point, but I suspect not very cleanly. The on-off cycling causes the BIOS to loose its settings BLACK EYES TO YOU ASUS. SERIOUSLY WHAT THE EF?! Also: DAMN YOU APC AGAIN! The UPS should be smart enough to wait for a few seconds of clean power before turning passing power though.

But, now that the BIOS has been reset to showing the SATA as IDE drives, GRUB can no longer load stage 2. Which might be damn stupid, or unavoidable. The bug report to me is "GRUB " on the screen, nothing further.

And this is where my blame comes in; I know that ASUS motherboards can reset the BIOS. But I'd just had a problem with /boot on a RAID1, so I'm thinking that was the problem. I eat supper, drive 2 hours, boot the computer with Knoppix, reinstall grub so it can find stage 2. Reboot, yay grub! But then initrd can't find md0, which has VolGroup00 on it! WHA! It's there! KNOPPIX can find it, why can't you?

Messing around for a while until the light goes on: BIOS RESET BECAUSE ASUS HATES LIFE! OK, set the SATA back to AHCI. Boot again. Still no go. FAH!

And then the other shoe drops. One of those things that if you've never pulled an initrd apart and poked at the init script inside one, you wouldn't notice: init was looking for md0, but KNOPPIX was calling it md127. And it turns out that md devices have a prefered minor device number. So when KNOPPIX was calling it md127, it was writing that to the array. Which means when init was trying to activate md0, it goes BUH CAN'T FIND IT.

BLACK EYES TO YOU, KNOPPIX FOR CHANGING THAT! Seriously, changing the preferred name of an array is really bad form.

So how to change it back. First you deactivate the arrays :

mdadm --stop /dev/md126

mdadm --stop /dev/md127

BUH! That last doesn't work; LVM is still holding a lock on the array. I strongly suspect that vgremove would be enough to drop the lock, but there's no way I'm going to test that on live data.

So reboot, don't activate mdadm-raid. Do the following

mdadm --assemble --update=name --name=0 /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2

mdadm --detail /dev/md0

mdadm --assemble --update=name --name=1 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1

mdadm --detail /dev/md1

The --detail lines should show Preferred Minor : 0 or 1 depending.

Note: Those are hairy commands. They could potentially kill your arrays if you get the partitioning wrong. DO NOT JUST CUT AND PASTE THEM IF YOU RUN INTO THE SAME PROBLEM AS ME! Read the docs, understand them, then adapt the commands to your setup.

PS : you pull initrd apart with

cd /boot

mkdir t ; cd t

gzip -dc ../initrd-$(uname -r).img | cpio -i

ls

Now go poke at init

I take it back

The war on noise continues apace. Next up, remove Jimmy from service, replacing it with George.

# fdisk -H 224 -S 56 /dev/sdd  

# fdisk -H 224 -S 56 /dev/sde

# sfdisk -l /dev/sdd



Disk /dev/sdd: 182401 cylinders, 255 heads, 63 sectors/track

Warning: The partition table looks like it was made

  for C/H/S=*/224/56 (instead of 182401/255/63).

For this listing I'll assume that geometry.

Units = cylinders of 6422528 bytes, blocks of 1024 bytes, counting from 0



   Device Boot Start     End   #cyls    #blocks   Id  System

/dev/sdd1          0+ 233598  233599- 1465132900   fd  Linux raid autodetect

/dev/sdd2          0       -       0          0    0  Empty

/dev/sdd3          0       -       0          0    0  Empty

/dev/sdd4          0       -       0          0    0  Empty

sdd and sde are a pair WD15EARSs. With 4k blocks. Normaly I use sfdisk -d to copy partitions, but that failed on 4k blocks.

mdadm -C -n 2 -l 1 /dev/md2 /dev/sdd1 /dev/sde1

pvcreate /dev/md2

pvs -o name,pe_start

vgcreate -s 32M T00 /dev/md2

lvcreate -l 99%VG --name LV00 T00

mkfs -t ext4 -E stride=32 -m 1 -O extents,uninit_bg,dir_index,filetype,has_journal /dev/T00/LV00 

tune4fs -c 0 -i 0 /dev/T00/LV00

I got the formating commands from I Do Linux. I read up on all those mkfs options. I'd never have guessed they were the "best" options to use.

The tune4fs is basically turning off the fsck that happens automatically every X days or Y reboots; fsck is SLOW, and the automatic fsck always happens when you least want it. And with journals, UPSes and so on, an unsafe shutdown isn't supposed to happen.

I left some space on the VG free so I could do snapshots.

But then I got to thinking. Blocking off one large LV means that the file server VM gets access to the entire disk and all that space can't be used by a different VM or the host for another purpose. With LVM, I can grow the LV if I need it. So why not give it 500G at a time?

e4fsck -f /dev/T00/LV00 

resize4fs -p /dev/T00/LV00 500G 

lvresize -L 500G -t -v /dev/T00/LV00 

resize4fs -p /dev/T00/LV00

This was as much a test of ext4 resizing as anything else. And it worked flawlessly. Btw, fsck on an empty FS is fast.

So what does it look like:

# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/G00-root  572G  537G  5.8G  99% /

/dev/md0              950M   31M  870M   4% /boot

tmpfs                 5.9G     0  5.9G   0% /dev/shm

/dev/mapper/T00-LV00  493G  493G     0 100% /T00

WHAT 100% FULL ALREADY?

Well, no. I wanted to make sure the 500G as resize4fs sees it and 500G as lvresize sees it were the same thing, so I filled the FS to the brim:

# dd if=/dev/zero of=/T00/HUGE

dd: writing to /T00/HUGE': No space left on device

1031717425+0 records in

1031717424+0 records out

528239321088 bytes (528 GB) copied, 6608.14 seconds, 79.9 MB/s

See that? Bare in mind that this dd was happening during the initial RAID build. I hypothesis that the mobo in Corey (ASUS M2N-MX SE) that I was doing the previous tests on SUCKS. And that the mobo in George (ASUS DSBV-DX) does not.

# time rm -f /T00/HUGE

real    0m31.020s

user    0m0.000s 

sys     0m20.302s

It's a server, not a telephone!

Dear the people at Red Hat,

I'm trying to set up a VMware server on top of CentOS 5.5. I want / to be LVM on top of RAID5, /boot as RAID also. I want as few packages installed as possible, only the bare minimum services running. This seems like simple but widespread scenario.

But why did it take me 3 hours?

Why did LVM on RAID5 require a bleeding graphical install? This is a server we're talking about.

Why did a huge pile of useless stuff like bluetooth, pcsc, xfs (?!), nfs, cups, wpa_supplicant, tux, apache, nscd, IRDA get installed? If I need these things, yum allows me to install them in seconds. Instead I spend an hours hunting down those and other useless bits.

Maybe I'm just getting old. But I pine for Redhat 9, where everything was small, simple, understandable. Now everything is 5 levels of indirections, lots of smoke and a dash of magic.

Grrrrrr!

-Philip