Do not try this at home

What you are about to see is bad and wrong. There are many better and easier ways to do this. I'm documenting it here so you can see how hoary it is.

Say you have a system set up with RAID1 on / and /boot, 2 active, 1 spare disks. This system can function even if reduced to a single active disk. Surely one can clone the system just by rebuilding the arrays with new disks?

Short answer is "Yes".

The longer answer is "No, don't do that. Use Clonezilla."

The reason you shouldn't do it this way is that Linux RAID (aka mdadm aka dm) uses UUIDs to identify arrays. It also has a field that contains the hostname the array was created on. What's more LVM has volume group names. These need to be unique if ever these the original and clone array are going to appear on the same system. Even if they you are sure they never will, some admin software you are going to try out some day.

But can't you change the UUIDs, VGs, hostname and so on? Yes you can. I just did it for a client. It was more pain then it was worth.

The following walk-through assumes my normal setup : first partition of each disk is part of a RAID1 3 active, no spares that goes on /boot (called md1 or md126). Second partition is partition of each disk is part of a RAID1 2 active, no spares that goes on / (called md0 or md127).

DO NOT TRY THIS AT HOME. I am a professional sysadmin with years of experience fucking up working systems. The following walk-through is provided without any warranty as to applicability or suitability to a any sane or useful or safe task. Back up your data. Verify your backup. RAID is not a backup. YMMV. HTH. HAND.

1. Make sure the arrays are fully in sync.
2. Do a clean shutdown
3. Make sure you can boot from the first and second drives of the array.
4. Remove the first active and the spare drive from the computer. Label them well and set aside
5. Disconnect the second drive. This will be the first drive on the new clone
6. Boot from a LiveDVD or USB or something. You will need a distro that has mdadm, uuidgen, lvm. I used the CentOS 6.5 LiveDVD.

7. telinit 1 # single user mode
   pstree # make sure nothing unwanted is running
   killall dhclient # kill everything unwanted.  You will need udevd

8. Plug the old-second-new-first drive in and wait for things to settle
9. cat /proc/mdstat Make sure your arrays are inactive. They will have (S) to mean they need to sync with something.
10. This is the hairy bit:

    # get /boot working
    mdadm --stop /dev/md126
    mdadm --assemble --update=uuid --uuid=$(uuidgen) /dev/md126 /dev/sda1
    mdadm --stop /dev/md126
    mdadm --assemble --update=name --name=$(hostname):1 /dev/md126 /dev/sda1
    mdadm --stop /dev/md126
    mdadm --assemble /dev/md126 /dev/sda1 --run
    tune2fs -U $(uuidgen) /dev/md126
    # get / working
    mdadm --stop /dev/md127
    mdadm --assemble --update=uuid --uuid=$(uuidgen) /dev/md127 /dev/sda2
    mdadm --stop /dev/md127
    mdadm --assemble --update=name --name=$(hostname):0 /dev/md127 /dev/sda2
    mdadm --stop /dev/md127
    mdadm --assemble /dev/md127 /dev/sda2 --run
    # activate LVM on /dev/md127
    vgscan
    # rename VG
    vgrename OLDVG NEWVG
    # mount /
    vgchange -a y NEWVG
    tune2fs -U $(uuidgen) /dev/mapper/NEWVG-root
    mount /dev/mapper/NEWVG-root /mnt
    # mount /boot
    mount /dev/md1 /mnt

11. Now comes the really annoying part: You have to update /etc/fstab (CentOS 6 has the UUID of /boot array), /boot/grub/grub.conf (CentOS 6 has the UUID of / array and the VG of /) and and possibly /boot/grub/initramfs-MUTTER.img to use the new UUIDs. The really fun part (for me) is that the LiveDVD doesn't have joe. So I had to write the UUID down on a piece of paper, then write it into grub.conf.

    You can find the UUID of an array with

    mdadm --detail /dev/md0

    If you want more flexibility, do

    mount --bind /proc /mnt/proc
    mount --bind /dev /mnt/dev
    mount --bind /sys /mnt/sys
    mount --bind /tmp /mnt/tmp
    chroot /mnt

    This will allow you to run mkinitrd if you need to. Note that this assumes your live DVD has a kernel that is compatible with your Linux distro.
12. Reboot to new system. Keep your fingers crossed.
13. Now you just insert your 2 other disks and run

    sfdisk -d /dev/sda | sfdisk /dev/sdb
    sfdisk -d /dev/sda | sfdisk /dev/sdc
    mdadm --add /dev/md126 /dev/sdb1
    mdadm --add /dev/md126 /dev/sdc1
    # wait until rebuild is finished
    cat <<GRUB | grub
    device (hd0) /dev/sdb
    root (hd0,0)
    setup (hd0)
    GRUB
    cat <<GRUB | grub
    device (hd0) /dev/sdc
    root (hd0,0)
    setup (hd0)
    GRUB
    mdadm --add /dev/md127 /dev/sdb2
    mdadm --add /dev/md127 /dev/sdc2
    

14. Ask yourself - was this really worth it? Wouldn't Clonezilla have been so much easier?

That didn't seem to hard, you might be saying. What I'm omitting is that when I booted to the new array, I got a lot of checksum errors and a failed fsck. I did fsck -y /dev/md127 a bunch of times until it came up clean.

Also - how do I get my arrays back to md1 and md0? The old method (--update=super-minor) no longer works.

End of an era

A hard drive was dying on Billy.

Thu Jun 12 19:15:31 EDT 2014
19:15:31 up 1316 days, 19:15,  2 users,  load average: 0.97, 0.97, 0.77

But the server is rented. So while I would have tried to do a hot swap, iWeb wisely wanted to do a shutdown.

Thu Jun 12 20:36:03 EDT 2014
20:36:03 up 8 min, 12 users,  load average: 2.79, 1.28, 0.56

Oh well.

Furthermore.

A corollary to my previous dictum that a method may make a decisions OR do something is that you want to cut a larger into smaller pieces. And each piece generally looks like the following:

sub doing_something {
    my( $self ) = @_;
    $self->prepare_something;
    if( $self->is_it_time_to_do_something ) {
        $self->before_something;
        $self->something;
        $self->after_something;
   }
   $self->unprepare_something;
}

In the above, something is just the name of the particular small piece of the larger task. The prepare_something/unprepare_something calls are there to avoid all possible side-effects in is_it_time_to_do_something. I would use before_something/after_something are there for things logging, timing, transactions and other "admin" actions that aren't related to something.

I feel like I've been infected by all the Java I did last spring.

Parsing HTTP::Request->content with CGI.pm

Back in the mists of time, when the Web was young and unconquered, Lincoln Stein wrote a module for Perl that would allow people to easily deal with parameters handed to a CGI program and to generate HTML. This module eventually grew to include not one but several kitchen sinks. It includes its own autoload mechanism, it's own file handle class and more. It Just Works when called FastCGI, Perlex, mod_perl and others.

While CGIs have all but disappeared, this module is still very useful for handling all the finicky edge cases for dealing with HTTP request content. But if you write your own web server environment, using CGI.pm to parse the HTTP content can get be hard. You basically have to fake it out.

This is how you get the params from a GET request.

# $req is a HTTP::Request object
local $ENV{REQUEST_METHOD} = 'GET';
local $CGI::PERLEX = $CGI::PERLEX = "CGI-PerlEx/Fake";
local $ENV{CONTENT_TYPE} = $req->header( 'content-type' );
local $ENV{'QUERY_STRING'} = $req->uri->query;
my $cgi = CGI->new();

# Now use $cgi as you wish

And here we parse the params from a POST request. Note that POST request can be big. Very big. If you aren't careful, they will fill up your memory. Always check Content-Length before reading in a POST request. In the following code, all the content was written to a file.

# $req is a HTTP::Request object
# $file is a filename that contains the unparsed request content
local $ENV{REQUEST_METHOD} = 'POST';
local $CGI::PERLEX = $CGI::PERLEX = "CGI-PerlEx/Fake";
local $ENV{CONTENT_TYPE} = $req->header( 'content-type' );
local $ENV{CONTENT_LENGTH} = $req->header( 'content-length' );
local $CGITempFile::TMPDIRECTORY = "/YOUR/TEMP/DIR/HERE";
# CGI->read_from_client reads from STDIN
my $keep = IO::File->new( "<&STDIN" ) or die "Unable to reopen STDIN: $!";
open STDIN, "<$file" or die "Reopening STDIN failed: $!";
my $cgi = CGI->new();
open STDIN, "<&".$keep->fileno or die "Unable to reopen $keep: $!";
undef $keep;
unlink $file

# Now use $cgi as you wish

The fun is that CGI will only read POST data from STDIN, so we have to redirect that to our file, saving and restoring the previous STDIN.

The above code also works when you are uploading a file with multipart/form-data which is how I got caught up in all this kerfuffle.

It's really to bad that one can't just do

my $cgi = CGI->new( $req );

Dave Brubeck Quartet

And now for something completely different : Dave Brubeck Quartet's Time Out album is currently on heavy play. I know I'm late to the party. In fact, I was born to late for this party. But I really really digging the combination of mellow, swing and exotica on this album.

25,000 Linux/UNIX Servers Infected with Malware

And this is a big reason is why you pay a real sysadmin to do your system administration.

In short, people were installing WordPress badly (friends don't let friends use PHP). They were allowing password authenticated ssh login over the internet. They were doing chmod 0777 ~apache/html_docs. They were doing other highly unsafe things.

If you can't see the problem with these things, then you need to talk to a professional sysadmin.

Remove an LVM volume group from the kernel

I FINALLY found it!

The problem: You are messing around with loopback files and volume groups. You've removed the loopback, but the VG stays in the kernel's internal list. vgremove is of course NOT how you get rid of it.

The sane way of doing this is:

vgchange -a n VolGroup00
kpartx -d /dev/nbd0
qemu-nbd -d /dev/nbd0
vgscan

But maybe you killed qemu-nbd by mistake? Or maybe your partition is long gone. Then you need to use dmsetup to remove all traces:

# ls -l /dev/mapper/
total 0
lrwxrwxrwx. 1 root root      7 Mar 18 22:08 GEORGE2-root -> ../dm-0
lrwxrwxrwx. 1 root root      7 Mar 18 22:08 GEORGE2-swap -> ../dm-1
lrwxrwxrwx. 1 root root      7 Mar 19 15:44 Test02-LogVol00 -> ../dm-4
lrwxrwxrwx. 1 root root      7 Mar 19 15:44 Test02-LogVol01 -> ../dm-5
crw-rw----. 1 root root 10, 58 Mar 18 22:08 control
# dmsetup info /dev/dm-4
Name:              Test02-LogVol00
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-zo1BvaMXr6TS1knhxoyjhtItHEaIVH4wGJz2s2w8w24za3486Aa9ur0igGMxpLf7
# dmsetup remove /dev/dm-4
# dmsetup remove /dev/dm-5
# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "GEORGE2" using metadata type lvm2

And there was much rejoicing.