A better backup

I do backups badly. Basically, rsync to a large partition somewhere. That's not really a backup. It protects against hardware failure, yes. But not against "oops! I deleted that file 3 weeks ago." What's more I'm sure I'm not doing it as well as it could be; by backing-up to a hard disk, why not backup the entire OS, and the hard disk bootable? Would be complicated if multiple machines backup to one backup server, but for my clients, I most often have one server which backs-up to one set of removable disks

What's more, I moved all my VMs from Jimmy to George yesterday. When I say "move VM" I should say "moved all the services to new CentOS 5 VMs." Which sort of shows up another problem: keeping track of what you've set up where and why. Jimmy had lighttpd running on it. Why? Oh... to see the RRD graphs of the temperature and humidity in the attic. I should document all this, now that I "know it" but ideally it should be automated.

And conformance tests; a bit like unit tests, you run some scripts to see if everything in the new install is working as expected. After all was done, I realised that I hadn't copied over my subversion repositories, nor set them up.

One central issue, I suppose, is config files. Ideally, you just copy in the backed-up config file, start the service, run the test script, verify success. I notice that rpm provides a --configfiles option. Combined with rpm's verify options, maybe one could detect what config files have changed and keep a backup set of them. Of course, things like /var/spool/hylafax/etc/config.ttyS0 would have to be added by hand. As would stuff installed by hand into /opt and/or /usr/local

And a modified config file implies that the package is being used, so the package would get flagged as important. And then, maybe once a week say, you'd get email "hey, you don't have a conformance script for package X." Or "You didn't write a changelog for the latest changes to file Y."

How not to spend a friday night.

I cascade of stupidity caused me to drive to Montreal and back on a Friday night. In the dark. In heavy rain. If you know me, this isn't my idea of fun. While I am slightly to blame, most of the blame is elsewhere.

First, power outage at a client's in St-Constant at 13h00 (roughly). DAMN YOU HYDRO. Though really, black-outs are par for the course. Client has a UPS though. BUT the USB cable went missing 6 years ago, so no way for the computer to turn itself off cleanly. DAMN YOU APC. Why not just put a USB-B port on the back of your damn UPSes instead of having using a secret-sacred RJ45 with 10 pins that costs way to much. Oh, yeah, that would be why. And DAMN YOU JEAN-PHILLIPP, a UPS without apcupsd (or equiv) is less then useful.

So anyway, battery eventually drains, BAM! Hard shutdown. Power comes back at some point, but I suspect not very cleanly. The on-off cycling causes the BIOS to loose its settings BLACK EYES TO YOU ASUS. SERIOUSLY WHAT THE EF?! Also: DAMN YOU APC AGAIN! The UPS should be smart enough to wait for a few seconds of clean power before turning passing power though.

But, now that the BIOS has been reset to showing the SATA as IDE drives, GRUB can no longer load stage 2. Which might be damn stupid, or unavoidable. The bug report to me is "GRUB " on the screen, nothing further.

And this is where my blame comes in; I know that ASUS motherboards can reset the BIOS. But I'd just had a problem with /boot on a RAID1, so I'm thinking that was the problem. I eat supper, drive 2 hours, boot the computer with Knoppix, reinstall grub so it can find stage 2. Reboot, yay grub! But then initrd can't find md0, which has VolGroup00 on it! WHA! It's there! KNOPPIX can find it, why can't you?

Messing around for a while until the light goes on: BIOS RESET BECAUSE ASUS HATES LIFE! OK, set the SATA back to AHCI. Boot again. Still no go. FAH!

And then the other shoe drops. One of those things that if you've never pulled an initrd apart and poked at the init script inside one, you wouldn't notice: init was looking for md0, but KNOPPIX was calling it md127. And it turns out that md devices have a prefered minor device number. So when KNOPPIX was calling it md127, it was writing that to the array. Which means when init was trying to activate md0, it goes BUH CAN'T FIND IT.

BLACK EYES TO YOU, KNOPPIX FOR CHANGING THAT! Seriously, changing the preferred name of an array is really bad form.

So how to change it back. First you deactivate the arrays :

mdadm --stop /dev/md126

mdadm --stop /dev/md127

BUH! That last doesn't work; LVM is still holding a lock on the array. I strongly suspect that vgremove would be enough to drop the lock, but there's no way I'm going to test that on live data.

So reboot, don't activate mdadm-raid. Do the following

mdadm --assemble --update=name --name=0 /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2

mdadm --detail /dev/md0

mdadm --assemble --update=name --name=1 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1

mdadm --detail /dev/md1

The --detail lines should show Preferred Minor : 0 or 1 depending.

Note: Those are hairy commands. They could potentially kill your arrays if you get the partitioning wrong. DO NOT JUST CUT AND PASTE THEM IF YOU RUN INTO THE SAME PROBLEM AS ME! Read the docs, understand them, then adapt the commands to your setup.

PS : you pull initrd apart with

cd /boot

mkdir t ; cd t

gzip -dc ../initrd-$(uname -r).img | cpio -i

ls

Now go poke at init

Reinventing the wheel. Badly

So, I had x11vnc all nice and tested with the TightVNC viewer on Linux and RealVNC on Windows but I get a bug report. User couldn't log on. -unixpw was displaying Username: user was entering username, but pressing Enter wouldn't move the the Password field.

Looking at /var/log/x11vnc.log I see a bunch of:

12/10/2010 15:17:39 unixpw_keystroke: bad keysym4: 0xff8d

Looking in /usr/include/X11/keysymdef.h, I see that 0xff8d is XK_KP_Enter, that is, the Enter key to the right of most number pads. And not XK_Return, the Enter key that's just to the right of the alphabet.

Looking at x11vnc's source code, I see that

1. x11vnc/unixpw.c only checks for XK_Return and XK_Linefeed. But what's more;
   
2. reimplements the huge bloody-effing-inputting-text-with-editing wheel;
   
3. x11vnc's -remap function doesn't happen in the code path that leads to unixpw_keystroke, so is bloody useless for this problem.
   

So the solution is to use TightVNC viewer on Windows. I already know that TightVNC is the better viewer for Linux. So now that's 2 out of 3.

Of course, the other solution would be to patch x11vnc. But I already have my time fully commited to reading Irregular Webcomic!

Third point being why was it working for my test setup but not in the field? Well, I had Windows Server 2003 as a VMware Server guest, via the VMware server console running on Linux. So something somewhere was remapping something somehow. "It is always possible to add another layer of indirection."

Watch that command

I was reminded to day of how useful watch is. I omitted it from my previous list of important and useful commands because I rarely use it. But today I had George open on the bench and was finding out what fan connectors on the motherboard corresponded to which speed sensor, as reported by lm-sensors.

watch -n 0 "sensors w83793-i2c-0-2f | grep fan"

Then, as I plugged and unplugged a 3-wire fan here and there, I could see on screen what was going on. This is especially important, cause you don't want to unplug the CPU fan for any length of time.

FYI, the sensor:fan port mapping for a DSBV-DX is as follows:

fan1  CPU_FAN1

fan2  CPU_FAN2

fan3  FRNT_FAN1

fan4  FRNT_FAN2

fan5  FRNT_FAN3

fan6  FRNT_FAN4

fan7  REAR_FAN1

fan8  REAR_FAN2

fan9  FBD_FAN1

fan10 N/C

fan11 N/C

fan12 N/C

I take it back

The war on noise continues apace. Next up, remove Jimmy from service, replacing it with George.

# fdisk -H 224 -S 56 /dev/sdd  

# fdisk -H 224 -S 56 /dev/sde

# sfdisk -l /dev/sdd



Disk /dev/sdd: 182401 cylinders, 255 heads, 63 sectors/track

Warning: The partition table looks like it was made

  for C/H/S=*/224/56 (instead of 182401/255/63).

For this listing I'll assume that geometry.

Units = cylinders of 6422528 bytes, blocks of 1024 bytes, counting from 0



   Device Boot Start     End   #cyls    #blocks   Id  System

/dev/sdd1          0+ 233598  233599- 1465132900   fd  Linux raid autodetect

/dev/sdd2          0       -       0          0    0  Empty

/dev/sdd3          0       -       0          0    0  Empty

/dev/sdd4          0       -       0          0    0  Empty

sdd and sde are a pair WD15EARSs. With 4k blocks. Normaly I use sfdisk -d to copy partitions, but that failed on 4k blocks.

mdadm -C -n 2 -l 1 /dev/md2 /dev/sdd1 /dev/sde1

pvcreate /dev/md2

pvs -o name,pe_start

vgcreate -s 32M T00 /dev/md2

lvcreate -l 99%VG --name LV00 T00

mkfs -t ext4 -E stride=32 -m 1 -O extents,uninit_bg,dir_index,filetype,has_journal /dev/T00/LV00 

tune4fs -c 0 -i 0 /dev/T00/LV00

I got the formating commands from I Do Linux. I read up on all those mkfs options. I'd never have guessed they were the "best" options to use.

The tune4fs is basically turning off the fsck that happens automatically every X days or Y reboots; fsck is SLOW, and the automatic fsck always happens when you least want it. And with journals, UPSes and so on, an unsafe shutdown isn't supposed to happen.

I left some space on the VG free so I could do snapshots.

But then I got to thinking. Blocking off one large LV means that the file server VM gets access to the entire disk and all that space can't be used by a different VM or the host for another purpose. With LVM, I can grow the LV if I need it. So why not give it 500G at a time?

e4fsck -f /dev/T00/LV00 

resize4fs -p /dev/T00/LV00 500G 

lvresize -L 500G -t -v /dev/T00/LV00 

resize4fs -p /dev/T00/LV00

This was as much a test of ext4 resizing as anything else. And it worked flawlessly. Btw, fsck on an empty FS is fast.

So what does it look like:

# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/G00-root  572G  537G  5.8G  99% /

/dev/md0              950M   31M  870M   4% /boot

tmpfs                 5.9G     0  5.9G   0% /dev/shm

/dev/mapper/T00-LV00  493G  493G     0 100% /T00

WHAT 100% FULL ALREADY?

Well, no. I wanted to make sure the 500G as resize4fs sees it and 500G as lvresize sees it were the same thing, so I filled the FS to the brim:

# dd if=/dev/zero of=/T00/HUGE

dd: writing to /T00/HUGE': No space left on device

1031717425+0 records in

1031717424+0 records out

528239321088 bytes (528 GB) copied, 6608.14 seconds, 79.9 MB/s

See that? Bare in mind that this dd was happening during the initial RAID build. I hypothesis that the mobo in Corey (ASUS M2N-MX SE) that I was doing the previous tests on SUCKS. And that the mobo in George (ASUS DSBV-DX) does not.

# time rm -f /T00/HUGE

real    0m31.020s

user    0m0.000s 

sys     0m20.302s

Neat bash tricks

One idiom I like is:

ssh $host "commands;" 2>&1 | while read line ; do

    # react to any error messages or messages from commands in $line

done

For instance, say you were running x11vnc on a remote host. x11vnc has the annoying habit of using a port other then the one you specify, if the one you want is already taken. Very annoying. So:

ssh $host "x11vnc ...." 2>&1 | while read line ; do

    if [[ $line =~ 'PORT=([[:digit:]]+)' ]] ; then

        port=${BASH_REMATCH[1]}

        # now set up some sort of port forwarding so that $port is a sane, known port

        ssh -L '*:9600:localhost'$port $host

    fi

done

This has some problems, in that the second ssh can survive x11vnc exiting. I thought "hey, how about $?" but that has other problems; say the second ssh exits before its time. The $? you saved could be reused. The kill you'd want to do would provoke hilarity. While complaining about this on IRC, a wise soul suggested that I open a lock file, and then any process with that file still open must be killed. I don't need locking, and didn't want to learn about flock in bash right away, so what I roughly did was:

child_kill () {

    if [[ ! $LOCKFILE ]] ; then

        return 0

    fi

    lsof -F '' $LOCKFILE | while read ppid ; do

            if [[ $ppid =~ '^p([[:digit:]]+)$' ]] ; then

                pid=${BASH_REMATCH[1]}

                if [[ $pid != $$ ]] ; then

                    kill -HUP $pid

                fi

            fi

        done

    rm -f $LOCKFILE

}



local LOCKFILE=$(mktemp -p /tmp)

trap "child_kill" EXIT

ssh ... | while read line ; do 

    ....

    ( exec 123>$LOCKFILE

      ssh -L ..... $host &

    )

done

child_kill

I wish there was a better way to deal with lsof's output, but this works so why complain?

What's more, I wish I could use SSH's ControlMaster to make the second connection that much faster. But the quick testing I did with 4.9p1 failed. Bugger

It's a server, not a telephone!

Dear the people at Red Hat,

I'm trying to set up a VMware server on top of CentOS 5.5. I want / to be LVM on top of RAID5, /boot as RAID also. I want as few packages installed as possible, only the bare minimum services running. This seems like simple but widespread scenario.

But why did it take me 3 hours?

Why did LVM on RAID5 require a bleeding graphical install? This is a server we're talking about.

Why did a huge pile of useless stuff like bluetooth, pcsc, xfs (?!), nfs, cups, wpa_supplicant, tux, apache, nscd, IRDA get installed? If I need these things, yum allows me to install them in seconds. Instead I spend an hours hunting down those and other useless bits.

Maybe I'm just getting old. But I pine for Redhat 9, where everything was small, simple, understandable. Now everything is 5 levels of indirections, lots of smoke and a dash of magic.

Grrrrrr!

-Philip