upstart after rc

I run daemontools from upstart. I ran into a problem that one of my daemons needed dbus to be working, but upstart was starting daemontools before rc had a chance to start it. The bigger problem is that CentOS 6 is only a partial conversion to upstart in that it still uses rc files for most things.

I used to have start on runlevel [345] in /etc/init/daemontools.conf but that means daemontools is started at the same time (or even before) as rc.

I tried start on started rc but that means daemontools runs after rc is launched, not after rc has finished.

The solution is a custom event :

cat <<'SH' >/etc/init.d/rc-done
#!/bin/sh
#
# rc-done Emits events when rc has finished or nearly
#
# chkconfig:   2345 99 01

[ -e /etc/sysconfig/rc-done ] && . /etc/sysconfig/rc-done

case "$1" in
    start)
 initctl emit rc-done
        ;;
    stop)
 initctl emit rc-start
        ;;
    *)
        echo $"Usage: $0 {start|stop}"
        exit 2
esac
exit $?
SH
chmod +x /etc/init.d/rc-done
chkconfig --add /etc/init.d/rc-done

Now we put start on rc-done into daemontools.conf

Of course the interesting part is initctl emit rc-done. We could just as easily have put that at end of rc.local.

gmail G3 SSL certs

The error messsage:

fetchmail: Server certificate verification error: unable to get local issuer certificate
fetchmail: This means that the root signing certificate (issued for /C=US/O=Google Trust Services/CN=Google Internet Authority G3) is not in the trusted CA certificate locations, or that c_rehash needs to be run on the certificate directory. For details, please see the documentation of --sslcertpath and --sslcertfile in the manual page.
139677826279240:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1178:
fetchmail: SSL connection failed.

The solution:

cd /your/sslcertpath
wget https://pki.goog/gtsr1/GTSR1.crt
wget https://pki.goog/gtsr2/GTSR2.crt
wget https://pki.goog/gtsr3/GTSR3.crt
wget https://pki.goog/gtsr4/GTSR4.crt
wget https://pki.goog/gsr2/GSR2.crt
wget https://pki.goog/gsr4/GSR4.crt
for n in *.crt ; do openssl x509 -in $n -out ${n/crt/der} -outform DER; done
for n in *.der ; do openssl x509 -in $n -inform DER -out ${n/der/pem} -outform PEM; done
c_rehash .
killall fetchmail

Upstart and Digi Etherlite 32

Here's how to get your Etherlite working with upstart. But really, this technique would apply to any serial connection.

First we need /etc/init/dgrp.conf.

# Digi EtherLite
# Starts a agetty for each serial port
# Normally run from the start-dgrp.conf task

stop on runlevel [S016]

env TERM=vt100
env BAUD=19200
respawn
instance $ID
exec /sbin/agetty -L tty$ID ${BAUD:-19200} ${TERM:-vt100}

usage 'dgrp ID=XX [BAUD=YYYYY] [TERM=ZZZZ]  - where XX is terminal id (A10) and YYYYY is baud (12900) and ZZZZ is the termcap entry (vt100)'

You can test this by starting an agetty on ttyA01 with:

initctl start dgrp ID=A01

Stop it with:

initctl stop dgrp ID=A01

Now create /etc/sysconfig/dgrp. Each line is the ID with an optional baud rate (default 19200) and TERM (default vt100) setting:

A01
A02 9600 vt100
A10 19200 wy60

Now we create the controlling task. This gets run after rc?.d is finished. It reads /etc/sysconfig/dgrp and runs dgrp.conf where needed.

# Digi Etherlite
# This task will read /etc/sysconfig/dgrp and will run dgrp.conf for each line in it
# It is run after all rc?.d the scripts are run

start on stopped rc RUNLEVEL=[2345]

env CONFIG=/etc/sysconfig/dgrp
task
script
    test -f $CONFIG || ( logger -t start-dgrp.conf "$CONFIG doesn't exist"; exit 0 )
    test -d /proc && test -d /proc/dgrp || ( logger -t start-dgrp.conf "drpd daemon is not running" ; exit 3 )

    temp=$(mktemp --tmpdir)
    grep -v '#' $CONFIG >$temp

    while read ID BAUD TERM ; do
        ID=$(basename $ID)
        initctl start dgrp ID=${ID/tty} BAUD=$BAUD TERM=$TERM
    done < $temp
end script

We use a temporary file so that we can ignore comments in the dgrp config file.

Far to many spelling options in Thunderbird and Firefox

Mozilla projects offer to check the spelling in every language installed on your computer. Which is great, except I don't need South African English, nor will I ever use Swiss French. And wading through the 20 items in the menu to switch from Canadian English to Québecois French is annoying.

And there is no easy way to fix this.

The hard way is to go into /usr/share/myspell and remove all the .dic and .aff files you don't want.

$ sudo su -
Password for fil@scott: 
# cd /usr/share/myspell/
# mkdir NOT
# mv *.dic *.aff NOT
# ln -s NOT/fr_CA.aff
# ln -s NOT/fr_CA.dic
# ln -s NOT/en_CA.aff
# ln -s NOT/en_CA.dic

Restart Thunderbird and you will now have 2 spelling options.

Yes, these changes are system-wide. There is currently no other way to do it, apparently.

vmware-vdiskmanager and CentOS 6

This is how you install vmware-vdiskmanager on CentOS 6. I needed to do this so I could read my old vmware-server vmdk.

First go this old KB article and download 1023856-vdiskmanager-linux.7.0.1.zip. It's at the bottom, in the Attachements section. Now you do the following little dance:

unzip 1023856-vdiskmanager-linux.7.0.1.zip
cp 1023856-vmware-vdiskmanager-linux.7.0.1  /usr/local/sbin/vmware-vdiskmanager 
chmod +x  /usr/local/sbin/vmware-vdiskmanager
yum -y install zlib.i686 glibc.i686 openssl098e.i686
md -pv /usr/lib/vmware/lib
cd /usr/lib/vmware/lib/
ln -s /usr/lib/libcrypto.so.0.9.8e 
ln -s libcrypto.so.0.9.8e libcrypto.so.0.9.8 
ln -s libcrypto.so.0.9.8e libcrypto.so.0
ln -s libcrypto.so.0.9.8e libcrypto.so
ln -s /usr/lib/libssl.so.0.9.8e
ln -s libssl.so.0.9.8e libssl.so.0.9.8
ln -s libssl.so.0.9.8e libssl.so.0
ln -s libssl.so.0.9.8e libssl.so

That fucking around in /usr/lib/vmware/lib is because even though VMware claims this is a static binary, it in fact dynamically loads crypto libraries at run time from non-standard places.

You can now convert your split vmdk to a single file and mount it:

vmware-vdiskmanager -r sda.vmdk -t 0 sda-single.vmdk
modprobe nbd max_part=8
qemu-nbd -r --connect=/dev/nbd0 sda-single.vmdk
kpartx -a /dev/nbd0
vgscan
vgchange -a y YOURVG
mount -o ro /dev/mapper/YOURVG-YOURLV /mnt

Aren't you glad you created a unique VG for each of your VMs?

To unmount:

umount /mnt
vgchange -a n YOURVG
kpartx -d /dev/nbd0
qemu-nbd -d sda-single.vmdk

daemontools, system V init and mysql

This is how you setup a babysitter for a service started with system V init scripts using DJB's deamontools. We can't just put service $service start into a run file, because sys V init scripts start up background daemons. We have to use the daemon's PID file to watch what's going on.

First, I create mysql-babysit. I'm using mysql as an example. For other services, adjust $service and $pidfile.

# mkdir /var/daemontools/supervised/mysql-babysit
# cd /var/daemontools/supervised/mysql-babysit
# cat <<'SH' > mysql-babysit
#!/bin/bash

service=mysql

datadir=/var/lib/mysql
pidfile=$datadir/$(hostname).pid


##################
sleepPID=
function sig_finish () {
    echo $(date) $service "$1"
    service $service stop
    [[ $sleepPID ]] && kill $sleepPID
}
trap 'sig_finish TERM' TERM
trap 'sig_finish KILL' KILL


##################
echo $(date) $service start

service $service start

if [[ -f $pidfile ]] ; then
    pid=$(< $pidfile)
    if [[ $pid ]] ; then
        while grep -q $service /proc/$pid/cmdline 2>/dev/null ; do
            sleep 60 & sleepPID=$!
            wait $sleepPID
        done
        echo $(date) $service exited
        exit 0
    fi
fi
echo $(date) $service failed to start
sleep 5
exit 3
SH

Next we create and activate the run script

cd /var/daemontools/supervised/mysql-babysit
# cat <<'SH' >run
#!/bin/bash

exec /var/daemontools/supervised/mysql-babysit/mysql-babysit
SH
# chmod +x run
# chkconfig mysql off
# service mysql stop
# cd ../../service
# ln -s ../supervised/mysql-babysit

We can control mysql with

svc -d /var/daemontools/supervised/mysql-babysit # shutdown mysql
svc -u /var/daemontools/supervised/mysql-babysit # startup mysql
killall mysql # restart mysql

Creating glue records with bind 9

It's not pretty. Basically, you have to create a zone with the exact name of your name servers. Even if one of those name servers are probably controlled by your ISP. Even if you already have an A record for your local NS

In the following examples, ns1.example.com is your primary name server, sdns1.isp.com is the secondary name server your ISP is letting you use.

Add the following to /etc/named.conf:

zone "ns1.example.com" {
        type master;
        file "master/ns1.example.com.zone";
};

zone "sdns1.isp.com" {
        type master;
        file "master/sdns1.isp.com.zone";
};

This is master/ns1.example.com.zone:

$TTL 300
@               IN      SOA     ns1.example.com. root.example.com. (
                                2017042702                      ; yymmdd##
                                2h                              ; Refresh
                                1h                              ; Retry
                                2W                              ; Expire
                                1h                              ; Minimum
                        )
                IN NS   ns1.example.com.
                IN NS   sdns1.isp.com.

@               IN A 1.2.3.4  // change this to the real IP

This is master/sdns1.isp.com.zone:

$TTL 300
@               IN      SOA     ns1.example.com. root.awale.qc.ca. (
                                2017042702                      ; yymmdd##
                                2h                              ; Refresh
                                1h                              ; Retry
                                2W                              ; Expire
                                1h                              ; Minimum
                        )
                IN NS   ns1.isp.com.
                IN NS   ns2.isp.com.

@               IN A 4.3.2.1 // change this to the real IP of sdns1.isp.com

Get the real IP of sdns1.isp.com with

host sdns1.isp.com
sdns1.isp.com has address 66.51.199.62

You can find the NS records for sdns1.isp.com with

# dig NS isp.com
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.1 <<>> NS isp.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30596
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;isp.com.   IN NS

;; ANSWER SECTION:
isp.com.  7200 IN NS ns2.isp.com.
isp.com.  7200 IN NS ns1.isp.com.

;; ADDITIONAL SECTION:
ns2.isp.com.  172799 IN A 66.51.206.98
ns1.isp.com.  172799 IN A 66.51.202.50

;; Query time: 210 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Thu Apr 27 16:10:12 2017
;; MSG SIZE  rcvd: 93