Showing posts with label sysadmin. Show all posts
Showing posts with label sysadmin. Show all posts

2025/02/03

grub2 vs xfs

So I just tried

# grub2-install --boot-directory=/boot2 /dev/sdb1
Installing for i386-pc platform.
grub2-install: error: hd0 appears to contain a xfs filesystem which isn't known to reserve space for DOS-style boot.  Installing GRUB there could result in FILESYSTEM DESTRUCTION if valuable data is overwritten by grub-setup (--skip-fs-probe disables this check, use at your own risk).

And then I did

# grub2-install --boot-directory=/boot2 /dev/sdb1 --skip-fs-probe
Installing for i386-pc platform.
grub2-install: warning: File system `xfs' doesn't support embedding.
grub2-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
grub2-install: error: will not proceed with blocklists.

But of course I'm an idiot; I don't want to install the grub loader on sdb1, I want to install it on sdb, where the BIOS can actually find it

# grub2-install --boot-directory=/boot2 /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.

2024/04/23

thunderbird vs self-signed certs

A default dovecot install on AlmaLinux 9 creates a self-signed SSL certifiate. Thunderbird is now very picky about SSL certs. It used to tell you a certificate wasn't valid and allow you to create an exception. Now it just spins and does nothing. You will see the following in your dovecot logs:

Apr 23 18:47:42 sHOST dovecot[12484]: imap-login: Disconnected: Connection closed: SSL_accept() failed: error:0A000412:SSL routines::sslv3 alert bad certificate: SSL alert number 42 (no auth attempts in 0 secs): user=<>, rip=CLIENTIP, lip=HOSTIP, TLS handshaking: SSL_accept() failed: error:0A000412:SSL routines::sslv3 alert bad certificate: SSL alert number 42, session=<BNl+V8sWFOgKAAAF>

I spent 4-5 hours running around in circles to try and find a solution

First step is to import the key, tell dovecot to listen on port 443 (https) by adding the following lines to the service imap-login stanza in /etc/dovecot/conf.d/10-master.conf:

#service imap-login {
  inet_listener https {
    port = 443
    ssl = yes
  }

Note that you could also set up lighttpd to serve up the cert.

Restart dovecot with:

systemctl restart dovecot

Test the above with:

openssl s_client -connect YOURHOST:443

Then, in Thuderbird, you go into Hamburger > Preferences > Privacy & security > (scroll way down) > Manage Certificates... In the Certificate Manager window, you select the Servers tab and click Add Exception... and enter https://YOURHOST:443. Then click on Get Certificate and Confirm Security Exception.

We now have an exception for YOURHOST:443, but we want YOURHOST:993 (if you are using SSL/TLS) or YOURHOST:143 (if you are using STARTTLS). To fix the port number, you need to close Thunderbird, then modify the Thunderbird profile directly. Under Linux, this is ~USER/.thunderbird/SOMETHING-NON-OBVIOUS. I had a half dozen directories. To find the one you want:

cd ~/.thunderbird
find . -name cert_override.txt | xargs ls -l --sort=time

The most recently modified file is the one you want to edit.

YOURHOST:443    OID.2.16.840.1.101.3.4.2.1      HEX-STRING-HERE U       BASE64-STRING-HERE

Change the :443 on that line to :993 (for SSL/TLS) or :143 (for STARTTLS).

You can confirm you have the correct line by comparing the HEX-STRING-HERE with your dovecot cert's SHA256 fingerprint:

openssl x509 -sha256 -in /etc/pki/dovecot/certs/dovecot.pem -noout -fingerprint

2022/02/25

mecab-devel, where are you?

To compile MySQL from srpm on AlmaLinux8, you need mecab-devel, which doesn't seem to exist. After some digging around, this is the solution I found :

sudo yum --enablerepo=powertools group install "Development Tools"
sudo yum install make gcc-c++ rpmbuild

mkdir -pv ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
cd ~/rpmbuild/SOURCES
wget 'https://drive.google.com/uc?export=download&id=0B4y35FiV1wh7cENtOXlicTFaRUE' -O mecab-0.996.tar.gz
cd ~/rpmbuild/SPECS
wget https://git.almalinux.org/rpms/mecab/raw/branch/c8-stream-8.0/SPECS/mecab.spec

rpmbuild -ba mecab.spec

cd ~/rpmbuild/RPMS/x86_64/
sudo yum install mecab*.rpm

This isn't perfect. Why would someone host their code on Google drive? But it seems this is what the author wanted.

2021/04/14

Sendmail smart relay with TLS and plain auth

Instructions on how I set up sendmail smart relay with TLS and plain authenetication on CentOS 6.

First, make sure you have enough installed :

yum -y install ca-certificates sendmail sendmail-cf

Create /etc/mail/authinfo:

AuthInfo:YOUR.HOST.COM    "U:YOUR-USER@YOUR.HOST.COM" "I:YOUR-USER" "P:YOUR-PASSWORD" "M:LOGIN PLAIN"

Replace YOUR.HOST.COM, YOUR-USER and YOUR-PASSWORD with the correct stuff. LOGIN PLAIN stays as-is if you are using plaintext logins. Make sure to chmod 0600 this file.

Add the following to /etc/mail/sendmail.mc, making sure you use m4's dumbass `quotation' style

define(`SMART_HOST', `YOUR.HOST.COM')dnl
define(`RELAY_MAILER',`esmtp')dnl
define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl
FEATURE(`authinfo')dnl
define(`confCACERT_PATH', `/etc/pki/tls/certs')dnl
define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.crt')dnl

Note that the above is TCP port 587, which you might need to change.

Finally you restart sendmail and test as you normally woudl.

chmod 0600 /etc/mail/authinfo
service sendmail restart
echo "Testing" | mail -s "Test 1" somebody@example.com
tail -F /var/log/maillog

2020/07/02

Automatic backups in Windows

The following powershell script copies all .docx files from $global:SRC to $global:DEST with a timestamp.

Save this file to something like "StartClone.ps1". Then launch it with "[right click] > Run with PowerShell." Minimize the resulting window.

### Configuration - CUSTOMIZE THESE
$global:SRC = "C:\Users\fil\test"
$global:DEST = "C:\Users\fil\backup"
$global:LOGFILE = "C:\Users\fil\log.txt"


### SET FOLDER TO WATCH + FILES TO WATCH + SUBFOLDERS YES/NO
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = $global:SRC
$watcher.Filter = "*.docx"
$watcher.IncludeSubdirectories = $true
$watcher.EnableRaisingEvents = $true  

$rv=Test-Path "$global:DEST"
if ($rv -eq $False) {
    New-Item -ItemType "directory" -Path "$global:DEST"
}

function global:logging ($text) {
    $logline = "$(Get-Date -uformat "%Y/%m/%d %H:%M:%S") - $text"
    Add-content $global:LOGFILE -value $logline
}

### DEFINE ACTIONS AFTER AN EVENT IS DETECTED
$action = { 
    $path = $Event.SourceEventArgs.FullPath
    $changeType = $Event.SourceEventArgs.ChangeType
    global:logging "$changeType $path"    
    $file = Split-Path $path -leaf
    $now = "$(Get-Date -uformat "%Y-%m-%d %H.%M.%S")"
    $dest = "$global:DEST\$now $file"
    # global:logging "file=$file now=$now dest=$dest"
    copy-item "$path" "$dest"
}    
### DECIDE WHICH EVENTS SHOULD BE WATCHED 
echo "Hello world $global:LOGFILE"
Register-ObjectEvent $watcher "Created" -Action $action
Register-ObjectEvent $watcher "Changed" -Action $action
# Register-ObjectEvent $watcher "Deleted" -Action $action
# Register-ObjectEvent $watcher "Renamed" -Action $action
global:logging Started
while ($true) {sleep 60}

2020/01/16

daemontools vs selinux

While CentOS ships with a policy module for daemontools, it expects you to install things in /admin and /supervised. I don't, I put things in /var/daemontools/admin and /var/daemontools/supervised. So I spent far to much time trying to make a policy module that would work with my setup. My initial attempt worked and gave me hope :

cp /usr/share/selinux/devel/include/contrib/daemontools.{if,te,cf} .
joe daemontools.cf # changed all the dirs to /var/daemontools
make -f /usr/share/selinux/devel/include/Makefile
semodule -i daemontoos.pp

However, this replaced the default daemontools module and might mess with other systems. I'm probably the last person to care about daemontools outside of qmail. Also, I wanted to learn some selinux.

My next thought was that I could create a daemontools_quaero policy module that gave the executables an fcontext from daemontools module. This didn't work and I don't know why.

After turning to IRC and much messing around and back and forth, grift led me to the following solution:

cat - > daemontools_quaero.cil <<'CIL'
(filecon "/var/daemontools/admin/daemontools-0.76/command/envdir" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/envuidgid" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/fghack" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/multilog" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/pgrphack" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/setlock" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/setuidgid" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/softlimit" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/svc" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/svok" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/svscan" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/svscanboot" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/admin/daemontools-0.76/command/supervise" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/prog-log" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/prog-user" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/sudo-user" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/.+/env" dir (system_u object_r svc_conf_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/.+/run" file (system_u object_r bin_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/.+/log/env" dir (system_u object_r svc_conf_t ((s0)(s0))))
(filecon "/var/daemontools/supervised/.+/log/run" file (system_u object_r bin_t ((s0)(s0))))
CIL
sudo semodule -i daemontools_quaero.cil
sudo restorecon -RvF /var/daemontools/admin/daemontools-0.76/command /var/daemontools/supervised/

And there was much rejoicing.

Along the way, I discovered semodule -E, matchpathcon and the policy language as well as sesearch, seinfo, ps auxZ, ls -lZ.

2019/10/30

daemontools vs systemd

Here's the bare minimum to get daemontools running under systemd

cat <<CNF > /usr/lib/systemd/system/daemontools.service
[Unit]
Description=DJB daemontools
After=network.target

[Service]
ExecStart=/var/daemontools/command/svscanboot
Restart=always

[Install]
WantedBy=multi-user.target
CNF
systemctl enable daemontools
systemctl start daemontools

Note that the default daemontools install would put the executable in /command/svscanboot.

2018/07/30

intel drivers on CentOS 6

Furthermore, to get X working nicely on a Shuttle DX30 with CentOS 6, you have to install a newer intel driver:

yum install libpciaccess-devel.x86_64 xorg-x11-server-devel.x86_64 libXfont-devel.x86_64 libXfont2-devel.x86_64
git clone git://anongit.freedesktop.org/xorg/driver/xf86-video-intel
cd xf86-video-intel
git checkout d39197bb10b7d88cb4c456e7a5e8d34c1dc6eeaf
autoreconf -i
./configure
make
sudo mv /usr/lib64/xorg/modules/drivers/intel_drv.so /usr/lib64/xorg/modules/drivers/intel_drv.so.ORIG
sudo rsync -a ./src/.libs/intel_drv.so /usr/lib64/xorg/modules/drivers/intel_drv.so

(Copied from https://www.centos.org/forums/viewtopic.php?t=64780)

2018/06/08

upstart after rc

I run daemontools from upstart. I ran into a problem that one of my daemons needed dbus to be working, but upstart was starting daemontools before rc had a chance to start it. The bigger problem is that CentOS 6 is only a partial conversion to upstart in that it still uses rc files for most things.

I used to have start on runlevel [345] in /etc/init/daemontools.conf but that means daemontools is started at the same time (or even before) as rc.

I tried start on started rc but that means daemontools runs after rc is launched, not after rc has finished.

The solution is a custom event :

cat <<'SH' >/etc/init.d/rc-done
#!/bin/sh
#
# rc-done Emits events when rc has finished or nearly
#
# chkconfig:   2345 99 01

[ -e /etc/sysconfig/rc-done ] && . /etc/sysconfig/rc-done

case "$1" in
    start)
 initctl emit rc-done
        ;;
    stop)
 initctl emit rc-start
        ;;
    *)
        echo $"Usage: $0 {start|stop}"
        exit 2
esac
exit $?
SH
chmod +x /etc/init.d/rc-done
chkconfig --add /etc/init.d/rc-done

Now we put start on rc-done into daemontools.conf

Of course the interesting part is initctl emit rc-done. We could just as easily have put that at end of rc.local.

2017/08/31

Upstart and Digi Etherlite 32

Here's how to get your Etherlite working with upstart. But really, this technique would apply to any serial connection.

First we need /etc/init/dgrp.conf.

# Digi EtherLite
# Starts a agetty for each serial port
# Normally run from the start-dgrp.conf task

stop on runlevel [S016]

env TERM=vt100
env BAUD=19200
respawn
instance $ID
exec /sbin/agetty -L tty$ID ${BAUD:-19200} ${TERM:-vt100}

usage 'dgrp ID=XX [BAUD=YYYYY] [TERM=ZZZZ]  - where XX is terminal id (A10) and YYYYY is baud (12900) and ZZZZ is the termcap entry (vt100)'

You can test this by starting an agetty on ttyA01 with:

initctl start dgrp ID=A01

Stop it with:

initctl stop dgrp ID=A01

Now create /etc/sysconfig/dgrp. Each line is the ID with an optional baud rate (default 19200) and TERM (default vt100) setting:

A01
A02 9600 vt100
A10 19200 wy60

Now we create the controlling task. This gets run after rc?.d is finished. It reads /etc/sysconfig/dgrp and runs dgrp.conf where needed.

# Digi Etherlite
# This task will read /etc/sysconfig/dgrp and will run dgrp.conf for each line in it
# It is run after all rc?.d the scripts are run

start on stopped rc RUNLEVEL=[2345]

env CONFIG=/etc/sysconfig/dgrp
task
script
    test -f $CONFIG || ( logger -t start-dgrp.conf "$CONFIG doesn't exist"; exit 0 )
    test -d /proc && test -d /proc/dgrp || ( logger -t start-dgrp.conf "drpd daemon is not running" ; exit 3 )

    temp=$(mktemp --tmpdir)
    grep -v '#' $CONFIG >$temp

    while read ID BAUD TERM ; do
        ID=$(basename $ID)
        initctl start dgrp ID=${ID/tty} BAUD=$BAUD TERM=$TERM
    done < $temp
end script

We use a temporary file so that we can ignore comments in the dgrp config file.

2017/05/01

daemontools, system V init and mysql

This is how you setup a babysitter for a service started with system V init scripts using DJB's deamontools. We can't just put service $service start into a run file, because sys V init scripts start up background daemons. We have to use the daemon's PID file to watch what's going on.

First, I create mysql-babysit. I'm using mysql as an example. For other services, adjust $service and $pidfile.

# mkdir /var/daemontools/supervised/mysql-babysit
# cd /var/daemontools/supervised/mysql-babysit
# cat <<'SH' > mysql-babysit
#!/bin/bash

service=mysql

datadir=/var/lib/mysql
pidfile=$datadir/$(hostname).pid


##################
sleepPID=
function sig_finish () {
    echo $(date) $service "$1"
    service $service stop
    [[ $sleepPID ]] && kill $sleepPID
}
trap 'sig_finish TERM' TERM
trap 'sig_finish KILL' KILL


##################
echo $(date) $service start

service $service start

if [[ -f $pidfile ]] ; then
    pid=$(< $pidfile)
    if [[ $pid ]] ; then
        while grep -q $service /proc/$pid/cmdline 2>/dev/null ; do
            sleep 60 & sleepPID=$!
            wait $sleepPID
        done
        echo $(date) $service exited
        exit 0
    fi
fi
echo $(date) $service failed to start
sleep 5
exit 3
SH

Next we create and activate the run script

cd /var/daemontools/supervised/mysql-babysit
# cat <<'SH' >run
#!/bin/bash

exec /var/daemontools/supervised/mysql-babysit/mysql-babysit
SH
# chmod +x run
# chkconfig mysql off
# service mysql stop
# cd ../../service
# ln -s ../supervised/mysql-babysit

We can control mysql with

svc -d /var/daemontools/supervised/mysql-babysit # shutdown mysql
svc -u /var/daemontools/supervised/mysql-babysit # startup mysql
killall mysql # restart mysql

2017/04/27

Creating glue records with bind 9

It's not pretty. Basically, you have to create a zone with the exact name of your name servers. Even if one of those name servers are probably controlled by your ISP. Even if you already have an A record for your local NS

In the following examples, ns1.example.com is your primary name server, sdns1.isp.com is the secondary name server your ISP is letting you use.

Add the following to /etc/named.conf:

zone "ns1.example.com" {
        type master;
        file "master/ns1.example.com.zone";
};

zone "sdns1.isp.com" {
        type master;
        file "master/sdns1.isp.com.zone";
};

This is master/ns1.example.com.zone:

$TTL 300
@               IN      SOA     ns1.example.com. root.example.com. (
                                2017042702                      ; yymmdd##
                                2h                              ; Refresh
                                1h                              ; Retry
                                2W                              ; Expire
                                1h                              ; Minimum
                        )
                IN NS   ns1.example.com.
                IN NS   sdns1.isp.com.

@               IN A 1.2.3.4  // change this to the real IP

This is master/sdns1.isp.com.zone:

$TTL 300
@               IN      SOA     ns1.example.com. root.awale.qc.ca. (
                                2017042702                      ; yymmdd##
                                2h                              ; Refresh
                                1h                              ; Retry
                                2W                              ; Expire
                                1h                              ; Minimum
                        )
                IN NS   ns1.isp.com.
                IN NS   ns2.isp.com.

@               IN A 4.3.2.1 // change this to the real IP of sdns1.isp.com

Get the real IP of sdns1.isp.com with

host sdns1.isp.com
sdns1.isp.com has address 66.51.199.62

You can find the NS records for sdns1.isp.com with

# dig NS isp.com
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.1 <<>> NS isp.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30596
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;isp.com.   IN NS

;; ANSWER SECTION:
isp.com.  7200 IN NS ns2.isp.com.
isp.com.  7200 IN NS ns1.isp.com.

;; ADDITIONAL SECTION:
ns2.isp.com.  172799 IN A 66.51.206.98
ns1.isp.com.  172799 IN A 66.51.202.50

;; Query time: 210 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Thu Apr 27 16:10:12 2017
;; MSG SIZE  rcvd: 93

2014/08/05

Do not try this at home

What you are about to see is bad and wrong. There are many better and easier ways to do this. I'm documenting it here so you can see how hoary it is.

Say you have a system set up with RAID1 on / and /boot, 2 active, 1 spare disks. This system can function even if reduced to a single active disk. Surely one can clone the system just by rebuilding the arrays with new disks?

Short answer is "Yes".

The longer answer is "No, don't do that. Use Clonezilla."

The reason you shouldn't do it this way is that Linux RAID (aka mdadm aka dm) uses UUIDs to identify arrays. It also has a field that contains the hostname the array was created on. What's more LVM has volume group names. These need to be unique if ever these the original and clone array are going to appear on the same system. Even if they you are sure they never will, some admin software you are going to try out some day.

But can't you change the UUIDs, VGs, hostname and so on? Yes you can. I just did it for a client. It was more pain then it was worth.

The following walk-through assumes my normal setup : first partition of each disk is part of a RAID1 3 active, no spares that goes on /boot (called md1 or md126). Second partition is partition of each disk is part of a RAID1 2 active, no spares that goes on / (called md0 or md127).

DO NOT TRY THIS AT HOME. I am a professional sysadmin with years of experience fucking up working systems. The following walk-through is provided without any warranty as to applicability or suitability to a any sane or useful or safe task. Back up your data. Verify your backup. RAID is not a backup. YMMV. HTH. HAND.

  1. Make sure the arrays are fully in sync.
  2. Do a clean shutdown
  3. Make sure you can boot from the first and second drives of the array.
  4. Remove the first active and the spare drive from the computer. Label them well and set aside
  5. Disconnect the second drive. This will be the first drive on the new clone
  6. Boot from a LiveDVD or USB or something. You will need a distro that has mdadm, uuidgen, lvm. I used the CentOS 6.5 LiveDVD.
  7. telinit 1 # single user mode
    pstree # make sure nothing unwanted is running
    killall dhclient # kill everything unwanted.  You will need udevd
  8. Plug the old-second-new-first drive in and wait for things to settle
  9. cat /proc/mdstat Make sure your arrays are inactive. They will have (S) to mean they need to sync with something.
  10. This is the hairy bit:
    # get /boot working
    mdadm --stop /dev/md126
    mdadm --assemble --update=uuid --uuid=$(uuidgen) /dev/md126 /dev/sda1
    mdadm --stop /dev/md126
    mdadm --assemble --update=name --name=$(hostname):1 /dev/md126 /dev/sda1
    mdadm --stop /dev/md126
    mdadm --assemble /dev/md126 /dev/sda1 --run
    tune2fs -U $(uuidgen) /dev/md126
    # get / working
    mdadm --stop /dev/md127
    mdadm --assemble --update=uuid --uuid=$(uuidgen) /dev/md127 /dev/sda2
    mdadm --stop /dev/md127
    mdadm --assemble --update=name --name=$(hostname):0 /dev/md127 /dev/sda2
    mdadm --stop /dev/md127
    mdadm --assemble /dev/md127 /dev/sda2 --run
    # activate LVM on /dev/md127
    vgscan
    # rename VG
    vgrename OLDVG NEWVG
    # mount /
    vgchange -a y NEWVG
    tune2fs -U $(uuidgen) /dev/mapper/NEWVG-root
    mount /dev/mapper/NEWVG-root /mnt
    # mount /boot
    mount /dev/md1 /mnt
  11. Now comes the really annoying part: You have to update /etc/fstab (CentOS 6 has the UUID of /boot array), /boot/grub/grub.conf (CentOS 6 has the UUID of / array and the VG of /) and and possibly /boot/grub/initramfs-MUTTER.img to use the new UUIDs. The really fun part (for me) is that the LiveDVD doesn't have joe. So I had to write the UUID down on a piece of paper, then write it into grub.conf.

    You can find the UUID of an array with

    mdadm --detail /dev/md0
    If you want more flexibility, do
    mount --bind /proc /mnt/proc
    mount --bind /dev /mnt/dev
    mount --bind /sys /mnt/sys
    mount --bind /tmp /mnt/tmp
    chroot /mnt
    This will allow you to run mkinitrd if you need to. Note that this assumes your live DVD has a kernel that is compatible with your Linux distro.
  12. Reboot to new system. Keep your fingers crossed.
  13. Now you just insert your 2 other disks and run
    sfdisk -d /dev/sda | sfdisk /dev/sdb
    sfdisk -d /dev/sda | sfdisk /dev/sdc
    mdadm --add /dev/md126 /dev/sdb1
    mdadm --add /dev/md126 /dev/sdc1
    # wait until rebuild is finished
    cat <<GRUB | grub
    device (hd0) /dev/sdb
    root (hd0,0)
    setup (hd0)
    GRUB
    cat <<GRUB | grub
    device (hd0) /dev/sdc
    root (hd0,0)
    setup (hd0)
    GRUB
    mdadm --add /dev/md127 /dev/sdb2
    mdadm --add /dev/md127 /dev/sdc2
    
  14. Ask yourself - was this really worth it? Wouldn't Clonezilla have been so much easier?

That didn't seem to hard, you might be saying. What I'm omitting is that when I booted to the new array, I got a lot of checksum errors and a failed fsck. I did fsck -y /dev/md127 a bunch of times until it came up clean.

Also - how do I get my arrays back to md1 and md0? The old method (--update=super-minor) no longer works.

2014/06/12

End of an era

A hard drive was dying on Billy.

Thu Jun 12 19:15:31 EDT 2014
19:15:31 up 1316 days, 19:15,  2 users,  load average: 0.97, 0.97, 0.77

But the server is rented. So while I would have tried to do a hot swap, iWeb wisely wanted to do a shutdown.

Thu Jun 12 20:36:03 EDT 2014
20:36:03 up 8 min, 12 users,  load average: 2.79, 1.28, 0.56

Oh well.

2014/03/20

25,000 Linux/UNIX Servers Infected with Malware

And this is a big reason is why you pay a real sysadmin to do your system administration.

In short, people were installing WordPress badly (friends don't let friends use PHP). They were allowing password authenticated ssh login over the internet. They were doing chmod 0777 ~apache/html_docs. They were doing other highly unsafe things.

If you can't see the problem with these things, then you need to talk to a professional sysadmin.

2014/03/19

Remove an LVM volume group from the kernel

I FINALLY found it!

The problem: You are messing around with loopback files and volume groups. You've removed the loopback, but the VG stays in the kernel's internal list. vgremove is of course NOT how you get rid of it.

The sane way of doing this is:

vgchange -a n VolGroup00
kpartx -d /dev/nbd0
qemu-nbd -d /dev/nbd0
vgscan

But maybe you killed qemu-nbd by mistake? Or maybe your partition is long gone. Then you need to use dmsetup to remove all traces:

# ls -l /dev/mapper/
total 0
lrwxrwxrwx. 1 root root      7 Mar 18 22:08 GEORGE2-root -> ../dm-0
lrwxrwxrwx. 1 root root      7 Mar 18 22:08 GEORGE2-swap -> ../dm-1
lrwxrwxrwx. 1 root root      7 Mar 19 15:44 Test02-LogVol00 -> ../dm-4
lrwxrwxrwx. 1 root root      7 Mar 19 15:44 Test02-LogVol01 -> ../dm-5
crw-rw----. 1 root root 10, 58 Mar 18 22:08 control
# dmsetup info /dev/dm-4
Name:              Test02-LogVol00
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-zo1BvaMXr6TS1knhxoyjhtItHEaIVH4wGJz2s2w8w24za3486Aa9ur0igGMxpLf7
# dmsetup remove /dev/dm-4
# dmsetup remove /dev/dm-5
# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "GEORGE2" using metadata type lvm2

And there was much rejoicing.

2014/03/18

I am INVINCIBLE!

Nothing is beyond me. When it comes to computers, I am all conquering.

That actually might be an exaggeration. But I just pulled off a stunt that really impressed me.

I'm moving all my systems from CentOS 5 to CentOS 6. (Why so soon? Shut up) In the process, I need to move my VMs from VMware Server 1 (Seriously? Shut up) to KVM (libvirt specificly). For the most part, I'm actually starting up whole new VMs and reconfiging them. But I still might want to look at my old data, fetch old files, and what not. This means being able to read VMware's vmdk files. This is "easy":

modprobe nbd max_part=8
qemu-nbd -r --connect=/dev/nbd0 /vmware/files/sda.vmdk
kpartx -a /dev/nbd0
vgscan
vgchange -a y VolGroup00
mount -o ro /dev/mapper/VolGroup00-LogVol00 /mnt/files

There are 3 complications to this:

First, you can't have multiple VGs with the same name active at once. Work around is to only mount one at a time. You can renamed VGs with vgrename but that's a job for another day.

Next off, I chose to have my vmdk split into multiple 2GB files. This makes copying them around so much more fun. But qemu only understands monolithic files, so you need vmware-vdiskmanager to convert them. Specificly

vmware-vdiskmanager -r /vmware/files/sda.vmdk -t 0 /vmware/files/single.vmdk

Lastly (and this is the main point of this post) CENTOS 6 DOESN'T SHIP WITH NBD! After WTFing about as hard as I could, I googled around for one. Someone must have needed nbd at some point, surely. The only solution I found was to recompile the kernel from scratch. Which is stupid. As a work around, I used the kernel-lt from elrepo. But the real solution would be a kmod. I thought doing a kmod would be hard, so I set aside a few hours. Turns out, it's really easy and I got it right on the first try.

tl;dr - rpm -ivh kmod-nbd-0.0-1.el6.x86_64.rpm

I based my kmod on kmod-jfs from elrepo.

  1. Install the kmod-jfs SRPM;
  2. Copy jfs-kmod.spec to nbd-kmod.spec;
  3. Copy kmodtool-jfs-el6.sh to kmodtool-nbd-el6.sh
  4. Edit nbd-kmod.spec. You have to change kmod_name and the %changelog section. You might also want to change kversion to your current kernel (uname -r). If not, you need to add --define "kversion $(uname -r)" when running rpmbuild;
  5. Create nbd-0.0.tar.bz2;
  6. Build, install and test the new module.
    rpmbuild -ba nbd-kmod.spec
    rpm -ivh ~/rpmbuild/RPMS/x86_64/kmod-nbd-0.0-1.el6.x86_64.rpm
    modprobe nbd
    ls -l /dev/nbd*
  7. FLAWLESS VICTORY!

The hard part (of course) is that I wasn't sure what to put in nbd-0.0.tar.bz2. The contents of jfs-0.0.tar.bz2 just look like the files from drivers/jfs in the kernel tree with Kconfig and Makefile added on. So I pull down the kernel SRPM, did a rpmbuild -bp on that (just commend out all the BuildRequires that give you grief. You aren't doing a full build.) Then I poked around for nbd in ~/rpmbuild/BUILD/vanilla-2.6.32-431.5.1.el6/. Turns out there's only nbd.c and nbd.h. So that goes in the pot. I copied over the Makefile from jfs, modifying it slightly because jfs is spread over multiple source files. Kconfig looked like kernel configuration vars. I just copied BLK_DEV_NBD out of vanilla-2.6.32-431.5.1.el6/drivers/block/Kconfig.

This entire process took roughly 1 hours. It worked on the first try. Of course, all the magic is in kmodtool-nbd-el6.sh. But I was expecting a lot of pain. Instead it worked on the first try. I was so surprised I did modprobe -r nbd ; ls -l /dev/nbd* just to make sure I wasn't getting a false positive.

2014/03/13

encfs

WARNING The encryption of encfs is severly broken. Do not rely on it to keep anything secret.

So one can layer encfs on top of google-drive-ocamlfuse.

Here's how I set it up.

yum --enablerepo=epel install rlog-devel boost-devel
wget http://encfs.googlecode.com/files/encfs-1.7.4.tgz
tar zxvf encfs-1.7.4.tgz
cd encfs-1.7.4
./configure --prefix=/opt/encfs-1.7.4   \
        --with-boost-serialization=boost_serialization-mt \
        --with-boost-filesystem=boost_filesystem-mt

make all && sudo make install
sudo sh -c "echo /opt/encfs-1.7.4/lib >/etc/ld.so.conf.d/encfs-1.7.4.conf"  
sudo ldconfig 
for n in /opt/encfs-1.7.4/bin/encfs* ; do
    sudo ln -s $n /usr/local/bin 
done 

encfs ~/googledrive/Backup/Encoded ~/encfs

And now "all" I have to do is rsync -av /remote/pictures/ ~/encfs/Pictures/ --progress. And wait. A lot, given I'm getting roughly 12.43kB/s though this setup.

Backups in the cloud.

Given that 1TB of backup from Google is now 10$ a month, I had to look into doing cloud backups again.

Google doesn't have a native Linux client. So one has to use google-drive-ocamlfuse. Installing this on CentOS 6 is surprisingly complex. And Google Drive's auth mechanism is based around a web interface.

But once I got it working, it Just Worked. Or rather, I could copy small files to ~/googledrive, see them via the web interface, delete them there and they are now missing.

Of course you wouldn't leave unencrypted backups on Google's servers. I futzed around with gpg a bit, but maybe layering encfs on top of google-drive-ocamlfuse would be a better idea.

My experimetation was cut short by supper, and by the fact that uploading a 400MB file was very slow :-) (sent 418536429 bytes received 31 bytes 219416.23 bytes/sec aka 1.6 Mbit/S)

For the record, here is how I installed google-drive-ocamlfuse

yum install m4 libcurl-devel fuse-devel sqlite-devel zlib-devel \
    libzip-devel openssl-devel
curl -kL https://raw.github.com/hcarty/ocamlbrew/master/ocamlbrew-install \
    | env OCAMLBREW_FLAGS="-r" bash
source /home/fil/ocamlbrew/ocaml-4.01.0/etc/ocamlbrew.bashrc
opam init
eval `opam config env --root=/home/fil/ocamlbrew/ocaml-4.01.0/.opam`
opam install google-drive-ocamlfuse
sudo usermod -a -G fuse fil
google-drive-ocamlfuse 
google-drive-ocamlfuse ~/googledrive

The second to last command will open a browser to get an OAuth token. This means you need htmlview and a valid DISPLAY. The token is only good for 30 days. This is something that needs to be better automated.

2014/03/12

Death To Proprietary Drivers

I was working on a CentOS 6 install for work and figured "hey, I should upgrade Mustang to the latest version." Normally this means

yum upgrade
shutdown -r now

Of course that didn't work; Mustang has an APU and uses a proprietary driver from AMD for X.org. I pretty much never use Mustang's console so I didn't notice this for 2 days, when my wife complained about not being able to watch Lost.

After much futzing, I find the error message: symbol lookup error: /usr/lib64/xorg/modules/drivers/fglrx_drv.so: undefined symbol: GlxInitVisuals2D. This means AMD's driver is doing something stupid. I of course can't compile it nor fix it. I tried to download the latest driver, but that refused to install. Curse swear, google google and then I found it.

rpm -ivh http://elrepo.org/linux/elrepo/el6/x86_64/RPMS/elrepo-release-6-6.el6.elrepo.noarch.rpm
rpm -e fglrx64_p_i_c-12.104-1 --nodeps
yum -y install fglrx-x11-drv-32bit fglrx-x11-drv kmod-fglrx
aticonfig --initial

First line installs ELRepo, which you might already have. Second line removes the previous drivers, which conflict with the new ones. The --nodeps is because Adobe really wants OpenGL installed. Third line is the important one, it installs the new drivers and does all the magic to get them working. Yes, the X.org driver needs to install a kernel module. Last line just makes sure that Xorg.conf is set up properly. I'd been playing around in it to try to get it to work.

So Death to Proprietary Drivers! And long live the guys at ELRepo!