virsh vs old VMs

So I upgraded my VM server. Fresh new NVMe drives in RAID1 with AlmaLinux 8 on them. I then copied all my VMs over, updated their machines to pc or q35 as needed and launched them to test them. Most booted up fine. Three of them failed with the following error.

qemu-kvm: block/io.c:1438: bdrv_aligned_preadv: Assertion `(offset & (align - 1)) == 0' failed.

Fortunately they weren't important VMs so I could wait until today to fix the problem. The solution was to use qemu-img to convert the rewrite the disk images.

cd /kvm/VM/pool
mkdir bad
mv vda.img bad
qemu-img convert bad/vda.img vda.img -p -O qcow2

That should work for most people. Note that I assume your images are qcow2 (which they should be). If you had some raw images, you could change qcow2 to raw above. Or change types='raw' to type='qcow2' via virsh edit VM.

Here we verify the new images.

modprobe nbd max_part=8
qemu-nbd --connect=/dev/nbd0 /kvm/VM/pool/vda.img
# disconcertingly, Alma will autoactivate any VGs on the image.  Otherwise we'd do
partx -a /dev/nbd0
pvscan /dev/nbd0p2
vgchange -ay VGNAME
# now things should be available
fsck -f /dev/nbd0p1
fsck -f /dev/mapper/VGNAME-lvname
vgchange -an VGNAME
qemu-nbd --disconnect=/dev/nbd0

My image had /boot as partition 1 and / as a LV in in partition2.

If you're like me, you run libvirt on a headless server and look at VM consoles with virt-viewer. You also probably see the following warnings:

Gtk-Message: 14:08:40.348: Failed to load module "canberra-gtk-module"
Gtk-Message: 14:08:40.348: Failed to load module "pk-gtk-module"
Gtk-Message: 14:08:40.477: Failed to load module "canberra-gtk-module"
Gtk-Message: 14:08:40.477: Failed to load module "pk-gtk-module"

The solution is as follows

yum install PackageKit-gtk3-module libcanberra-gtk3

Lighttpd vs Let's Encrypt

If you are getting SSL_ERROR_NO_CYPHER_OVERLAP error with lighttpd and an SSL certificate issued by Let's Encrypt, make sure you are using the latest version of lighttpd, openssl and have your root certs up-to-date.

yum --enable-repo=epel update lighttpd openssl openssl-devel openssl-libs openssl-static ca-certificates

CentOS 6 vs CPAN and Let's Encrypt

Here is the magic to get CPAN CLI to work with https.

# cpan

cpan[1]> o conf urllist https://www.perl.com/CPAN
Please use 'o conf commit' to make the config permanent!

cpan[2]> o conf urllist                                 
    urllist           
        0 [https://www.perl.com/CPAN]
Type 'o conf' to view all configuration items

cpan[3]> o conf commit
commit: wrote '/usr/share/perl5/CPAN/Config.pm'

If it is giving you problems with SSL certificat verification, then you have to upgrade openssl, ca-certificate to the latest version. Perl also maintains it's own SSL certificates in Mozilla::CA, so you might need to do

SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem cpan Mozilla::CA

CentOS 6 vs the world

If, like me, you are a fool and still have CentOS 6 installs you have to maintain, you might run into the following problem:

https://vault.centos.org/6.10/os/x86_64/repodata/repomd.xml: [Errno 14] problem making ssl connection
Trying other mirror.
https://vault.centos.org/6.10/extras/x86_64/repodata/repomd.xml: [Errno 14] problem making ssl connection
Trying other mirror.
https://vault.centos.org/6.10/updates/x86_64/repodata/repomd.xml: [Errno 14] problem making ssl connection
Trying other mirror.

The solution is to update curl and yum by hand:

wget https://vault.centos.org/6.10/os/x86_64/Packages/python-urlgrabber-3.9.1-11.el6.noarch.rpm
wget https://vault.centos.org/6.10/updates/x86_64/Packages/yum-3.2.29-81.el6.centos.0.1.noarch.rpm
wget https://vault.centos.org/6.10/updates/x86_64/Packages/curl-7.19.7-54.el6_10.x86_64.rpm
wget https://vault.centos.org/6.10/updates/x86_64/Packages/libcurl-7.19.7-54.el6_10.x86_64.rpm
sudo rpm -Uvh libcurl-7.19.7-54.el6_10.x86_64.rpm curl-7.19.7-54.el6_10.x86_64.rpm yum-3.2.29-81.el6.centos.0.1.noarch.rpm python-urlgrabber-3.9.1-11.el6.noarch.rpm

Sendmail smart relay with TLS and plain auth

Instructions on how I set up sendmail smart relay with TLS and plain authenetication on CentOS 6.

First, make sure you have enough installed :

yum -y install ca-certificates sendmail sendmail-cf

Create /etc/mail/authinfo:

AuthInfo:YOUR.HOST.COM    "U:YOUR-USER@YOUR.HOST.COM" "I:YOUR-USER" "P:YOUR-PASSWORD" "M:LOGIN PLAIN"

Replace YOUR.HOST.COM, YOUR-USER and YOUR-PASSWORD with the correct stuff. LOGIN PLAIN stays as-is if you are using plaintext logins. Make sure to chmod 0600 this file.

Add the following to /etc/mail/sendmail.mc, making sure you use m4's dumbass `quotation' style

define(`SMART_HOST', `YOUR.HOST.COM')dnl
define(`RELAY_MAILER',`esmtp')dnl
define(`RELAY_MAILER_ARGS', `TCP $h 587')dnl
FEATURE(`authinfo')dnl
define(`confCACERT_PATH', `/etc/pki/tls/certs')dnl
define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.crt')dnl

Note that the above is TCP port 587, which you might need to change.

Finally you restart sendmail and test as you normally woudl.

chmod 0600 /etc/mail/authinfo
service sendmail restart
echo "Testing" | mail -s "Test 1" somebody@example.com
tail -F /var/log/maillog

sshd vs selinux

If you're like me and want to use autossh, you'll ideally want to use ~sshd/.ssh/authorized_keys. This doesn't work out of the box on CentOS 7 at least. SELinux prevents sshd from reading its own authorized_keys. The following fixes it.

semanage fcontext --add -t ssh_home_t '/var/empty/sshd/.ssh(/.*)?'
restorecon -vFR .ssh/

Example error message:

setroubleshoot[58151]: SELinux is preventing /usr/sbin/sshd from read access on the file authorized_keys. For complete SELinux messages run: sealert -l 5a9832e8-d7c0-4e5d-af15-a977db1232e9
SELinux is preventing /usr/sbin/sshd from read access on the file authorized_keys.#012#012*****  Plugin catchall_labels (83.8 confidence) suggests   *******************#012#012If you want to allow sshd to have read access on the authorized_keys file#012Then you need to change the label on authorized_keys#012Do#012# semanage fcontext -a -t FILE_TYPE 'authorized_keys'#012where FILE_TYPE is one of the following: [huge list goes here]