Hacked

Today I discovered that one of my customer's servers was hacked. Very few of my customers take computer security seriously. These people were worse then the average. Googling around for details on this root kit, I found two articles with a description of some techniques for attack analysis.

Of course, ideally we'd take off and nuke it from orbit. Only way to be sure. I MEAN take that the server out of service and reinstall from scratch.. This is not an option in this case.

One thing this root kit did was replace /usr/bin/ssh with a compromised one that logged all passwords to a file that was mailed out every night. I was depressed to find the file had the root password to my associate's home computer. So I changed the root password, and looked around for more stuff. Most of it was spread around and given important sounding names like klogd1, popauth and dnsquery.

I won't go into all the details because a- they are boring and b- probably only specific to this rootkit, but here is a few commands that will help find compromised files:

rpm -V -a
rpm -V -f /some/file

This assumes you have an RPM-based distro. Go read `man rpm` for details. TL;DR: any line with a 5 has changed since it was installed. If it's a config file, review it to make sure it doesn't do something bad. If it's an executable or shared library, reinstall that package.

ls -lint | sort -g | less

Picked this up from the articles above. The first column is the inode of the file. If you see any inodes that are far from the others in the list (they will be at the bottom of the list) they were installed after the initial setup. This isn't a 100% sign that you've been hacked; it happens when you do a `yum upgrade`. But those files do merit a closer look.