25,000 Linux/UNIX Servers Infected with Malware

And this is a big reason is why you pay a real sysadmin to do your system administration.

In short, people were installing WordPress badly (friends don't let friends use PHP). They were allowing password authenticated ssh login over the internet. They were doing chmod 0777 ~apache/html_docs. They were doing other highly unsafe things.

If you can't see the problem with these things, then you need to talk to a professional sysadmin.

Hacked

Today I discovered that one of my customer's servers was hacked. Very few of my customers take computer security seriously. These people were worse then the average. Googling around for details on this root kit, I found two articles with a description of some techniques for attack analysis.

Of course, ideally we'd take off and nuke it from orbit. Only way to be sure. I MEAN take that the server out of service and reinstall from scratch.. This is not an option in this case.

One thing this root kit did was replace /usr/bin/ssh with a compromised one that logged all passwords to a file that was mailed out every night. I was depressed to find the file had the root password to my associate's home computer. So I changed the root password, and looked around for more stuff. Most of it was spread around and given important sounding names like klogd1, popauth and dnsquery.

I won't go into all the details because a- they are boring and b- probably only specific to this rootkit, but here is a few commands that will help find compromised files:

rpm -V -a
rpm -V -f /some/file

This assumes you have an RPM-based distro. Go read `man rpm` for details. TL;DR: any line with a 5 has changed since it was installed. If it's a config file, review it to make sure it doesn't do something bad. If it's an executable or shared library, reinstall that package.

ls -lint | sort -g | less

Picked this up from the articles above. The first column is the inode of the file. If you see any inodes that are far from the others in the list (they will be at the bottom of the list) they were installed after the initial setup. This isn't a 100% sign that you've been hacked; it happens when you do a `yum upgrade`. But those files do merit a closer look.

Phishing

I just got called by someone from "Global Security Services." He had a very pleasant South Asian accent. He started with "I assume you are running the Microsoft Windows operating system." To which I answered "Of course not!" He quickly hung up.

Now, I'm going to be highly suspicious of anyone calling me long distance without caller ID info. The "Global Security mutter mutter" made me highly suspicious that this was part of a new phishing scam that's been going on. The goal is to have you install some "Microsoft Security product" which is of course malware.

Notice to readers and family members: They are calling people in Canada. Never ever install anything a cold-caller says to. Ideally stop using Windows at all. It's security is atrocious and the knowledge required to keep it secured is beyond most normal users.