A Third Time, Uncharmed

Spamwashers hijacked, a wake-up call for lazy sysadmins everywhere. The slow bruteforcers are back for another round.

A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.

Note: This piece is also available with trackers but nicer formatting here.

I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.

A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.

We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either

a maintenance regime that's disorganized enough that software with known and exploitable bugs is left in place for long enough to open the doors to undesirables, or

at least one user (whoever is manning root or any of the other user IDs we know they will be sniffing out later) with a guessable password and a system administration regime that lets weak passwords exist in the first place.

You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.

But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.

Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.

Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.

We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.

The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.

References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (original, December 2, 2008), A Small Update About The Slow Brutes (original, December 6, 2008), Into a new year, slowly pounding the gates (original, December 21, 2008), The slow brutes, a final roundup (original, January 22, 2009) and (The slow brute zombies are back (original, April 12, 2009).

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.

The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.

Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.

Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.

Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -



Oct 28 23:58:35 rosalita sshd[61664]: Invalid user anthony from 62.225.63.99

Oct 28 23:58:35 rosalita sshd[61664]: error: PAM: authentication error for illegal user anthony from 62.225.63.99

Oct 28 23:58:35 rosalita sshd[61664]: Failed keyboard-interactive/pam for invalid user anthony from 62.225.63.99 port 58683 ssh2

Oct 28 23:59:43 rosalita sshd[61668]: Invalid user anthony from 217.136.253.239

Oct 28 23:59:43 rosalita sshd[61668]: error: PAM: authentication error for illegal user anthony from 239.253-136-217.adsl-static.isp.belgacom.be

Oct 28 23:59:43 rosalita sshd[61668]: Failed keyboard-interactive/pam for invalid user anthony from 217.136.253.239 port 54728 ssh2

Oct 29 00:00:26 rosalita sshd[61695]: Invalid user barbara from 112.78.124.159

Oct 29 00:00:27 rosalita sshd[61695]: error: PAM: authentication error for illegal user barbara from 112.78.124.159

Oct 29 00:00:27 rosalita sshd[61695]: Failed keyboard-interactive/pam for invalid user barbara from 112.78.124.159 port 56990 ssh2

Oct 29 00:03:42 rosalita sshd[61722]: Invalid user brian from 122.224.128.197

Oct 29 00:03:42 rosalita sshd[61722]: error: PAM: authentication error for illegal user brian from 122.224.128.197

Oct 29 00:03:42 rosalita sshd[61722]: Failed keyboard-interactive/pam for invalid user brian from 122.224.128.197 port 47058 ssh2

Oct 29 00:04:21 rosalita sshd[61725]: Invalid user brian from 218.69.27.138

Oct 29 00:04:21 rosalita sshd[61725]: error: PAM: authentication error for illegal user brian from 218.69.27.138

Oct 29 00:04:21 rosalita sshd[61725]: Failed keyboard-interactive/pam for invalid user brian from 218.69.27.138 port 41622 ssh2

Oct 29 00:05:00 rosalita sshd[61738]: Invalid user carol from 58.60.106.121

Oct 29 00:05:01 rosalita sshd[61738]: error: PAM: authentication error for illegal user carol from 58.60.106.121

Oct 29 00:05:01 rosalita sshd[61738]: Failed keyboard-interactive/pam for invalid user carol from 58.60.106.121 port 55916 ssh2

Oct 29 00:05:51 rosalita sshd[61744]: Invalid user carol from 200.248.242.218

Oct 29 00:05:52 rosalita sshd[61744]: error: PAM: authentication error for illegal user carol from 200.248.242.218

Oct 29 00:05:52 rosalita sshd[61744]: Failed keyboard-interactive/pam for invalid user carol from 200.248.242.218 port 54547 ssh2

Oct 29 00:07:24 rosalita sshd[61748]: Invalid user charles from 222.128.36.60

Oct 29 00:07:24 rosalita sshd[61748]: error: PAM: authentication error for illegal user charles from 222.128.36.60

Oct 29 00:07:24 rosalita sshd[61748]: Failed keyboard-interactive/pam for invalid user charles from 222.128.36.60 port 42322 ssh2

Oct 29 00:08:09 rosalita sshd[61755]: Invalid user christopher from 72.148.242.188

Oct 29 00:08:10 rosalita sshd[61755]: error: PAM: authentication error for illegal user christopher from adsl-072-148-242-188.sip.asm.bellsouth.net

Oct 29 00:08:10 rosalita sshd[61755]: Failed keyboard-interactive/pam for invalid user christopher from 72.148.242.188 port 37157 ssh2

Oct 29 00:09:06 rosalita sshd[61758]: Invalid user christopher from 58.223.251.232

Oct 29 00:09:06 rosalita sshd[61758]: error: PAM: authentication error for illegal user christopher from 58.223.251.232

Oct 29 00:09:06 rosalita sshd[61758]: Failed keyboard-interactive/pam for invalid user christopher from 58.223.251.232 port 46350 ssh2

Oct 29 00:09:48 rosalita sshd[61762]: Invalid user daniel from 64.69.114.115

Oct 29 00:09:49 rosalita sshd[61762]: error: PAM: authentication error for illegal user daniel from mulligan.softspike.org

Oct 29 00:09:49 rosalita sshd[61762]: Failed keyboard-interactive/pam for invalid user daniel from 64.69.114.115 port 39960 ssh2

Oct 29 00:11:29 rosalita sshd[61781]: Invalid user david from 80.255.179.150

Oct 29 00:11:29 rosalita sshd[61781]: error: PAM: authentication error for illegal user david from ppp-vpdn-80.255.179.150.yarnet.ru

Oct 29 00:11:29 rosalita sshd[61781]: Failed keyboard-interactive/pam for invalid user david from 80.255.179.150 port 49013 ssh2

More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.