Rickrolled? Get Ready for the Hail Mary Cloud!

© 2009 Peter N. M. Hansteen

If you publish your user name and password, somebody who is not you will use it, sooner or later.

It's been a fun few weeks. Over in Microsoft land, it must have been a big issue that according to malware hunters Sophos, the newly released Windows 7 with no extras is roughly as vulnerable as its older siblings. No great surprises there, I suppose.

Note: This piece is also available with trackers but nicer formatting here.

For those of us with a more Unixish leaning, the more interesting piece of news involved Apple iPhones. These phones apparently run a version of MacOS that has enough Unix in it that with a bit of tinkering, it is possible to install a variety of Unix software, such as the ubiquitous secure shell daemon sshd. To some, there is a certain attraction in knowing that you have an SSH server in your pocket or handbag. Too bad, then that enough of those adventurous iPhone owners never read up on the instructions and chose to run their toy with the default password for the root account and were vulnerable to a wonderful prank perpetrated by a programmer down under.

The prank (described in the inimitable The Register style here) demonstrated just how bad an idea it is to publish your user name and password. If you followed the news around last weekend you would notice that a large segment of the Microsoft-attached instapunditry got their facts wrong as usual with the this proves that Apple (and by extension any Unix and of course Linux) is just as vulnerable as Microsoft mantra repeated over and over.

In fact, there are two historical incidents that point to Unix being no silver bullet: the 2002 Linux Slapper Worm and the original network-enabled worm, the 1988 Morris Worm. Those two prove mainly that yes, some bugs are exploitable, and the way forward is to fix bugs and make them harder to exploit in the first place. Now these two famous exploits is possibly to be joined by a third, the rickrolling prank.

I beg to differ. The rickroller is about bad passwords, no more, no less. I've spent considerable time ranting about passwords in earlier columns, and this incident only underscores what we've been repeating until your eardrums wear thin an my vocal cords swell from exhaustion:

Publishing your username and password is a really bad idea.
It's almost as bad as picking a guessable password.


Add to this that the fact, as we've noted here earlier, there is a whole cloud of hijacked machines out there beavering away at guessing passwords right now, and they have been at it for quite a while.

The most remarkable of these botnets is the one that tries to avoid detection by distributing the password guessing for any target across a large number of hosts, so each guesser never shows high enough rates of activity to trigger any of the traditional bruteforce detection mechanism. The attempts look something like this in your authentication log:

Nov 13 21:10:14 rosalita sshd[50401]: error: PAM: authentication error for illegal user mars from 125.40.69.208
Nov 13 21:10:14 rosalita sshd[50401]: Failed keyboard-interactive/pam for invalid user mars from 125.40.69.208 port 38052 ssh2
Nov 13 21:11:20 rosalita sshd[50419]: reverse mapping checking getaddrinfo for 115-186-131-90.nayatel.pk [115.186.131.90] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:11:20 rosalita sshd[50419]: Invalid user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: error: PAM: authentication error for illegal user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: Failed keyboard-interactive/pam for invalid user mars from 115.186.131.90 port 42235 ssh2
Nov 13 21:13:43 rosalita sshd[50428]: Invalid user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: error: PAM: authentication error for illegal user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: Failed keyboard-interactive/pam for invalid user mars from 58.247.222.163 port 35134 ssh2
Nov 13 21:15:59 rosalita sshd[50440]: Invalid user mars from 89.76.186.99
Nov 13 21:16:00 rosalita sshd[50440]: error: PAM: authentication error for illegal user mars from chello089076186099.chello.pl
Nov 13 21:16:00 rosalita sshd[50440]: Failed keyboard-interactive/pam for invalid user mars from 89.76.186.99 port 52156 ssh2
Nov 13 21:17:16 rosalita sshd[50448]: Invalid user mars2 from 88.134.166.31
Nov 13 21:17:16 rosalita sshd[50448]: error: PAM: authentication error for illegal user mars2 from 88-134-166-31-dynip.superkabel.de
Nov 13 21:17:16 rosalita sshd[50448]: Failed keyboard-interactive/pam for invalid user mars2 from 88.134.166.31 port 39254 ssh2
Nov 13 21:18:14 rosalita sshd[50452]: Invalid user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: error: PAM: authentication error for illegal user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: Failed keyboard-interactive/pam for invalid user room from 62.198.66.107 port 47557 ssh2
Nov 13 21:19:27 rosalita sshd[50458]: Invalid user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: error: PAM: authentication error for illegal user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: Failed keyboard-interactive/pam for invalid user room from 61.74.75.43 port 57794 ssh2
Nov 13 21:21:41 rosalita sshd[50472]: Invalid user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: error: PAM: authentication error for illegal user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: Failed keyboard-interactive/pam for invalid user room from 212.243.41.9 port 38396 ssh2
Nov 13 21:22:58 rosalita sshd[50491]: reverse mapping checking getaddrinfo for static-ip-cr1901468058.cable.net.co [190.146.80.58] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:22:58 rosalita sshd[50491]: Invalid user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: error: PAM: authentication error for illegal user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: Failed keyboard-interactive/pam for invalid user room from 190.146.80.58 port 4420 ssh2
Nov 13 21:24:01 rosalita sshd[50509]: Invalid user room from 62.23.130.173
Nov 13 21:24:01 rosalita sshd[50509]: error: PAM: authentication error for illegal user room from host.173.130.23.62.rev.coltfrance.com
Nov 13 21:24:01 rosalita sshd[50509]: Failed keyboard-interactive/pam for invalid user room from 62.23.130.173 port 3904 ssh2
Nov 13 21:25:21 rosalita sshd[50517]: reverse mapping checking getaddrinfo for hn.kd.ny.adsl [125.40.69.208] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:25:21 rosalita sshd[50517]: Invalid user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: error: PAM: authentication error for illegal user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: Failed keyboard-interactive/pam for invalid user room from 125.40.69.208 port 3294 ssh2

and so on.

I put it to you: What you see here is the cybercrime equivalent of the "Hail Mary Pass".

Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.

The fact that we see the Hail Mary Cloud keeping up the guessing is a strong indicator that there are a lot of guessable passwords and possibly badly maintained systems out there, and that even against the very long odds they are succeeding often enough in their attempts to gain a foothold somewhere that it is worth keeping up the efforts. For one thing, the cost of using other people"s equipment is likely to be quite low.

There are a lot of things about the Hail Mary Cloud and its overseers that we do not know. People who responded to the earlier articles with reports of similar activity also reported pretty consistently something like a sixty to seventy percent match in hosts making the attempts.

With 1767 hosts in the current sample it is likely that we have a cloud of at least several thousand, and most likely no single guessing host in the cloud ever gets around to contacting every host in the target list. The busier your SSH deamon is with normal traffic, the harder it will be to detect the footprint of Hail Mary activity, and likely a lot of this goes undetected.

The data, as I am sure you have been waiting for it, is available in these forms: Raw log data, with 3-4 lines per attempt (as illustrated above and requested by some correspondents), one line per attempt (shows the pattern a little more clearly), a list of the hosts participating in the Hail Mary Cloud sorted by number of attempts, and the user names attempted, sorted by number of attempts.

The pattern is fairly familiar by now, but this time the alphabetic cycles are shorter and at times the coordination seems to have broken down. My guess is that the apparent breakdowns are due to silly factors like the guessing machines running without time synchronization or other signs of incompetence.

And finally, some words of advice for those of you who want to avoid both rickrolling and getting cracked by other password guessing.

You should at least consider setting a password policy and enforcing it with something like John the ripper, which more than likely is available at the cost of a few keystrokes from your package system. And of course there is the fine art of sshd configuration. Some of the things you could do are, in no particular order:
whatever your users can bear to live with.

If you see traces of the Hail Mary Cloud's activity in your logs and you want to share and study, I would very much like to hear from you. I will most likely be updating the log data and extracts at intervals.

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.