Blair's Blog

Plain English:

Sorry for our absence over the past week, but internet ne’er-do-wells attacked our computers and I had to sort some stuff out before I could put the website back on-line.

The Geek-splanation:

Bad, bad script kiddies and spammers!! Bad! Go to your rooms – no supper for you!

The last 10 days have been exceedingly frustrating as the site has been down due to a rather nasty email based attack on our servers. The attack was actually against our mailserver, but it was difficult to sort out what was really going on, so the webserver has been down while I analyzed mail and system logs to try figure out what was happening.

The main symptom was that our mailserver was forwarding spam out through our ISP in sufficient volume that we ended up having all our outbound port 25 traffic blocked for a while last week. Funny thing was that a check for open relays ( http://www.abuse.net/relay.html) showed that we were fine – no open smtp relays at all! None the less, the mail logs clearly showed that we were sending out lots of email to addresses all over the net. But no from addresses were being logged and the mailserver configuration only trusted computers on the LAN. Reviews of the logs from as far back as I could go showed plenty of inbound spam – same as anyone with a mailserver has to cope with. However the outbound spam was pretty recent – over the past 3 weeks or so.

First thing I did was to clobber the mailserver box and rebuild it, but there was no change. Still had lots of spam going out. This led me to think that there might be a virus on one of the Windoze boxes – maybe an Outlook exploit of some kind. That led to a couple of days of cycling boxes on and off the LAN in various combinations to see when the spam stopped going out, but it never did. So, maybe it was a NAT/router hack? Ended up going out and buying a new router. Still no solution.

After that, since I had two routers and the ability to get two IP addresses from my ISP, I segregated my network and placed a freshly built and configured mailserver all by itself on the new router. That looked good for almost 2 days and I was just about ready to do a full rebuild on both Windows boxes when all of a sudden a wave of spam started to go out – so it couldn’t be a virus or any kind of interaction with any of my other computers. The problem had to be some kind of exploit from the internet. But what?

Several hours of detailed review of mail logs and system logs eventually showed a common pattern to the outbound spam.

• First there would be an inbound spam from some address (lets call it victim@spamtarget.com) with a “to” address of uucp which would be bounced by the mailserver.
• The system log would record an error writing to the uucp mail directory.
• Immediately after that the mail system would generate the outbound spam message to victim@spamtarget.com which would get relayed out.

I think that the exploit is one of a class of increasingly common non-smtp relay exploits (some of these exploits are described here: http://dsbl.org/relay-methods). However, I just didn’t know what the solution was.

In the meantime, my friend Sean was aware of the problem I was having and was getting tired of dealing with the load of incoming spam on his own mailserver. Sean discovered that his domain registrar (which I use as well) allows a free email account with each domain registration (they also act as an ISP providing web and email hosting services to their clients). After some pretty clever thinking about the problem, Sean realized that it would be possible to use the ISP mail services of the registrar to act as an intermediary to funnel all his email through to his server. This takes advantage of the registrar’s mail filtering capability to eliminate spam. Legitimate email is forwarded automatically to a dedicated subdomain on the local mailserver. In conjunction with this, a firewall rule on the router denies all port 25 traffic that isn’t originating from the domain registrar’s address space.

Problem solved! Well, maybe not solved exactly, but at least it is neatly avoided for both the inbound and outbound spam problems, and we are back on the web with the blog and everything. It’s nice to have a hobby, ain’t it?

</geekspeak>

1 Comment »