The view from a blacklist operator

We run top-level DNS servers for several blacklists including the CBL, the blacklist of infected machines that the SpamHaus XBL is based on. We don’t run the CBL blacklist itself (so we aren’t the right people to contact about a CBL listing) we just run some of the DNS servers – but that means that we do get to see how many different ways people mess up their spam filter configurations.
This is what a valid CBL query looks like:

  • “14.23.177.10.cbl.abuseat.org”

It’s just the IP address being queried (10.177.23.14) with the numbers reversed, with “.cbl.abuseat.org” added on the end. Not rocket science.
Here’s a tiny sample of some of the invalid queries:

  • “70.46.6.10.abuseat.org”
  • “202.204.219.10cbl.abuseat.org”
  • “252.94.193.10.ns1-cbl.abuseat.org”
  • “255.190.244.10 cbl.abuseat.org”
  • “166.193.222.10#cbl.abuseat.org”
  • “214.6.224.10.*@cbl.abuseat.org”
  • “212.9.185.10.http://cbl.abuseat.org”
  • “76.207.80.10.bl.abuseat.org”
  • “185.124.73.10.cbb.abuseat.org”
  • “201.54.179.10.cbl-xbl.abuseat.org”
  • “54.191.254.10.opm.abuseat.org”
  • “181.4.133.10.sbl-xbl.abuseat.org”
  • “176.33.165.10.cbl.abuseat.orgcbl.abuseat.org”
  • “101.126.133.10.cbl.abuseat.org:Mail from %IP% refused by blackhole site cbl.abuseat.org”

Those are just 15 of about 1800 different misconfigurations I have on file, just for queries to the CBL. I’ve seen similar things at other domains I host, and I’ve heard of just the same sort of thing from other people who own domains that are similar in some way to a domain used by a blacklist. It’s not unusual.
What happens when someone misconfigures a blacklist lookup in this way? Because of the way DNS based blacklists work the response to any of these invalid queries will be “no, that IP address isn’t listed”. So all these people are attempting to use the CBL to filter out spam and haven’t noticed that it’s never actually stopped any email. And all the time they’re doing this, they’re hammering my DNS servers (and many other peoples) with millions of pointless queries every day.
What can the DNS server operators do about that? Because of the way DNS works, blocking the broken queries will actually increase the amount of traffic they have to deal with by several times. Contacting all the people making the queries and pointing out the problem would be a huge task, and even when I have tracked down contact information and notified people by email I’ve never had a response and the problem has never been fixed.
So the only remaining option is to make the misconfiguration more obvious to the user – by responding to the invalid queries with “yes, that IP address is listed” and hoping that causing them to reject all the mail sent to their users will encourage them to fix their configuration. I check my nameserver statistics every so often and add “poison” entries for the more obvious misconfigurations I find. I did that for a bunch of misconfigurations manually yesterday, which will probably cause a lot of domains to reject a bunch of email they didn’t want to this morning.
There are fairly simple ways to make sure you’re querying a real blacklist – pretty much all of the legitimate blacklists include the IP address “127.0.0.2” as a test entry. You can use that to check that a blacklist is live manually – if the blacklist domain is sbl.spamhaus.org then a dns lookup for “2.0.0.127.sbl.spamhaus.org” should return an answer (typically 127.0.0.2) while a dns lookup for “1.0.0.127.sbl.spamhaus.org” should return “not found” / “NXDOMAIN”. If either of those tests fails, the blacklist is broken in some way, and you shouldn’t use it.
The choice of 127.0.0.2 for the test entry wasn’t arbitrary: 127.0.0.2 is a “local” address that’s always available on machine, though it’s usually never used for anything. But you can use it – if you open a commandline on your mailserver you can run an SMTP transaction by hand (as I discussed yesterday) from 127.0.0.2 using “telnet -b 127.0.0.2 your.hostname 25” (on Linux-ish systems, anyway – some other telnets use “-s” instead of “-b”). That way you can see whether you’re really rejecting based on a blacklists, and what error you’re giving. (It would be nice if every blacklist also had another test entry in 127.* as well as 127.0.0.2, so you could check them individually, but they don’t. Hint to blacklist operators.).
It’s very easy for spam filter authors to check those test entries once a day for each of the blacklists they were configured to use, and to disable the ones that failed. If you’re a postmaster who uses blacklists as part of your spam filter (and you probably should) you should check with the people who provide the filter whether it makes those checks – and if it doesn’t, ask them to add them. That will protect you from misconfigurations, blacklists being shut down, blacklists being abandoned and bought up by domain squatters and all sorts of other things that can cause you to lose a lot of mail.

Related Posts

Spamtraps

There is a lot of mythology surrounding spamtraps, what they are, what they mean, how they’re used and how they get on lists.
Spamtraps are very simply unused addresses that receive spam. They come from a number of places, but the most common spamtraps can be classified in a few ways.

Read More

How to disable a domain

Sometimes you might want to make it clear that a domain isn’t valid for email.
Perhaps it’s a domain or subdomain that’s just used for infrastructure, perhaps it’s a brand-specific domain you’re only using for a website. Or perhaps you’re a target for phishing and you’ve acquired some lookalike domains, either pre-emptively or after enforcement action against a phisher, and you want to make clear that the domain isn’t legitimate for email.
There are several things to check before disabling email.
1. Are you receiving email at the domain? Is anyone else?
Check the MX records for the domain, using “host -t mx example.com” from a unix commandline, or using an online DNS tool such as xnnd.com.
If they’re pointing at a mailserver you control, check to see where that mail goes. Has anything been sent there recently?
If they’re pointing at a mailserver that isn’t yours, try and find out why.
If there are no MX records, but there is an A record for the domain then mail will be delivered there instead. Check whether that machine receives email for the domain and, if so, what it does with it.
Try sending mail to postmaster@ the domain, for instance postmaster@example.com. If you don’t get a bounce within a few minutes then that mail may be being delivered somewhere.
2. Are you sending email from the domain? Is anyone else?
You’re more likely to know whether you’re sending mail using the domain, but there’s a special case that many people forget. If there’s a server that has as it’s hostname the domain you’re trying to shut down then any system software running no that server – monitoring software, security alerts, output from cron and so on – is probably using that hostname to send mail. If so, fix that before you go any further.
3. Will you need mail sent to that domain for retrieving passwords?
If there are any services that might have been set up using an email address at the domain then you might need a working email address there to retrieve lost passwords. Having to set email back up for the domain in the future to recover a password is time consuming and annoying.
The domain registration for the domain itself is a common case, but if there’s any dns or web hosting being used for the domain, check the contact information being used there.
4. How will people contact you about the domain?
Even if you’re not using the domain for email it’s quite possible that someone may need to contact you about the domain, and odds are good they’ll want to use email. Make sure that the domain registration includes valid contact information that identifies you as the owner and allows people to contact you easily.
If you’re hosting web content using the domain, make sure there’s some way to contact you listed there. If you’re not, consider putting a minimal webpage there explaining the ownership, with a link to your main corporate website.
5. Disabling email
The easiest way to disable email for a domain is to add three DNS records for the domain. In bind format, they look like:

Read More

Which is better UTF-8 or ISO-?

Someone asked today on a mailing list whether they should be using UTF-8 or “ISO” encoding for sending email. What’s the best choice depends on some of the details of the situation, but here’s the answer I gave:
UTF-8 will work for pretty much anything, as it’s just an 8 bit encoding scheme for Unicode (which is supposed to be the one character encoding to rule them all). It’s well supported in most languages and development environments – Windows has been native UTF-16 under the covers since the mid 90s, for instance – and typical messages that use mainstream glyphs should render well from utf-8 in most western MUAs and browsers.
There are still a very few old or broken clients out there that will not handle UTF-8 well but (outside the asian language market, where there’s still some non-ASCII, non-Unicode legacy usage) they’re typically ones that don’t really handle any character set encoding well and the only thing safe to send to them is either plain ASCII or whichever ASCII superset their OS happens to support natively (which is probably an argument for sending Windows-1252 codepage, but not a terribly strong one).
The various extended ASCIIs (such as ISO-8859-*) will only work for messages that are written solely using characters from that character set. If you have even one character in a message that cannot be expressed in ISO-8859-1, then you can’t use ISO-8859-1 to send that message.
ISO-8859-1 (aka Latin1) is fairly sloppy in some respects – it has no apostrophe, nor single quotes, for instance – but it can handle an awful lot of languages, from Kurdish to Swahili. It can’t handle Dutch, Estonian, Finnish, Hungarian and Welsh particularly well, nor can it show the Euro symbol (ISO-8859-14 or -15 are needed for some characters there).
A common problem is that many people (and the software they write) think that Windows uses Latin1. It doesn’t, it uses Windows-1252. If you accept messages written on Windows, using the Windows-1252 code page, and throw them out on the wire as ISO-8859-1 what you end up with is not quite right. It mostly works, as the two codepages overlap quite a bit, but they have different glyphs in the 0x80-0x9f range. So if you use single or double quotes (“smart quotes”), or the Euro symbol, or ellipses, or bullet, or the trademark symbol in your message they’ll be garbled. This is so common that some mail clients and web browsers will actually treat a document that claims to be ISO-8859-1 as Windows-1252, but that’s a bug workaround and not something it’s really safe to rely on.
If you’re doing personalized messages, and you’re sending one of them to Győző and one of them to Eiður then you may have to use different character sets for the two messages. If you’re talking about Győző and personalizing it for Eiður then you might find things break horribly.
Someone probably has some concrete data on mail client character set support, broken down by region and language, but my understanding is that this is a reasonable approach:

Read More