SPF Fail: too many DNS lookups

I’ve had a couple folks come to me recently for help troubleshooting SPF failures. The error messages said the SPF record was invalid, but by all checks it was valid.
Eventually, we tracked the issue down to how many include files were in the SPF record.
The SPF specification specifically limits the number of lookups that can happen during a SPF check.

SPF implementations MUST limit the number of mechanisms and modifiers that do DNS lookups to at most 10 per SPF check, including any lookups caused by the use of the “include” mechanism or the “redirect” modifier. If this number is exceeded during a check, a PermError MUST be returned. The “include”, “a”, “mx”, “ptr”, and “exists” mechanisms as well as the “redirect” modifier do count against this limit. The “all”, “ip4”, and “ip6” mechanisms do not require DNS lookups and therefore do not count against this limit. Processing limits

Some senders are using include: records that then have include: records that then have yet other include: records. Looking up all these include records caused the SPF lookup limits to be reached, thus causing the SPF lookups to fail (all the records can’t be pulled). Note, a lot of SPF checks don’t actually comply with the SPF spec in this area. Because of the spotty compliance, some too long SPF records won’t fail everywhere.
How can you fix this?

  • Prune unnecessary :include files from the SPF record.
  • Use different domains for mail from different places so you don’t need one SPF record to rule them all.
  • Don’t publish SPF records, rely on DKIM for authentication. Even Hotmail is checking DKIM these days.

If you need to find out how many lookups your SPF record entails you can use the SPF checker at emailstuff.org. Enter in a domain in the box and hit go. Then click on the “DNS” tab to see the actual DNS lookups that are happening. Each red domain is a separate DNS lookup.

Secureserver.net has a lot of include: files and a lot of lookups. Because this is a checking tool, it doesn’t limit the number of queries (because that wouldn’t be useful). But in places that are implementing the spec correctly, and are refusing mail based on SPF failures, secureserver.net would experience mail failures.

Related Posts

Who can you trust?

I’ve been recently dealing with a client who is looking at implementing authentication on their domains. He’s done a lot of background research into the schemes and has a relatively firm grasp on the issue. At this point we’re working out what policies he wants to set and how to correctly implement those policies.
His questions were well informed for the most part. A few of them were completely out of left field, so I asked him for some of his references. One of those references was the EEC Email Authentication Whitepaper.
My client was doing the best he could to inform himself and relies on industry groups like the EEC to provide him with accurate information. In this case, their information was incomplete and incorrect.
We all have our perspectives and biases (yes, even me!) but there are objective facts that can be independently verified. For instance, the EEC Authentication whitepaper claimed that Yahoo requires DKIM signing for access to their whitelist program. This is incorrect, a sender does not have to sign with DKIM in order to apply for the Yahoo whitelist program. A bulk sender does have to sign with DKIM for a Y! FBL, but ISPs are given access to an IP based FBL by Yahoo. I am shocked that none of the experts that contributed to the document caught that error.
Independent verification is one reason I publish the Delivery Wiki. It’s a resource for everyone and a way to share my knowledge and thought processes. But other experts can “check my work” as it were and provide corrections if my information is outdated or faulty. All too often, senders end up blaming delivery problems on evil spirits, or using “dear” in the subject line or using too much pink in the design.
Delivery isn’t that esoteric or difficult if you have a clear understanding of the policy and technical decisions at a range of ESPs and ISPs, the history and reasoning behind those decisions, and enough experience to predict the implications when they collide.
Many senders do face delivery challenges and there is considerable demand for delivery experts to provide delivery facts. That niche has been filled by a mix of people, of all levels of experience, expertise and technical knowledge, leading to the difficult task of working out which of those “experts” are experts, and which of those “facts” are facts.

Read More

DMARC: Please Be Careful!

(Cross posted from Spam Resource.)
Every couple of days, somebody new pops up on the DMARC-Discuss mailing list to ask some question or share an observation. It’s great to see people interested and joining the conversation. Clearly, DMARC interest and adoption are growing. What’s really frustrating, though, is that for about a quarter of the new subscribers, their first mailing list message goes to the spam folder in my Gmail account. It has become sort of an intelligence test I apply to new subscribers — I’ve stopped digging those messages out of the spam folder. I’m figuring that if they can’t figure out how to implement a DMARC record, or they don’t understand that it’s not really compatible with mailing lists nor is it meant for hobbyist domains, then I think perhaps they’ve got some things they’ve got to figure out before they’re ready to join the discussion.
To that end, let me take a moment to jot down some recommendations for folks who are considering implementing DMARC.

Read More

The death of IP based reputation

Back in the dark ages of email delivery the only thing that really mattered to get your email into the inbox was having a good IP reputation. If your IP sent good mail most of the time, then that mail got into the inbox and all was well with the world. All that mattered was that good IP reputation. Even better for the people who wanted to game the system and get their spam into the inbox, there were many ways to get around IP reputation.
Every time the ISPs and spam filtering companies would work out a way to block spam using IP addresses, spammers would figure out a way around the problem. ISPs started blocking IPs so spammers moved to open relays. Filters started blocking open relays, so spammers moved to open proxies. Filters started blocking mail open proxies so spammers created botnets. Filters started blocking botnets, so spammers started stealing IP reputation by compromising ESP and ISP user accounts.  Filters were constantly playing catchup with the next new method of getting a good IP reputation, while still sending spam.
While spammers were adapting and subverting IP based filtering a number of other things were happening. Many smart people in the email space were looking at improving authentication technology. SPF was the beginning, but problems with SPF led to Domains Keys and DKIM. Now we’re even seeing protocols (DMARC) layered on top of DKIM. Additionally, the price of data storage and processing got cheaper and data mining software got better.
The improvement in processing power, data mining and data storage made it actually feasible for ISPs and filtering companies to analyze content at standard email delivery speeds. Since all IPv4 addresses are now allocated, most companies are planning for mail services to migrate to IPv6. There are too many IPv6 IPss to rely on IP reputation for delivery decisions.
What this means is that in the modern email filtering system, IPs are only a portion of the information filters look at when making delivery decisions. Now, filters look at the overall content of the email, including images and URLs. Many filters are even following URLs to confirm the landing pages aren’t hosting malicious software, or isn’t content that’s been blocked before. Some filters are looking at DNS entries like nameservers and seeing if those nameservers are associated with bad mail. That’s even before we get to the user feedback, in the form of “this is spam” or “this is not spam” clicks, which now seem to affect both content, domain and IP reputation.
I don’t expect IP reputation to become a complete non-issue. I think it’s still valuable data for ISPs and filters to evaluate as part of the delivery decision process. That being said, IP reputation is so much less a guiding factor in good email delivery than it was 3 or 4 years ago. Just having an IP with a great reputation is not sufficient for inbox delivery. You have to have a good IP reputation and good content and good URLs.
Anyone who wants good email delivery should consider their IP reputation, but only as one piece of the delivery strategy. Focusing on a great IP reputation will not guarantee good inbox delivery. Look at the whole program, not just a small part of it.

Read More