Stop with the incorrect SPF advice

Another day, another ESP telling a client to publish a SPF include for the wrong domain. It shouldn’t annoy me, really. It’s mostly harmless and it’s just an extra DNS look up for most companies. Heck, we followed Mailchimp’s advice and added their include to our bare root domain and it’s not really a huge deal for companies with only a couple SaaS providers. Still, it’s an incorrect recommendation and it does cause problems for some senders who are using multiple SaaS providers and Google.

Both Steve and I have written different posts about SPF over the years. In fact, the Authenticating with SPF: -all or ~all post is the most visited post on the blog. I’ve even written almost this same post, pointing out that a lot of ESPs have bad recommendations for SPF records. Steve’s written about the technical ins and outs of SPF records in DNS and how to stay under the 10 lookup limit. There’s also a time a client made a pretty huge mistake in their SPF record and a spammer took advantage and trashed their reputation.

There are also dozens of other good write-ups and discussions on how SPF works done by other deliverability experts, filtering companies, deliverability monitoring companies and the Open SPF folks themselves. Yet still so many places get it wrong.

One of the errors comes because a lot of folks, even a lot of email experts, don’t always know or remember that there are two separate yet equally important From: addresses in an email. One, called the envelope from or return path is defined by RFC 821 (and its successors 2821 and 5321). It’s used during the SMTP transaction and is where all the bounces go. The second is what most people think of when we talk about a From address. This address is defined by RFC 822 (and its successors 2822 and 5322) and is the what we see when we look at a message displayed in our email client.

Image of a stylised envelope with a return address of address@bounce.domain.example with an arrow pointing to it and a label of "5321 From" and an image of an email as displayed in the mailbox with a From address of Your Favorite Company <ceo@domain.example> with an arrow pointed to it and a label of 5322 From
An email envelope and an email showing the differences between the 5321 and 5322 From addresses

SPF authenticates the 5321 From address. This is an address no one ever sees unless they go out of their way to look at the headers. It’s a functional header in that it identifies the source of the mail and is where delayed bounces should go. In a bulk mail context every source of email should have its own hostname. 1 So my clients that send through two ESPs use two different subdomains for their mail (esp1.bounce.client.example and esp2.bounce.client.example). Each hostname has its own TXT record with the relevant SPF file. They don’t need to publish either ESP include in client.example because SPF doesn’t look up that domain.

This doesn’t stop many ESPs from recommending the wrong thing.2 Part of the blame for this we can lay at Microsoft’s feet. Back in the early 00’s Microsoft tried to push a standard called SenderID that would apply SPF lookups to the From address inside the email. This wasn’t successful and SenderID was deprecated not long after they started pushing it.3 However, not to be deterred, Microsoft started doing SPF-like lookups on the From address. Some ESPs noticed this and started recommending their clients publish SPF for both the 5321 and the 5322 domains. You can still see some traces of this with TXT records that start with spf2.0/pra.

The long and short of it is that SPF authenticates the domain in the envelope from, and that is not the address displayed in the email client. I’ve been trying to encourage ESPs and in fact anyone that offers mail through their platform to clean up their documentation. With the explosion of SaaS products, many of which handle email, too many senders end up with SPF records with more than 10 lookups.

1 – Last night in the pub we had a discussion about whether or not an ESP should ever use the 5322.from address in this string. I said ESPs should never do this. Steve said weeeelll… if they had good filtering on the message that would handle bounces and then forward any replies you could do it. And he’s technically right, but even he admits that’s going to be really, really fragile and asking for trouble.

2 – This entire post started out as a bit of a rant because a client’s ESP is refusing to turn on custom DKIM signing unless the customer also publishes a SPF include on their bare root domain. The ESP is wrong here. There’s no reason to require a SPF include on the bare root domain in order to use a custom DKIM d=. Even if we accept that sometimes publishing SPF on the 5322.from domain seems to improve Microsoft delivery, then the include should be in the bare root domain regardless of the d= in the DKIM signature.

3 – You will sometimes see TXT records that look like a SPF record but start with spf2.0/pra. Old DNS rarely gets cleaned out.

Related Posts

Are they using DKIM?

It’s easy to tell if a domain is using SPF – look up the TXT record for the domain and see if any of them begin with “v=spf1”. If one does, they’re using SPF. If none do, they’re not. (If more than one does? They’re publishing invalid SPF.)
AOL are publishing SPF. Geocities aren’t.
For DKIM it’s harder, as a DKIM key isn’t published at a well-known place in DNS. Instead, each signed email includes a “selector” and you look up a record by combining that selector with the fixed string “._domainkey.” and the domain.
If you have DKIM-signed mail from them then you can find the selector (s=) in the DKIM-Signature header and look up the key. For example, Amazon are using a selector of “taugkdi5ljtmsua4uibbmo5mda3r2q3v”, so I can look up TXT records for “taugkdi5ljtmsua4uibbmo5mda3r2q3v._domainkey.amazon.com“, see that there’s a TXT record returned and know there’s a DKIM key.
That’s a particularly obscure selector, probably one they’re using to track DKIM lookups to the user the mail was sent to, but even if a company is using a selector like “jun2016” you’re unlikely to be able to guess it.
But there’s a detail in the DNS spec that says that if a hostname exists, meaning it’s in DNS, then all the hostnames “above” it in the DNS tree also exist (even if there are no DNS records for them). So if anything,_domainkey.example.com exists in DNS, so does _domainkey.example.com. And, conversely, if _domainkey.example.com doesn’t exist, no subdomain of it exists either.
What does it mean for a hostname to exist in DNS? That’s defined by the two most common responses you get to a DNS query.
One is “NOERROR” – it means that the hostname you asked about exists, even if there are no resource records returned for the particular record type you asked about.
The other is “NXDOMAIN” – it means that the hostname you asked about doesn’t exist, for any record type.
So if you look up _domainkey.aol.com you’ll see a “NOERROR” response, and know that AOL have published DKIM public keys and so are probably using DKIM.
(This is where Steve tries to find a domain that isn’t publishing DKIM keys … Ah! Al’s blog!)
If you look up _domainkey.spamresource.com you’ll see an “NXDOMAIN” response, so you know Al isn’t publishing any DKIM public keys, so isn’t sending any DKIM signed mail using that domain.
This isn’t 100% reliable, unfortunately. Some nameservers will (wrongly) return an NXDOMAIN even if there are subdomains, so you might sometimes get an NXDOMAIN even for a domain that is publishing DKIM. shrug
Sometimes you’ll see an actual TXT record in response – e.g. Yahoo or EBay – that’s detritus left over from the days of DomainKeys, a DomainKeys policy record, and it means nothing today.

Read More

Authentication

Some notes on some of the different protocols used for authentication and authentication-adjacent things in email. Some of this is oral history, and some of it may be contradicted by later or more public historical revision.

Read More

Null sender address

A question came up on the email geeks slack channel about empty from addresses. I asked if they meant the 5321 or 5322 from address which prompted a question about if you could even have a null 5321 from.

Read More