Authentication

Some notes on some of the different protocols used for authentication and authentication-adjacent things in email. Some of this is oral history, and some of it may be contradicted by later or more public historical revision.

SPF

Associates an email with a domain that takes responsibility for it.

Originally Sender Permitted From, now Sender Policy Framework. It allows a domain owner to announce which IP addresses mail that uses a particular return path to be sent from, and whether a recipient should accept mail sent from other IP addresses as legitimate.

It authenticates the domain in the return path, not any hostname that’s visible to the recipient.

Original intent

By allowing recipients to detect “probably forged” return paths, SPF allows recipient ISPs to avoid backscatter, by not sending asynchronous bounces in response to spam with forged return paths.

Secondarily, it provided a way to tie a message to domain name – the one hidden in the return path. Simple blocking of mail that violated SPF stopped a lot of spam, though not in a particularly sustainable way.

It allowed senders to take responsibility for mail they sent (authentication) and deny responsibility for mail they didn’t send (repudiation).

There’s a lot of history documented here.

Current usage

SPF on it’s own is primarily seen as a way for a domain to advertise which IP addresses it expects to send email from, and so to associate an email with a domain that takes responsibility for it. This allows the reputation of a mail stream to be monitored keyed on the associated domain.

It’s widely used for authentication, but use of SPF for repudiation (blocking mail that fails SPF with a -all result) has pretty much disappeared.

It’s also a major building block for DMARC. Having the return path and the domain in the visible From: header be “in the same domain” has additional benefits on it’s own as well as being part of the DMARC process.

Failure modes

SPF authentication is tied to the peer IP address of the SMTP transaction. That means that most sorts of forwarding – vanity domains, mailing lists etc. – will cause SPF to fail. There were attempts to mitigate that, by having the forwarder rewrite the return path (sender rewriting scheme) but as recipient ISPs moved to ignoring SPF failures as far as delivery decisions were concerned they didn’t make that much impact (but see DMARC and ARC).

Benefits and risks

An SPF pass is considered a positive sign, at least if the associated domain has some history. A missing or failed SPF check isn’t widely considered a negative (unless the sender has opted-in to it being used that way, by publishing DMARC records).

Most of the benefits of SPF require that you use a domain you control in your return path rather than one you don’t own, such as your ESP’s bounce domain. Most ESPs should be able to support that, but it will require delegating control to the ESP via adding DNS records to your domain.

It’s generally cheap to deploy for most senders and deploying it has no real risks, but management of it can get more complex when multiple mail streams are in use.

SenderID

Original intent

To be a “better” SPF, and one that authenticated based on the hostname visible to the recipient in the From: header. It had a new style policy record that started with “spf2.0/pra” rather than “v=spf1” but would fall back to the old-style v=spf1 records.

Current usage

Not much. The working group that was working on it gave up due to a range of disagreements, many around intellectual property claims from Microsoft.

Microsoft still appear to check for traditional SPF records based on the domain in From: header, which is technically a SenderID check, but they say they don’t look for spf2.0/ style SenderID records just v=spf1 style SPF records.

Benefits and risks

Based on anecdotal evidence publishing an SPF record for the domain in the visible From: can improve deliverability at Microsoft properties.

There’s not really any risk to publishing that SPF record, whether accurate or not, and the only costs are the maintenance overhead and the (valuable) space it’s DNS record takes up in the root of a zone.

Publishing a spf2.0/ style SenderID record is probably pointless.

DKIM

DKIM allows the sender of an email to attach a hostname to a message in a way that can be cryptographically validated by a recipient.

It authenticates the domain (often called the “d=”) in the DomainKey-Signature header, not any hostname that’s visible to the recipient.

Original intent

DKIM was intended to allow a sender to take responsibility for an email via an attached domain name, allowing recipients to track reputation via that domain name rather than via, e.g., sending IP addresses.

Current usage

DKIM is used much as it was intended, to associate a domain with an email.

It’s also a major building block for DMARC. Having the d= domain and the domain in the visible From: header be “in the same domain” has additional benefits on it’s own as well as being part of the DMARC process.

Failure modes

DKIM relies on a cryptographic signature of the body of the email and a – sender-chosen – subset of the email headers. If any of those are changed at all then the DKIM signature will be broken.

There are obvious ways that the content can be modified – mailing lists adding annotations to the subject line or footers to the body, for instance. But there are also a lot of subtle ways it can be modified. If mail is sent using an “unusual” structure – overly long lines, unwrapped headers, unusual content transfer encoding – then an intermediate mail system may “fix up” the mail in a way that doesn’t change the semantics of the message and which wouldn’t be visible changes to the recipient, but which will break the DKIM signature.

Benefits and risks

DKIM is much more robust against being broken in transit than SPF, and gives similar reputation advantages. It also allows participation in some feedback loops.

It is more complex to deploy, requiring emails to be cryptographically signed as they’re sent, but that’s well supported on current MTAs.

To get most of the advantages of DKIM you need to sign it with a domain you control rather than with a domain you don’t own, such as that of your ESP. Most ESPs should be able to support that, but it will require adding DNS records to your domain to delegate control to your ESP.

DMARC

DMARC allows a domain owner to state that all mail they send with their domain in the visible From: header will by authenticated by them (via SPF or DKIM). It’s for repudiation, not authentication. It effectively changes the semantics of SPF and DKIM from having them being a positive signal to not having them being a (very) negative one.

It also allows a domain owner to request notifications about mail that appears to be sent from their domain but which isn’t correctly authenticated. This is critical to being able to check that you really are authenticating (most of) your email, and to do so before you ask recipients to discard potentially legitimate mail that’s not authenticated. This is the critical feature that distinguishes DMARC from previous attempts at email repudiation such as SSP, ADSP and SPF.

Original intent

Brand protection and anti-phishing.

Current usage

Mostly brand protection. Anti-phishing is still given lip service, and DMARC does mitigate phishing from the most naive phishers, but it’s not particularly effective against an adversary who’s learned to adapt in the years since DMARC began to be widely deployed.

It has also changed some of the semantics of SPF and DKIM authentication. DMARC introduced the idea of “DMARC-aligned” authentication, meaning that the return path used by SPF or the d= used by DKIM is “in the same domain as” the email address in the From: field. Many ISPs will provide preferential treatment to email that is authenticated in a DMARC-aligned way, even if the domain is not publishing DMARC records.

Benefits and risks

Using DMARC in “reporting only” mode, either “p=none” or “p=quarantine pct=0” is an extremely useful tool for mapping out your mail flows and finding sources of legitimate email that aren’t correctly authenticated. This is useful in itself, as well as being an essential step towards DMARC enforcement. It will potentially generate a lot of data, though, and you’ll need to budget for infrastructure and personnel time to handle and analyze those reports. There are very few deliverability or usability risks in this mode, though there are a few mailing lists which will modify their behaviour in a potentially user-surprising way for users of your domain.

To deploy DMARC in enforcing mode effectively requires deploying SPF and DMARC everywhere as a prerequisite (while DMARC only requires that either SPF or DKIM passes the two authentication approaches are fragile and will occasionally break, so you want to have them both in place to minimize the risk of DMARC failing). It also requires ongoing management and monitoring to ensure that authentication hasn’t stopped working or a new mailstream has been deployed without it.

DMARC in enforcing mode is likely to reduce your deliverability rather than increase it, as anything unexpected in mail flow – whether it be a mistake on the senders part, a mistake by the receiver ISP or some sort of forwarding – will cause mail to be lost. In theory an ISP may be using the existence of enforcing DMARC as a positive signal but that’s probably far outweighed by the positives of the DKIM and SPF you had to put in place to get there. (There’re a lot of “soft” effects that DMARC might conceivable have indirectly on recipient behaviour that could have an impact, but until someone independent does real research on that speculation about it is a bit ineffable.)

For domains that are entirely dedicated to sending bulk mail, and which have no humans sending mail with normal mail clients the overhead of deploying DMARC can be tiny, especially if use of DMARC is baked in from the beginning. For more complex domains, or for retrofitting DMARC to a domain that is already in use the effort required to deploy DMARC while being sure that ongoing operations aren’t going to be impacted can be significant, potentially requiring months and person-years of work.

Enforcing DMARC is a prerequisite for BIMI.

ARC

Naive or uncaring use of DMARC breaks mailing lists and forwarding. ARC fixes some cases of that.

Original intent

ARC is Authenticated Received Chain. It allows mail forwarders to communicate whether the mail they’re forwarding was authenticated before they forwarded it.

Current usage

It’s standardized, it’s being deployed by receiver ISPs and forwarders. It seems to do what it says on the box.

Benefits and risks

It’s something that most people don’t have to care about, other than knowing that by making forwarding of authenticated mail more robust it mitigates some of the risks of DMARC.

BIMI

BIMI lets a sender have their logo displayed next to their email in the inbox at some email providers.

Original intent

To allow email senders to loot their companies marketing budget to pay for the costs of deploying DMARC.

No, really. But it’s probably a good thing.

BIMI allows whitelisted senders to display an image next to their email if their mail passes DMARC and they’re vetted in some manner. There’s not really any technical reason for BIMI to require DMARC (as opposed to DKIM and SPF), but it’s a good carrot that mailbox providers can use to encourage senders to deploy DMARC.

Current usage

Gradually being rolled out at some providers.

Benefits and risks

Data on whether having the brands logo displayed next to an email helps with recipient trust or avoiding phishing has been mixed. There’s no real downside, though, beyond the risks of deploying DMARC and the costs of being vetted.

If you already have DMARC in place you should consider BIMI. If you’re considering deploying DMARC you should probably include BIMI as part of that proposal.

STARTTLS

STARTTLS is an extension to SMTP that allows mailservers to exchange mail over an encrypted channel rather than as plain text.

Current usage

Opportunistic STARTTLS is widely supported by receiver ISPs. It’s used for transport security, protecting traffic from passive interception during delivery.

Benefits and risks

Much as most of the web is quietly moving from unencrypted http to TLS protected https the same is true of email. Supporting it when sending email is typically just a configuration setting, and there’s no real drawback from turning it on beyond a slight increase in CPU usage (which is unlikely to be a limiting factor for delivery rates anyway).

Google have stated that they believe in TLS everywhere, to the extent that they provide search benefits to sites that offer it. It’s safe to assume that they’d like to see TLS on their inbound email too.

Null MX

Not exactly authentication, rather a way to say “this domain doesn’t accept email”.

Original intent

Ideally when you’re sending email you’d look up the MX records of the recipient in DNS, then send to one of those. If there were no MX records you wouldn’t send the email. But for backwards compatibility with the internet of the 1980s you also have to check for an A record if there are no MX records.

If a domain doesn’t want email, but does want a webserver, they’ll publish an A records for the webserver and so senders will try and deliver mail there.

Null MX is a formal way to publish MX records saying “don’t even bother trying to send mail”.

Current usage

Universally supported, and many smarthosts will special-case it so as to not even try to send the mail and immediately suppress recipients at those domains without any retry attempts.

Prehistory

Repudiated Mail From, Mail Transmitter, RMX, DMP

These were all early attempts, starting in the late 80s but crystalizing in the early 90s, to identify legitimate return paths in email. They evolved into SPF.

DomainKeys, Identified Internet Mail

DomainKeys was developed by Yahoo, and Identified Internet Mail by Cisco to verify the source of email. Rather than having two competing standards the authors merged them to give DKIM.

DomainKeys had an outbound signing policy that added repudiation of the From: field, much as DMARC does to DKIM/SPF.

SSP, ASP and ADSP

DKIM removed DomainKeys outbound signing policy, avoiding tying all the operational problems with it from the DKIM standardization process by deferring it to a separate standard.

That standard was called variously SSP (Sender Signing Practices), ASP (Author Signing Practices) and ADSP (Author Domain Signing Practices). They were standardized in 2009, but abandoned due to being unused. The experience from that experiment informed DMARC’s development.

Related Posts

Office365 checking DMARC on the inbound

According to a recent blog post, Office365 is starting to evaluate incoming messages for DMARC. I talked a little bit about DMARC in April when Yahoo started publishing a p=reject message.

Read More

What to expect in 2016

WttWColorEye_forBlogI don’t always do predictions posts, even though they’re  popular. Most years I skip them because I don’t see major changes in the email space. And, I’m not the type to just write a prediction post just to post a prediction.
This year, though, I do see changes for everyone in the email space. Most of them center on finally having to deal with the technical debt that’s been accumulating over the past few years. I see ISPs and ESPs spending a lot of development effort to cope with the ongoing evolution authentication requirements.
When people started seriously looking at how to authenticate email, the first goal was getting organizations to implement the protocols. This was a practical concession; in order for a new protocol to be used it needs to be widely implemented. Phase one of authenticating email was simply about publishing protocols and getting organizations to use them.
During phase one, the organization that authenticated a mail hasn’t been important. In fact, the SPF spec almost guarantees that the ESP domain is the authenticated domain. In DKIM, the spec says any domain could sign as long as they could publish a public key in that domain’s domainkeys record.
ESPs took full advantage of this and lowered their own development overhead by taking most of the authentication responsibility on themselves. Their domains were in the 5321.from and they published the SPF records. Domains they control were in the d= and they generated and published the DKIM keys. Mail was authenticated without ESP customers having to do much.
We’ve hit the end of phase one. Most of the major players in the email space are authenticating outbound email. Many of the major players are checking authentication on the inbound. Phase one was a success.
We’re now entering phase two, and that changes thing. In phase two, SPF and DKIM are used as the foundation for user visible authentication. Neither SPF nor DKIM were designed to be user visible protocols. To understand what they’re authenticating you have to understand SMTP and email. Even now there are days when I begin talking about one of them and have to take a step back and think hard about what is being authenticated. And I use these things every day!
DMARC is the first of these end user visible protocols built on SPF and DKIM. It uses the established and widespread authentication to validate the user visible from address. This authentication requires that the d= value or the 5321.from address belong belong to the same domain in the visible from address. While you can pick whether the alignment between the visible from and the authentication is “strict” or “relaxed” you have no choice about the alignment.
Prior to DMARC no one really paid much attention to the domain doing the authentication. Authentication was a yes or a no question. If the answer was yes, then receivers could use the authenticated domain to build a reputation. But they weren’t really checking much in the way of who was doing the authentication.
In the push to deploy authentication, ESPs assumed the responsibility for authentication deployed ESPs took the responsibility and did most of the work. For many or most customers, authentication was as simple as clicking a checkbox during deployment. Some ESPs do currently let customers authenticate the mail themselves, but there’s enough overhead in getting that deployed that they often charged extra to cover the costs.
DMARC is rapidly becoming an expectation or even a full on requirement for inbox delivery. In order to authenticate with DMARC, the authenticating domain must be in the same domain space as the visible from. If senders want to use their own domain in the visible from, DNS records have to be present in that domain space. Whether it’s a SPF TXT record or a domainkeys record the email sender customer needs to publish the correct information in DNS. Even now, if you try to authenticate with DKIM through google apps, they require you to publish DNS records.
ESPs aren’t in a situation where they can effectively manage authentication alignment for all their customers. Hosting companies are in even worse shape when it comes to letting customers authenticate email. Developers are facing the fact they need to go back and rework their authentication code. Businesses are facing the fact they need to change their processes so customers can authenticate with DMARC.
It’s not just the infrastructure providers that are facing challenges with authentication. Senders are going to discover they can no longer hand authentication off to their ESPs and not worry about it. They’re going to have to get DNS records published by their own staff.
Getting DNS updates through some big companies is sometimes more difficult than it should be. I had one client a few years ago where getting rDNS changed to something non-generic took over a month. From an IT standpoint, changing DNS should require approvals and proper channels. Marketers may find this new process challenging.
And, if organizations want to publish reject policies for their domains, then they will have to publish records for every outside provider they use. Some of those providers can’t support DMARC alignment right now.
In 2016 a lot of companies will discover their current infrastructure can’t cope with modern authentication requirements. A lot of effort, both in terms of product development and software development, will need to be spent to meet current needs. This means a lot of user visible features will be displaced while the technical debt is paid.
These changes will improve the security and safety of email for everyone. It won’t be very user visible, which will give the impression this was a slow year for email development. Don’t let that fool you, this will be a pivotal year in email.

Read More

Should you publish DMARC?

secure_email_blogI’ve been hearing a lot lately about DMARC. Being at M3AAWG has increased that. Last night we were at dinner and heard from the next table “And they’re not even publishing DMARC!!!!”
I know DMARC is the future. I know folks are going to have to start publishing DMARC records. I also know that the protocol is the future. I am also not sure that most companies are ready for DMARC.
So lets take a step back and talk about DMARC, what it is and why I’m still a little hesitant to jump on the PUBLISH DMARC NOW!! bandwagon.

Read More