Folding email headers

Why are my DKIM signatures failing at Microsoft?

There are several reasons why your DKIM signatures might be failing at Microsoft, but perhaps not at other major providers. Many of them are triggered by sending mail that’s “badly structured” in some way. This is one of them.

Long Lines

After we’ve composed our email and hit send it has to be packaged up and sent over the wires to the recipient. That low level protocol to actually transmit the mail from A to B is specified in RFC 5321, but at a high level we can mostly ignore the requirements in that RFC. Mostly.

One thing we can’t ignore is that RFC 5321 has a limit on the length of each line in the payload of a message. (The payload is what we call the content of the email as it’s sent over the wire - headers, body, mime encoded attachments, everything).

The maximum total length of a text line including the <CRLF> is 1000 octets (not counting the leading dot duplicated for transparency). This number may be increased by the use of SMTP Service Extensions.

The mention of “SMTP Service Extensions” might make you think that this is just a historical limitation and of course we have service extensions to remove the limitation on the modern Internet. Sadly, no. Service extensions that allow more flexibility in payload structure either have to be supported by every single mailserver involved in delivering a message, or intermediate mailservers need to rewrite a message they’re relaying to support the requirements of the server they’re sending the message on to. “Rewrite” doesn’t sound too terrible, but even when it’s done well it’ll break our DKIM signatures and then DMARC will lead to our mail ending up in the spam folder, at best, or rejected with an asynchronous bounce at worst.

So every line of the email payload has to be no longer than 998 bytes.

For the body of the email and any attachments there’s an easy way to meet that requirement. MIME provides us a couple of different ways to package up the content we want to send as ascii characters we can include in the mail payload, and have it transparently unpackaged by the recipients mail client. That’s called “content-transfer-encoding” and our two usual options are “quoted-printable” and “base64”. The first is better suited to western text, the latter to everything else, but using either of them will guarantee that every line of the payload is a reasonable length, mostly less than 80 characters long. You’re probably using it already, as it’s also the best way to have non-ascii characters in your email.

So that’s the email body sorted. What about the emails headers?

Long Headers

RFC 5322 specifies how the headers and body of an email are structured and has mechanics to deal with long headers by “folding” them. Folding means adding a CRLF (carriage return followed by line feed - the normal line-ending in email) at an appropriate place in the header.

Those “appropriate” places to fold a header are defined in the grammar for headers as “FWS”. That just means a space or tab character in a place where the header can be folded. There’s no simple way to tell whether a whitespace character is “FWS” or not - most are, but you need to check the grammar in the RFC that specifies the particular header field to know for sure.

(You’ll often see the term “CFWS” used instead of “FWS”. It’s a place in a header where both folds and comments can be added, where a comment is just human-readable text wrapped in parentheses.)

Folding a header doesn’t affect the “value” or semantic meaning of it and, more immediately useful to us, doesn’t affect DKIM signatures that use “relaxed” header canonicalization (which is the normal practice). That means it would be safe for a smarthost to fold any headers it’s given that are “too long” before sending the mail on towards the recipient. (Do smarthosts actually do this? I know some do, but until recently I’ve not seen many headers that are more than a thousand characters long in the wild).

So even long headers are probably OK, as long as they have some whitespace in them.

What if there’s no whitespace?

What if you have a header that, say, looks like this:

List-Unsubscribe: <https:// ... 1200 character long URL ... >

Will that get folded?

It might get folded in the space after “List-Unsubscribe” but the URL has no whitespace in it, so it can’t be folded anywhere else. It’s going to end up creating a line in the payload that’s >1200 characters long. And that’s not allowed.

When you do this you’re violating the SMTP RFCs so nothing is guaranteed about what’ll happen to your message. (What you’re sending isn’t technically an email so no rules or expectations have to be honored.)

A mailbox provider seeing this arrive may want to “fix up” the message, so they don’t have invalid emails floating around their internal system. Invalid and badly formed emails are a common security issue, as they may bypass malware filters, or look like one thing to a spam filter and something different to an end user, or cause buffer overflows in mail software, or all sorts of other bad things - so it’s reasonable to want to avoid them.

The only way to cleanly fix up a long header is to fold it - and we can’t fold this one because it doesn’t have FWS where we’d need it to be. So the only thing we can do is to brutally truncate the header, or insert a tab-CR-LF sequence to make the header look like a folded header - and doing either will change the value of the header.

Doing either of those things will mean the excessively long header can’t threaten tender, vulnerable mail software anymore. But it will also break the URL in the List-Unsubscribe field, and break any DKIM signature that covered that header.

If you do the fixup before checking the DKIM signature - which you would, as one of the bits of software you’re trying to protect is the DKIM validator - then the DKIM signature will be broken.

Microsoft appear to do this, so “my list unsubscribe URL is too long” is another potential answer to the question we started with - Why are my DKIM signatures failing at Microsoft?

The only way to fix it is to make sure that the header isn’t more than 950 or so characters long, or if it has to be that long to make sure that it has folding white space in places that allow it to be folded into chunks each no longer than that.

In the specific case of a List-Unsubscribe header you have to make the URL shorter; there’s no other fix. Links don’t need to be that long.