SMTP
Life of an Email
I’m repeating the presentation I gave at M3AAWG in London for the Certified Senders Alliance.
It’s all about how to send an email by hand, and how knowing the mechanics of how an email is sent can help us diagnose email delivery issues.
We’re starting in about five hours from when I post this.
Register at https://register.gotowebinar.com/register/2268789893122531343
New Deliverability Resource
The nice folks over at Postmark shared a new deliverability resource last week. The SMTP Field Manual. This is a collection of SMTP responses they’ve seen in the wild. This is a useful resource. They’re also collecting responses from other senders, meaning we can crowdsource a useful resource for email deliverability folks.
Read MoreAutomated link checking getting more sophisticated
As the volume and severity of malicious email increases, filters are increasingly following links in emails. This is really nothing new. Barracuda and other filters have been inspecting links automatically for years. From what I’ve seen there does seem to be some level of risk analysis based on domain reputation. That makes sense, not only is following links computationally expensive, it can also delay mail receipt.
Read MoreWhat’s a bounce?
Bounces and bounce handling is one of those topics I’ve avoided writing about for a long time. Part of my avoidance is because there are decades of confusing terminology that hasn’t ever been really defined. Untangling that terminology is the first step to being able to talk sensibly about what to do. Instead of writing a giant long post, I can break it into smaller, more focused posts.
Read MoreSend Actual SMTP
It’s rare I find mail that violates the SMTP spec (rfc5321 and rfc5322). I’ve even considered removing “send mail from a correctly configured mail server” from my standard Best Practices litany.
Read MoreThe history of email
My first access to “the internet” was through a dialup modem on a VAX at the FDA. I was a summer intern there through my college career and then worked full time after graduation and before grad school. My email address ended in .bitnet. I could mail some places but not others. One of the places I couldn’t send mail was to my friends back on campus.
A few of those friends were computer science majors, so one weekend they tried to help me troubleshoot things. . There were text files that they ended up searching through looking up how to send mail from .bitnet to .edu. But it was all a baffling experience. Why couldn’t it just work? I had email, they had email, why could we not talk?
I never did figure out how to send email to campus from .bitnet.
Eventually, the FDA moved from BITNET to the internet and I had a .gov address. I could send mail around just by getting the recipients’s address. But the mystery of why I could mail some .edus and not others still lingers. I wonder what our setup was that we couldn’t send mail. I’ll probably never know. I don’t even have enough details to explain the problem to someone who would know. I suspect the answer will be “bang paths” or “host.txt” files, but I really don’t know.
January 2016: The Month in Email
Happy 2016! We started off the year with a few different “predictions” posts. As always, I don’t expect to be right about everything, but it’s a useful exercise for us to look forward and think about where things are headed.
I joined nine other email experts for a Sparkpost webinar on 2016 predictions, which was a lot of fun (see my wrap up post here), and then I wrote a long post about security and authentication, which I think will be THE major topic in email this year both in policy and in practice (see my post about an exploit involving Trend Micro and another about hijacked Verizon addresses). Expect to hear more about this 2016 continues.
My other exciting January project was the launch of my “Ask Laura” column, which I hope will prove a great resource for people with questions about email. Please let me know if you have any questions you’d like to see me answer for your company or your clients — I’ll obscure any identifying information and generalize the answers to be most widely applicable for our readers.
In other industry news, it’s worth noting that Germany has ruled it illegal to harvest users’ address books (as Facebook and other services do). Why does that make sense? Because we’re seeing more and more phishing and scams that rely on social engineering.
In best practices, I wrote about triggered and transactional emails, how they differ, and what to consider when implementing them as part of your email program. Steve describes an easy-to-implement best practice that marketers often ignore: craft your mails so the most important information is shown as text.
I re-published an older post about SMTP rules that has a configuration checklist you might find useful as you troubleshoot any issues. And a newer issue you might be seeing is port25 blocking, which is important if you are hosting your own email senders or using SMTP to send to your ESP.
Finally, I put together some thoughts about reporting abuse. We work closely with high-volume abuse desks who use our Abacus software, and we know that it’s often not worth the time for an individual to report an incident – but I still think it’s worthwhile to have the infrastructure in place, and I wrote about why that is.
Following the SMTP rules
An old blog post from 2013, that’s still relevant today.
“Blocked for Bot-like Behavior”
An ESP asked about this error message from Hotmail and what to do about it.
“Bot-like” behaviour usually means the sending server is doing something that bots also do. It’s not always that they’re spamming, often it’s a technical issue. But the technical problems make the sending server look like a bot, so the ISP is not taking any chances and they’re going to stop accepting mail from that server.
If you’re an ESP what should you look for when tracking down what the problem is?
First make sure your server isn’t infected with anything and that you’re not running an open relay or proxy. Second, make sure your customers aren’t compromised or have had their accounts hijacked.
Then start looking at your configuration.
HELO/EHLO values
When did the reject happen?
Earlier today I approved a comment from Mike on a post about problems at AOL from 2012. The part of the comment that caught my attention:
Confusing the engineers
We went camping last weekend with a bunch of friends. Had a great time relaxing on the banks of the Tuolumne River, eating way too much and visiting.
On Saturday I was wearing a somewhat geeky t-shirt. It said 554: abort mission. (Thank you MessageSystems). At some point on Saturday every engineer came up to me, read my shirt and then looked at me and said “That’s not HTTP.”
That lead to various discussions about how their junior engineers don’t actually know SMTP at all. Why? Because the SMTP libraries just work. Apparently the HTTP libraries aren’t that great, so folks have to learn more about HTTP to troubleshoot and use them.
I’m sure there’s a joke in there somewhere: A Kindle engineer, an Android engineer and a robot engineer walk into a campsite…
It did leave me thinking, though, about how it’s not that easy to run your own mail server these days. Gone are the days when running your own server was cost effective and easy. These days, there is just too much spam coming in. Crafting filters is a skilled job. It’s not that hard to run good filters. But to run good filters takes time to do well.
There are also a lot of challenges to sending mail. One of the discussions I had at the campsite was how hard it was to configure outbound mail. The engineer was helping a friend set up a website and trying to get the website to send notifications to the friend. But without setting up authentication the mail kept silently failing.
Of course, we do run our own mail server. But it’s our job and, in many ways, it keeps us honest. We don’t run many filters meaning we see what spammers are doing and can use our own experiences to better understand what commercial filters are dealing with.
For most people, though, I really think using a service is the right solution. Find one with filters that meet your needs and just pay them to deal with the headache.
Email is inherently a malicious traffic stream
It’s something many people don’t think about, but the majority of the traffic coming into the SMTP port is malicious. Spam is passively malicious, in that it just uses resources and bothers people. But there is a lot of actively malicious traffic coming into the SMTP port. Email is used as a vector to spread viruses and other malware. Email is also used for phishing and scamming. Many of the major hacks we’ve heard about over the last few years, including those in the email space, started with a single user getting infected through email.
We talk a lot about delivery here with clients and primarily focus on making sure their mail looks as unlike malicious mail as possible. We focus on spam filters, but every piece of mail goes through filters that also look for viruses, phishes, malware and other malicious traffic.
Mail servers are under attack constantly. The only reason our inboxes are useful is through the hard work of many people to filter out the bad and keep users from seeing the bulk of the mess attacking them.
The best time to send email
This subject comes up over and over again. Many senders are convinced 
that there is a best time to send email. Countless research hours have been dedicated to finding that best time to send email. Numerous blog posts discuss what the best time to send email is.
From my perspective, there are better places for senders to spend time than figuring out what the exact right time is.But, senders still ask when the best time to send mail is.
There are a lot of reasons I can come up with as to why there’s no best time to send email. But the really big one is that when you send a mail has no impact on when it gets delivered. There are multiple steps between hitting the send button and the mail being delivered to the inbox totally outside the control of the individual sender.
Email is designed as “store-and-forward.” This means there are potential delays at multiple steps inside the process.
Sending queues are called queues for a reason. Emails are sent out individually, particularly when an ESP uses VERP as part of its sending. There is actually a time overhead for making a connection to a recipient server and sending the email.
Receivers have queues, too. They can only accept so many incoming connections at a time. They have limited resources to accept all the mail their users want.
Receivers may delay mail between accepting it at the MX and delivering it to the inbox. This isn’t ideal and it’s not usual, but it can happen.
Recipients using IMAP accounts may not check mail regularly. They may only collect mail a few times a day.
These are only a few of the reasons that send time doesn’t necessarily equate with delivery time. Of course, 99% of the time email is mostly instantaneous. The internet is robust enough that a message sent is delivered seconds later. I see it happen all the time, when colleagues and I send email during calls. But, when mail fails, it sometimes fails spectacularly. Back in the dark ages (of the early 90s) I had an email that took almost a year to get to the recipients. Best I can tell, it got stuck somewhere in the depths of a machine in the middle of the university mail system. Eventually that system fell over and someone noticed and rebooted it (maybe it was walled up somewhere?). The reboot shook my message out of where ever it was stuck.
SWAKS: the SMTP Swiss Army Knife
SWAKS is a general purpose testing tool for SMTP. For basic SMTP testing it’s a more convenient, scriptable alternative to running a transaction by hand, but it also lets you test things that are difficult to do manually, such as authentication or TLS encryption.
It’s a perl script that installs fairly easily on OS X or any Linux/unix system (and can be installed on Windows, if you have perl installed there).
It’s pretty well documented, but it can be a bit overwhelming to start with. Here are some simple recipes:
Send a test email:
SMTP Level Rejections
While discussing a draft of a Deliverability BCP document the issue came up of what rejections at different phases of the email delivery transaction can mean. That’s quite a big subject, but here’s a quick cheat sheet.
At initial connection
Dropped or failed connection:
Does email have a guarantee of delivery?
A client asked me earlier this week what SLAs ISPs provided for email delivery. The short answer is that there isn’t a SLA and that the only guarantee is that the email will get there when it gets there.
But as I was mentioning this to Steve, he pointed out that there was a recent change in the RFCs for email. In both RFC 821/2 and RFC 2821/2 (the original email related RFCs and the update in the early 2000’s) the RFCs stated that once a receiving MTA accepted an email that that MTA was required to either delivery the mail or generate an asynchronous bounce. While this isn’t a standard SLA, it does mean that a 2xy response after DATA meant the email would either be delivered to the user or be sent back to the sender. Despite the RFC requirements some receivers would still drop mail on the floor for various reasons, sometimes intentionally and sometimes not.
RFC 5321/2, the current SMTP standard, still says that once a server accepts the mail it must not lose that mail ‘for frivolous reasons.’ The RFC goes on to admit, though, that in recent years, SMTP servers are under a range of attacks and dropping mail on the floor is not frivolous in those cases.
8 things that make your mail look like spam
In the comments of last week’s Wednesday question John B. asked
Read MoreThe Physics of the Email Universe
We talk a lot about rules and best practices in email, but we’re mostly talking about “squishy” rules-of-thumb that are based on simplified models of how mail systems, spam filters, recipients, postmasters and blacklist operators behave. They’re the biology, ecology and sociology of the email ecosystem.
There’s another set of rules we tend to only mention in passing, if at all, though. They’re the steely, sharp-edged laws that control the email universe. They’re the RFCs that define how email works and make sure that mail systems written by hundreds of different people across the globe all work and all interoperate with each other.
Building a message from Zeros and Ones
RFC 5322 – Internet Message Format
This tells you everything you need to know about crafting a simple email, with a subject line, a sender, some recipients and a simple plain-text message. It’s also the foundation of all fancier emails. If you’re creating emails, this is where to start.
A little more than plain ASCII
RFC 2047 – MIME Part 3: Message Header Extensions for Non-ASCII Text
RFC 2047 is one small part of the MIME (Multipurpose Internet Mail Extensions) suite of protocols that allow you to include pictures and attachments and prettily formatted text and comic sans in your email. This part defines how you can put things other than the plainest of plain text in your subject lines or in the “friendly from” of your message. It’s what allows you to put Hiragana, or Cyrillic, or umlauts, or cedillas, or properly matched double quotes in your subject line. It also let’s you put hearts or smiley faces or other little pictograms there – but nothing this useful is going to be perfect.
RFC 2045 – MIME Part 1: Format of Internet Message Bodies
This shows how to send an image, or a plain text mail in a different character set, or an HTML mail. It doesn’t tell you how to send plain text and HTML, or to send HTML with embedded images, or a message with an attached document. For that you need…
Finally, Modern Email
RFC 2046 – MIME Part 2: Media Types
This builds on RFC 2045 to allow you to have many different chunks in a message – this is what you need if you want to send “proper” HTML mail with a plain text alternative, or if you want embedded images or attachments.
Getting From A To B
RFC 5321 – Simple Mail Transfer Protocol
A message isn’t much use unless you send it somewhere. RFC 5321 explains the mysteries of actually sending that message over the wire to the recipient. If you need to know about the different phases of a message delivery, what “4xx” and “5xx” actually mean, why there’s not really any such thing as a hard or soft bounce defined, just temporary or permanent failures, or anything else about actually sending mail or diagnosing mail delivery, this is your starting point.
The Rest Of The Iceberg
I’ve only touched on the very smallest tip of the email iceberg here. There’s much, much more – both in RFCs and ad-hoc non-RFC standards. If you’re interested in more, this is a decent place to start.
Email is store and forward
Many of us are so used to email appearing instantaneous, we forget that the underlying protocol was never designed for instant messaging. When the SMTP protocol was originally proposed it was designed to support servers that may have had intermittent connectivity. The protocol allowed for email to be spooled to disk and then sent when resources were available. In fact, almost everyone who was around more than 10 years ago knows of a case where an email took weeks, months or even years to deliver.
These days we’re spoiled. We expect the email we send to friends and relatives to show up in their mailbox within moments of sending it. We expect that sales receipt or e-ticket to show up in our mailbox within instants of a purchase. We expect that our ISPs will get us email immediately, if not sooner.
But there are a lot of things that can slow down email delivery. At several points in the process an email may be spooled to disk. It stays on the spool until the next part of the delivery process can happen. Other points of slowdown include the various anti-spam, anti-virus and anti-phishing protections that ISPs must implement. Then add in the extreme volume of email (around 10 billion messages a day) and all of a sudden email delivery is slower than many senders and recipients expect it to be. This delay is not ideal, but the system is designed so that mail is not silently discarded.
While individual emails may be delayed, most users will rarely see that delay in the email that they send. Bulk senders, who may be sending thousands or hundreds of thousands of emails a day, may see more delays in a single send than the average user sees in years of sending one-to-one email.
Email is store and forward, not instant. Sometimes that means there is a delay in getting email into the recipients inbox. And, sometimes there isn’t anything anyone can do to speed up delivery, except to adjust expectations of how email works.