The inbox is a moving target

laura
Industry
July 13, 2018

The more I look at the industry, the more convinced I am that we’re in the middle of a fundamental shift in how email is filtered. This shift will change how we handle email deliverability and what tools we have and what information we can use as senders to address challenges to getting to the inbox.

Early deliverability

This period is roughly between 2001 and 2006. Many of the email filters were focused on blocking unsolicited email. They were mostly IP based and were very all or nothing. Mail is either accepted or it’s not. It wasn’t until 2003 that Yahoo and AOL (now OATH) announced spam folders that they would put mail into. There were few formal communication channels between ISPs and mail senders.
We did, of course, have informal channels. Often delivery could be improved by asking the right people to make some adjustments to their internal filters. And, to be honest, there were some people who used this position of power to personally enrich themselves (the good news is most of them are no longer in the business).
As senders, there was very little information we had to work with. Most communication with ISPs went through gatekeepers. The primary gatekeepers were the companies providing certification services: Bonded Sender, Habeas, Goodmail.

First inflection point

The first inflection point happened in the middle of the 2000s. Over the course of a few years, there were some significant changes all of which served to create the email industry as we now know it. This post certainly isn’t going to cover the full range of changes, but there are a few notable things that happened.
Expansion of broadband. As broadband availability expanded and became more affordable, many of the old dialup ISPs simply folded. With it, folks stopped using ISP provided email addresses and moved to free email addresses provided by companies like Yahoo and Hotmail.
Gmail. Google launched the beta for Gmail in 2004. Most of what I remember about this was invites to the private beta being shared wildly. Everyone wanted to get in. Initial buzz aside, Gmail is consistently one of the top 3 domains for consumer and business (through hosted G Suite accounts) mailings alike.
Postmaster pages. ISPs started codifying what it took to get mail delivered to that ISP. One example is how pages published what they expected in terms of connections and messages per connection. Those pages also started explaining some of the SMTP return codes ISPs were using.
Feedback loops. In 2006 ISPs all over the US started announcing Feedback Loops. What a lot of folks don’t realize is this expansion in FBLs was driven in part by Return Path’s willingness to provide infrastructure to the ISPs to manage FBL registration.
These changes, along with some others, worked together to create a delivery environment that, while not always welcoming to bulk mail, was understandable.
For the next 5 to 8 years, this was the environment we worked in. Sure, there was some consolidation in the email space: ESPs were launched and bought, new deliverability companies arose, some accreditation companies folded. Overall, though, it was pretty solid and companies that followed a few simple rules were able to reach the inbox.

Present day

Today, though, I’m seeing a lot of those rules changing. Companies who keep their bounces and complaints low are still sometimes seeing delivery problems. They’re not able to reach the inbox, for at least some of their mail streams. Again, there are a number of things happening concurrently that is driving this change.
ISPs are changing. Two of the 4 major consumer ISPs Microsoft, AOL, Gmail, Yahoo (MAGY) providers have merged, to form what I’m fondly calling OMG (OATH, Microsoft, Gmail). We’re also seeing consolidation in the cable space with the Time Warner / ATT merger. Other cable companies are merging, too, and rethinking things like their postmaster pages. Additionally, FBLs are going away and no ISP offers whittling anymore.
Privacy changes. We’re also seeing a global focus on privacy and tracking and user data. A lot of this is translating into how the email providers and ISPs are handling incoming email. A lot of the changes we’ve seen in the last 12 months was driven, in part, by ISPs trying to upgrade their infrastructure to deal with the effects of GDPR. But California is passing its own GDPR style law, which will affect technology companies across the US.
Security Concerns. Email is a primary source of infections inside corporations. All of the major breeches we’ve heard about in the past few years have stared with email. Filters aren’t about spam any more, they’re about protecting end users and corporations from malicious traffic. This is going to affect how mail is processed and delivered and what can and cannot reach the inbox.
This is a very abbreviated list of the things that were happening at each stage in the process.
One of the bigger challenge is the ISPs are moving to less human involvement in filters and answering deliverability questions. They’re moving to the Hotmail “fill out the form and parse the boilerplate” model of deliverability support. Or they’re moving even further to the Gmail style “here’s a help page, have fun!” Even folks who know people inside the ISPs, are having problems getting the same helpfulness of answers that they had in the past. I’m not blaming the ISPs or the folks who work there. But, as I see it, providing support to senders was often a way for ISPs to tweak their filters. Now that the ISPs are using machine learning and AI as a way to filter mail, they need less in the way of direct feedback.
Overall, I think we’re in the early stages of another major shift in how email is managed at the ISPs. This will change how we have to send emails in order to reach the inbox. We have to think about the filters and what their goals are. Having good stats isn’t going to be enough. Almost anyone can have good stats, and there are large companies senders can pay to clean up their stats to be within the ISP thresholds. Once you can buy low bounce rates or low complaint rates, those stats are useless for determining the health of your email program.
Smart senders will start thinking about these issues sooner rather than later.

Politics and Delivery

Last week I posted some deliverability advice for the DNC based on their acquisition of President Obama’s 2012 campaign database. Paul asked a question on that post that I think is worth some attention.

Do system administrators have too much power?

Yesterday, Laura brought a thread from last week to my attention, and the old-school ISP admin and mail geek in me felt the need to jump up and say something in response to Paul’s comment. My text here is all my own, and is based upon personal experience as well as those of my friends. That said, I’m not speaking on their behalf, either. 🙂
I found Paul’s use of the word ‘SysAdmin’ to be a mighty wide (and — in my experience — probably incorrect) brush to be painting with, particularly when referring to operations at ISPs with any significant number of mailboxes. My fundamental opposition to use of the term comes down to this: It’s no longer 1998.
The sort of rogue (or perhaps ‘maverick’) behavior to which you refer absolutely used to be a thing, back when a clean 56k dial-up connection was the stuff of dreams and any ISP that had gone through the trouble to figure out how to get past the 64k user limit in the UNIX password file was considered both large and technically competent. Outside of a few edge cases, I don’t know many system administrators these days who are able to (whether by policy or by access controls) — much less want to — make such unilateral deliverability decisions.
While specialization may be for insects, it’s also inevitable whenever a system grows past a certain point. When I started in the field, there were entire ISPs that were one-man shows (at least on the technical side). This simply doesn’t scale. Eventually, you start breaking things up into departments, then into services, then teams assigned to services, then parts of services assigned to teams, and back up the other side of the mountain, until you end up with a whole department whose job it is to run one component of one service.
For instance, let’s take inbound (just inbound) email. It’s not uncommon for a large ISP to have several technical teams responsible for the processing of mail being sent to their users:

Troubleshooting delivery is hard, but doable

Even for those of us who’ve been around for a while, and who have a lot of experience troubleshooting delivery problems things are getting harder. It used to be we could identify some thing about an email and if that thing was removed then the email would get to the inbox. Often this was a domain or a URL in the message that was triggering bulk foldering.
Filters aren’t so simple now. And we can’t just randomly send a list of URLs to a test account and discover which URL is causing the problem. Sure, one of the URLs could be the issue, but that’s typically in context with other things. It’s rare that I can identify the bad URLs sending mail through my own server these days.
There are also a lot more “hey, help” questions on some of the deliverability mailing lists. Most of these questions are sticky problems that don’t map well onto IP or domain reputation.
One of my long term clients recently had a bad mail that caused some warnings at Gmail.
We tried a couple of different things to try and isolate the problem, but never could discover what was triggering the warnings. Even more importantly, we weren’t getting the same results for identical tests done hours apart. After about 3 days, all the warnings went away and all their mail was back in the inbox.
It seemed that one mailing was really bad and resulted in a bad reputation, temporarily. But as the client fixed the problem and kept mailing their reputation recovered.
Deliverability troubleshooting is complicated and this flowchart sums up what it’s like.

Here at Word to the Wise, we get a lot of clients who have gone through the troubleshooting available through their ESPs and sometimes even other deliverability consultants. We get the tough cases that aren’t easy to figure out.
What we do is start from the beginning. First thing is to confirm that there aren’t technical problems, and generally we’ll find some minor problems that should be fixed, but aren’t enough to cause delivery problems. Then we look at the client’s data. How do they collect it? How do they maintain it? What are they doing that allows false addresses on their list?
Once we have a feel for their data processes, we move on to how do we fix those processes. What can we do to collect better, cleaner data in the future? How can we improve their processes so all their recipients tell the ISP that this is wanted mail?
The challenging part is what to do with existing data, but we work with clients individually to make sure that bad addresses are expunged and good addresses are kept.
Our solutions aren’t simple. They’re not easy. But for clients who listen to us and implement our recommendations it’s worth it. Their mail gets into the inbox and deliverability becomes a solved problem.