Why Deliverability Matters to Me

laura
Industry
June 17, 2024

Welcome to deliverability week. I want to especially thank Al for doing a lot of work behind the scenes herding this group of cats. He’s an invaluable asset to the community.

The topic today is why deliverability matters to me. Until I started writing this, I hadn’t really thought about it much. Like most folks, I kinda fell into deliverability. I still describe myself as a scientist by training.

I think that to figure out why it matters, I need to start at the beginning and talk about the impact of email on my life and at why email matters to me. Email matters because it’s a communication tool and introduced me to thousands of people I would have never been able to interact with. Deliverability matters because it’s the key to keeping email as a viable and practical communication tool.

Ancient History (pre-2000)

My first job out of high school was interning in a molecular biology research lab at the FDA on Capitol Hill. That was where I had my first exposure to BITNET and then later the Internet. My interest in email actually started then. My boyfriend at the time stayed down in Blackburg for the summer, while I was up in DC. We tried to email each other, but couldn’t.

When I moved to grad school, I got a non .gov email address. When I got my first spam, sometime in 94 or 95, I used tracking spammers down as a way to understand email and learn about the internet. Throughout the 90s fighting spam was a hobby I did between experiments. I started participating in the news.admin.net-abuse USENET groups, anti-spam mailing lists like spam-l and some anti-spam related IRC channels. Many of the folks I met then are still friends and colleagues – including Al. This is also where Steve and I met even though we were living and working in two different time zones.

By 2000 Steve and I were married and living in California where he was working for UltraDNS. I was looking for jobs in the booming Bay Area biotech scene, but couldn’t find quite the right position. After a few months some of the other folks I met on USENET were working for MAPS and recruiting me to come work with them.

The job that really interested me was not on the blocklist side of things, but rather the education and compliance side of things. We were the carrot backed up by the blocklist stick. My first position there was managing a team of folks handling abuse complaints for a major network provider. As the company grew, I moved over to head up client services. I was tasked with developing an outsourced abuse desk product for MAPS to sell to third parties. I’d just made our first sale back in early 2001 when the company laid off most of us not on the blocklist side of things.

The start (early 00’s)

I mentioned Steve was working at UltraDNS at the time. The company was founded by Rodney Joffe who I talked about when he won the Mary Litynski Award. He started recommending me to his friends and colleagues to help them solve their MAPS blocklisting problems.

Deliverability wasn’t really a thing when I started consulting for companies. Mostly I would go into a company and look at their processes and explain to them how to comply with MAPS requirements to get delisted. It wasn’t about inbox or not inbox. It was solving the immediate blocking problem.

Spam filters were very primitive. They blocked by IPs and sometimes domains. There were the vague beginnings of spam folders at some mailbox providers, but they weren’t what we think of today. People were still manually writing rules to block spam.

The infrastructure we now rely on for deliverability just didn’t exist.

FBLs didn’t exist.
Gmail didn’t exist
Authentication didn’t exist.
CAN SPAM didn’t exist.
Report Spam buttons didn’t exist.
Best Practice recommendations consisted of “use confirmed opt-in”.

We were all flying by the seat of our pants and trying to navigate wholly unknown terrain. We were fighting to preserve an email ecosystem without the tools or the knowledge or the community that we have today.

What it means to me

At the end of the day, I still consider myself an anti-spammer. What I do is all about stopping spam. Many of those USENET colleagues and friends ended up founding or building filtering companies. Others work for blocklists or abuse or postmaster desks. I took a different path. Instead of going down the blocking route, I stayed on the side of the carrot. My anti-spam work is teaching companies ways to send email that are respectful of the recipient. They can make money from an opt-in model and they can be effective even if they’re not blasting out email to every address they can find.

To me, email is magical. It gave me a way to get to know and connect with people I would otherwise never have met. It let me continue relationships that spanned the globe. I’ve gotten multiple jobs due to the relationships I’ve created through email. My first post-grad school research job was for someone I met on an discussion list. My job at MAPS was definitely due to my online activities. I even met the guy I’ve been married to for 25 years (this July!) because of email and specifically spam.

Email is special. Email deserves to be protected because it’s valuable. It’s important. Filters and blocklists have their place. But someone needs to be the carrot. Someone needs to show marketers and companies how to use email non-destructively. How to make email work for everyone. That’s the path I chose so long ago when MAPS was courting me. It’s the path I’ve traveled for years now. It’s a path I’m proud of.

Deliverability is important because we are the protectors and the caretakers of the email ecosystem.

The inbox is a moving target

The more I look at the industry, the more convinced I am that we’re in the middle of a fundamental shift in how email is filtered. This shift will change how we handle email deliverability and what tools we have and what information we can use as senders to address challenges to getting to the inbox.

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays. By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list. If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list. Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list. RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list. You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,

Troubleshooting delivery is hard, but doable

Even for those of us who’ve been around for a while, and who have a lot of experience troubleshooting delivery problems things are getting harder. It used to be we could identify some thing about an email and if that thing was removed then the email would get to the inbox. Often this was a domain or a URL in the message that was triggering bulk foldering.
Filters aren’t so simple now. And we can’t just randomly send a list of URLs to a test account and discover which URL is causing the problem. Sure, one of the URLs could be the issue, but that’s typically in context with other things. It’s rare that I can identify the bad URLs sending mail through my own server these days.
There are also a lot more “hey, help” questions on some of the deliverability mailing lists. Most of these questions are sticky problems that don’t map well onto IP or domain reputation.
One of my long term clients recently had a bad mail that caused some warnings at Gmail.
We tried a couple of different things to try and isolate the problem, but never could discover what was triggering the warnings. Even more importantly, we weren’t getting the same results for identical tests done hours apart. After about 3 days, all the warnings went away and all their mail was back in the inbox.
It seemed that one mailing was really bad and resulted in a bad reputation, temporarily. But as the client fixed the problem and kept mailing their reputation recovered.
Deliverability troubleshooting is complicated and this flowchart sums up what it’s like.

Here at Word to the Wise, we get a lot of clients who have gone through the troubleshooting available through their ESPs and sometimes even other deliverability consultants. We get the tough cases that aren’t easy to figure out.
What we do is start from the beginning. First thing is to confirm that there aren’t technical problems, and generally we’ll find some minor problems that should be fixed, but aren’t enough to cause delivery problems. Then we look at the client’s data. How do they collect it? How do they maintain it? What are they doing that allows false addresses on their list?
Once we have a feel for their data processes, we move on to how do we fix those processes. What can we do to collect better, cleaner data in the future? How can we improve their processes so all their recipients tell the ISP that this is wanted mail?
The challenging part is what to do with existing data, but we work with clients individually to make sure that bad addresses are expunged and good addresses are kept.
Our solutions aren’t simple. They’re not easy. But for clients who listen to us and implement our recommendations it’s worth it. Their mail gets into the inbox and deliverability becomes a solved problem.