Why Deliverability Matters to Me

laura
Industry
June 17, 2024

Welcome to deliverability week. I want to especially thank Al for doing a lot of work behind the scenes herding this group of cats. He’s an invaluable asset to the community.

The topic today is why deliverability matters to me. Until I started writing this, I hadn’t really thought about it much. Like most folks, I kinda fell into deliverability. I still describe myself as a scientist by training.

I think that to figure out why it matters, I need to start at the beginning and talk about the impact of email on my life and at why email matters to me. Email matters because it’s a communication tool and introduced me to thousands of people I would have never been able to interact with. Deliverability matters because it’s the key to keeping email as a viable and practical communication tool.

Ancient History (pre-2000)

My first job out of high school was interning in a molecular biology research lab at the FDA on Capitol Hill. That was where I had my first exposure to BITNET and then later the Internet. My interest in email actually started then. My boyfriend at the time stayed down in Blackburg for the summer, while I was up in DC. We tried to email each other, but couldn’t.

When I moved to grad school, I got a non .gov email address. When I got my first spam, sometime in 94 or 95, I used tracking spammers down as a way to understand email and learn about the internet. Throughout the 90s fighting spam was a hobby I did between experiments. I started participating in the news.admin.net-abuse USENET groups, anti-spam mailing lists like spam-l and some anti-spam related IRC channels. Many of the folks I met then are still friends and colleagues – including Al. This is also where Steve and I met even though we were living and working in two different time zones.

By 2000 Steve and I were married and living in California where he was working for UltraDNS. I was looking for jobs in the booming Bay Area biotech scene, but couldn’t find quite the right position. After a few months some of the other folks I met on USENET were working for MAPS and recruiting me to come work with them.

The job that really interested me was not on the blocklist side of things, but rather the education and compliance side of things. We were the carrot backed up by the blocklist stick. My first position there was managing a team of folks handling abuse complaints for a major network provider. As the company grew, I moved over to head up client services. I was tasked with developing an outsourced abuse desk product for MAPS to sell to third parties. I’d just made our first sale back in early 2001 when the company laid off most of us not on the blocklist side of things.

The start (early 00’s)

I mentioned Steve was working at UltraDNS at the time. The company was founded by Rodney Joffe who I talked about when he won the Mary Litynski Award. He started recommending me to his friends and colleagues to help them solve their MAPS blocklisting problems.

Deliverability wasn’t really a thing when I started consulting for companies. Mostly I would go into a company and look at their processes and explain to them how to comply with MAPS requirements to get delisted. It wasn’t about inbox or not inbox. It was solving the immediate blocking problem.

Spam filters were very primitive. They blocked by IPs and sometimes domains. There were the vague beginnings of spam folders at some mailbox providers, but they weren’t what we think of today. People were still manually writing rules to block spam.

The infrastructure we now rely on for deliverability just didn’t exist.

FBLs didn’t exist.
Gmail didn’t exist
Authentication didn’t exist.
CAN SPAM didn’t exist.
Report Spam buttons didn’t exist.
Best Practice recommendations consisted of “use confirmed opt-in”.

We were all flying by the seat of our pants and trying to navigate wholly unknown terrain. We were fighting to preserve an email ecosystem without the tools or the knowledge or the community that we have today.

What it means to me

At the end of the day, I still consider myself an anti-spammer. What I do is all about stopping spam. Many of those USENET colleagues and friends ended up founding or building filtering companies. Others work for blocklists or abuse or postmaster desks. I took a different path. Instead of going down the blocking route, I stayed on the side of the carrot. My anti-spam work is teaching companies ways to send email that are respectful of the recipient. They can make money from an opt-in model and they can be effective even if they’re not blasting out email to every address they can find.

To me, email is magical. It gave me a way to get to know and connect with people I would otherwise never have met. It let me continue relationships that spanned the globe. I’ve gotten multiple jobs due to the relationships I’ve created through email. My first post-grad school research job was for someone I met on an discussion list. My job at MAPS was definitely due to my online activities. I even met the guy I’ve been married to for 25 years (this July!) because of email and specifically spam.

Email is special. Email deserves to be protected because it’s valuable. It’s important. Filters and blocklists have their place. But someone needs to be the carrot. Someone needs to show marketers and companies how to use email non-destructively. How to make email work for everyone. That’s the path I chose so long ago when MAPS was courting me. It’s the path I’ve traveled for years now. It’s a path I’m proud of.

Deliverability is important because we are the protectors and the caretakers of the email ecosystem.

Want some history?

I was doing some research today for an article I’m working on. The research led me to a San Francisco Law Review article from 2001 written by David E. Sorkin. Technical and Legal Approaches to Unsolicited Electronic Mail (.pdf link). The text itself is a little outdated, although not as much as I expected. There’s quite a good discussion of various ways to control spam, most of which are still true and even relevant.

From a historical perspective, the footnotes are the real meat of the document. Professor Sorkin discusses many different cases that together establish the rights of ISPs to filter mail, some of which I wasn’t aware of. He also includes links to then-current news articles about filtering and spam. He also mentions different websites and articles written by colleagues and friends from ‘back in the day’ discussing spam on a more theoretical level.
CNET articles on spam and filtering was heavily referenced by Professor Sorkin. One describes the first Yahoo spam folder. Some things never change, such as Yahoo representatives refusing to discuss how their system works. There were other articles discussing Hotmail deploying the MAPS RBL (now a part of Trend Micro) and then adding additional filters into the mix a few weeks later.
We were all a little naive back then. We thought the volumes of email and spam were out of control. One article investigated the effectiveness of filters at Yahoo and Hotmail, and quoted a user who said the filters were working well.

Content, trigger words and subject lines

There’s been quite a bit of traffic on twitter this afternoon about a recent blog post by Hubspot identifying trigger words senders should avoid in an email subject line. A number of email experts are assuring the world that content doesn’t matter and are arguing on twitter and in the post comments that no one will block an email because those words are in the subject line.
As usually, I think everyone else is a little bit right and a little bit wrong.
The words and phrases posted by Hubspot are pulled out of the Spamassassin rule set. Using those words or exact phrases will cause a spam score to go up, sometimes by a little (0.5 points) and sometimes by a lot (3+ points). Most spamassassin installations consider anything with more than 5 points to be spam so a 3 point score for a subject line may cause mail to be filtered.
The folks who are outraged at the blog post, though, don’t seem to have read the article very closely. Hubspot doesn’t actually say that using trigger words will get mail blocked. What they say is a lot more reasonable than that.

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays. By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list. If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list. Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list. RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list. You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,