Updating the filtering model

laura
Industry
April 6, 2018

One thing I really like about going to conferences is they’re often one of the few times I get to sit and think about the bigger email picture. Hearing other people talk about their marketing experiences, their email experiences, and their blocking experiences usually triggers big picture style thoughts.
Earlier this week I was at Activate18, hosted by Iterable. The sessions I attended were interesting and insightful. Of course, I went to the deliverability session. While listening to the presentation, I realized my previous model of email filtering needed to be updated.

The old model

The old model was pretty simple. It was based on the idea that ISPs ask two fundamental questions about email when it comes in.

Is it safe?
Is it wanted?

If the answer to both those questions is yes, the mail is delivered to the inbox.
Safe is something we don’t talk much about in the marketing space, because generally our mail is safe. But our mail has to go through the exact same filters that are set out to catch the bad guys. And sometimes we do things that trigger that set of filters unintentionally.
The wanted is where marketers can really shine. ISPs look at their user behavior to determine if mail is wanted or not. While the measurements are slightly different than what marketers use, Marketers must also look at how wanted mail is. Engagement with the email is part of it. But, you can also use other metrics you have. Do they visit your website? Are they active on your FB page? What other data do we have that says this person is engaged with your brand and won’t object to increased volume.
Spammers send unwanted mail. Spammers send mail to a lot of addresses where the address owner doesn’t log in. Spammers send mail to people who don’t want it, so they delete it immediately. If you are sending lots of mail and your recipient demographic looks like the typical spammer recipient demographic, then your mail will be treated like spam. It’s ALL about the recipient. That’s what the ISP uses to measure your mail. And if your recipients are reacting to your mail in the same way they react to spammer mail, then you’re going to face deliverability challenges.
Increasing volume is expected, but you need to be strategic with how you increase it. I’m working with one of my clients right now to help them with the final touches on the holiday marketing program. They are increasing mail, but we made some changes to their proposed program to protect them against deliverability challenges. They’re in the early stages of warmup (yes, you need to warmup for significant volume changes!) and it’s looking good.
(That’s from an email I wrote to answer a question on the Only Influencers mailing list back in 2015)

The newer model

This week, though, I realized that another question snuck into the equation. The ISPs are still asking if mail is safe. Does it have harmful content? Phishing? Viruses? Is it coming from a botnet? ISPs use IP reputation and domain/URL reputation to answer the bulk of these questions during the SMTP transaction. If an ISP determines the mail isn’t safe, they’ll reject it out of hand.
But with technology improvements and machine learning, ISPs are able to split the second question into two sub questions.

Is it unsolicited?
Is it wanted by the recipient?

The unsolicited piece was always part of the equation. Many of the metrics used to answer the “is it wanted” question were actually trying to determine if the mail was unsolicited. These measurements include the things we talk about: bounce rate, complaint rate, unknown user rate. There are two reasons I think this is worth pointing out. The first is that I have often glossed over how much unsolicited mail “legitimate” senders actually send. Every company who purchases a list, who uses lead gen, who collects addresses at point of sale send unsolicited email. They may not mean to, but they do. The second reason is this is the piece of spam filtering that data hygiene companies are addressing. Everyone who cleans your list, identifies bounces, finds bad users, they’re specifically addressing this filtering. And many senders don’t understand why their mail is still going to bulk even after they’ve purchased very expensive hygiene services.
The reality is, that hygiene services make mail look less unsolicited, by removing many of the markers that tell ISPs the mail is unsolicited. But that doesn’t make the mail any more wanted by the recipient. Hence the current focus on engagement and individual delivery metrics. These are metrics that can’t be faked by the sender. Third parties can’t identify those recipients that want a particular piece of mail. And, with privacy laws like GDPR, it’s unlikely those business models will be cost effective.

Politics and Delivery

Last week I posted some deliverability advice for the DNC based on their acquisition of President Obama’s 2012 campaign database. Paul asked a question on that post that I think is worth some attention.

Email marketing ulcers for the holiday

I’ve mentioned here before that I can usually tell when the big ISPs are making changes to their spam filtering as that ISP dominates my discussions with current and potential clients and many discussions on delivery mailing lists.
The last two weeks the culprit has been Yahoo. They seem to be making a lot of changes to their filtering schemes right at the busiest email marketing time of the year. Senders are increasing their volume trying to extract that last little bit of cash out of holiday shoppers, but they’re seeing unpredictable delivery results. What worked to get mail into the inbox a month ago isn’t working, or isn’t working as well, now.
Some of this could be holiday volume related. Many marketers have drastically increased their mail volume over the last few weeks. But I don’t think the whole issue is simply that there is more email marketing flowing into our mailboxes.
As I’ve been talking with folks, I have started to see a pattern and have some ideas of what may be happening. It seems a lot of the issue revolves around bulk foldering. Getting mail accepted by the MXs seems to be no different than it has been. The change seems to be based on the reputation of the URLs and domains in the email.
Have a domain with a poor reputation? Bulk. Have a URL seen in mail people aren’t interested in? Bulk. Have a URL pointing to a website with problematic content? Bulk.
In the past IPs that were whitelisted or had very good reputations could improve delivery of email with neutral or even borderline poor reputations. It seems that is no longer an effect senders can rely on. It may even be that Yahoo, and other ISPs, are going to start splitting IP reputation from content reputation. IP reputation is critical for getting mail in the door, and without a good IP reputation you’ll see slow delivery. But once the mail has been accepted, there’s a whole other level of filtering, most of it on the content and generally unaffected by the IP reputation.
I don’t think the changes are going to go away any time soon. I think they may be refined, but I do think that reputation on email content (particularly domains and URLs and target IP addresses) is going to play a bigger and bigger role in email delivery.
What, specifically, is going to happen at Yahoo? Only they can tell you and I’m not sure I have enough of a feel for the pattern to speculate about the future. I do think that it’s going to take a few weeks for things to settle down and be consistent enough that we can start to poke the black box and map how it works.

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays. By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list. If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list. Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list. RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list. You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,