Spamfilters

Mentally modelling filters

When we talk about filters, we often think there is one filter. But, in many cases there are multiple stages of filters, each examining mail in a different way.

I use tom@hotmail.com as my default bogus email address. Tom has subscribed to so many things because of me.
Read More

The death of IP based reputation

Back in the dark ages of email delivery the only thing that really mattered to get your email into the inbox was having a good IP reputation. If your IP sent good mail most of the time, then that mail got into the inbox and all was well with the world. All that mattered was that good IP reputation. Even better for the people who wanted to game the system and get their spam into the inbox, there were many ways to get around IP reputation.
Every time the ISPs and spam filtering companies would work out a way to block spam using IP addresses, spammers would figure out a way around the problem. ISPs started blocking IPs so spammers moved to open relays. Filters started blocking open relays, so spammers moved to open proxies. Filters started blocking mail open proxies so spammers created botnets. Filters started blocking botnets, so spammers started stealing IP reputation by compromising ESP and ISP user accounts. Filters were constantly playing catchup with the next new method of getting a good IP reputation, while still sending spam.
While spammers were adapting and subverting IP based filtering a number of other things were happening. Many smart people in the email space were looking at improving authentication technology. SPF was the beginning, but problems with SPF led to Domains Keys and DKIM. Now we’re even seeing protocols (DMARC) layered on top of DKIM. Additionally, the price of data storage and processing got cheaper and data mining software got better.
The improvement in processing power, data mining and data storage made it actually feasible for ISPs and filtering companies to analyze content at standard email delivery speeds. Since all IPv4 addresses are now allocated, most companies are planning for mail services to migrate to IPv6. There are too many IPv6 IPss to rely on IP reputation for delivery decisions.
What this means is that in the modern email filtering system, IPs are only a portion of the information filters look at when making delivery decisions. Now, filters look at the overall content of the email, including images and URLs. Many filters are even following URLs to confirm the landing pages aren’t hosting malicious software, or isn’t content that’s been blocked before. Some filters are looking at DNS entries like nameservers and seeing if those nameservers are associated with bad mail. That’s even before we get to the user feedback, in the form of “this is spam” or “this is not spam” clicks, which now seem to affect both content, domain and IP reputation.
I don’t expect IP reputation to become a complete non-issue. I think it’s still valuable data for ISPs and filters to evaluate as part of the delivery decision process. That being said, IP reputation is so much less a guiding factor in good email delivery than it was 3 or 4 years ago. Just having an IP with a great reputation is not sufficient for inbox delivery. You have to have a good IP reputation and good content and good URLs.
Anyone who wants good email delivery should consider their IP reputation, but only as one piece of the delivery strategy. Focusing on a great IP reputation will not guarantee good inbox delivery. Look at the whole program, not just a small part of it.

Now, in a new paper in the Journal of Economic Perspectives, Justin Rao of Microsoft and David Reiley of Google (who met working at Yahoo) have teamed up to estimate the cost of spam to society relative to its worldwide revenues. The societal price tag comes to $20 billion. The revenue? A mere $200 million. As they note, that means that the “‘externality ratio’ of external costs to internal benefits for spam is around 100:1. Spammers are dumping a lot on society and reaping fairly little in return.” In case it’s not clear, this is a suboptimal situation. The Atlantic
Read More

Gmail and the bulk folder

Earlier this week Gmail announced they were providing reasons for why they delivered a particular mail to the bulk folder. I’m sure a lot of senders are rejoicing over the clear feedback. After all this is exactly what they’ve been asking for “tell us why you’re filtering our mail and we’ll fix it.”
I am not sure, however, that this is going to help the majority of senders seeing mail going to the bulk folder. On the Gmail support pages, they list a number of the explanations they’re be providing.

Content, trigger words and subject lines

There’s been quite a bit of traffic on twitter this afternoon about a recent blog post by Hubspot identifying trigger words senders should avoid in an email subject line. A number of email experts are assuring the world that content doesn’t matter and are arguing on twitter and in the post comments that no one will block an email because those words are in the subject line.
As usually, I think everyone else is a little bit right and a little bit wrong.
The words and phrases posted by Hubspot are pulled out of the Spamassassin rule set. Using those words or exact phrases will cause a spam score to go up, sometimes by a little (0.5 points) and sometimes by a lot (3+ points). Most spamassassin installations consider anything with more than 5 points to be spam so a 3 point score for a subject line may cause mail to be filtered.
The folks who are outraged at the blog post, though, don’t seem to have read the article very closely. Hubspot doesn’t actually say that using trigger words will get mail blocked. What they say is a lot more reasonable than that.

Email fingerprinting

I’ve had a lot of people ask me about what I mean by email fingerprinting. It means that I’m able to tell these 4 emails were all sent by the same entity. The domains are (mostly) different, the To: addresses are different, but I get hundreds of these emails a day.

Poor delivery can't be fixed with technical perfection

There are a number of different things delivery experts can do help senders improve their own delivery. Yes, I said it: senders are responsible for their delivery. ESPs, delivery consultants and deliverability experts can’t fix delivery for senders, they can only advise.
In my own work with clients, I usually start with making sure all the technical issues are correct. As almost all spam filtering is score based, and the minor scores given to things like broken authentication and header issues and formatting issues can make the difference between an email that lands in the inbox and one that doesn’t get delivered.
I don’t think I’m alone in this approach, as many of my clients come to me for help with their technical settings. In some cases, though, fixing the technical problems doesn’t fix the delivery issues. No matter how much my clients tweak their settings and attempt to avoid spamfilters by avoiding FREE!! in the subject line, or changing the background, they still can’t get mail in the inbox.
Why not? Because they’re sending mail that the recipients don’t really want, for whatever reason. There are so many ways a sender can collect an email address without actually collecting consent to send mail to that recipient. Many of the “list building” strategies mentioned by a number of experts involve getting a fig leaf of permission from recipients without actually having the recipient agree to receive mail.
Is there really any difference in permission between purchasing a list of “qualified leads” and automatically adding anyone who makes a purchase at a website to marketing lists? From the recipient’s perspective they’re still getting mail they don’t want, and all the technical perfection in the world can’t overcome the negative reputation associated with spamming.
The secret to inbox delivery: don’t send mail that looks like spam. That includes not sending mail to people who have not expressly consented to receive mail.