Email filters

What makes the best email filter? There isn’t really a single answer to that question. Different people and different organizations have different tolerances for how false positives versus false negatives. For instance, we’re quite sensitive to false positives here, so we run extremely conservative filtering and don’t block very much at the MTA level. Other people I know are very sensitive to false negatives and run more aggressive filtering and block quite a bit of mail at the MTA level.
For the major ISPs, the people who plan, approve, design and monitor the filters usually want to maximize customer happiness. They want to deliver as much real mail as possible while blocking as much bad mail. Blocking real mail and letting through bad mail both result in unhappy customers and increase the ISP’s costs, either through customer churn or through support calls. And this is a process, filters are not static. ISPs roll out new filters all the time, sometimes they are an improvement and sometimes they’re not. When they’re not, they’re pulled out of production. This works both for positive filters like Return Path and negative filters like blocklists.
Then there is mail filtering that doesn’t have to do with spam. Business filters, for instance, often block non-business mail. Permission of the recipient often isn’t even a factor. Companies don’t often go out of their way to block personal mail, but if personal mail gets blocked (say the vacation plane ticket or the amazon receipt) they don’t often unblock it. But when you think about why a business provides email, it makes perfect sense. The business provides email to further its own business goals. Some personal usage is usually OK, but if someone notices and blocks personal email then it’s unlikely the business will unblock it, even if the employee opted in.
In the case of email filters, the free market does work. Different ISPs filter mail differently. Some people love Gmail’s filters. Other people think Hotmail has the best filtering. There are different standards for filtering, and that makes email stronger and more robust. Consumers have choices in their mail provider and spamfiltering.

Related Posts

Turn it all the way up to 11

I made that joke the other night and most of the folks who heard it didn’t get the reference. It made me feel just a little bit old.
Anyhow, Mickey beat me to it and posted much of what I was going to say about Ken Magill’s response to a very small quote from Neil’s guest post on expiring email headers last week.
I, too, was at that meeting, and at many other meetings where marketers and the folks that run the ISP spam filters end up in the same room. I don’t think the marketers always understand what is happening inside the postmaster and filtering desks on a day to day basis at the ISPs. Legitimate marketing? It’s a small fraction of the mail they deal with. Ken claims that marketing pays the salaries of these employees and they’d be out of a job if marketing didn’t exist. Possibly, but only in the context that they are paid to keep their employers servers up and running so that the giant promises made by the marketing team of faster downloads and better online experiences actually happen.
If there wasn’t an internet and there weren’t servers to maintain, they’d have good jobs elsewhere. They’d be building trains or designing buildings or any of the thousands of other jobs that require smart technical people.
Ken has no idea what these folks running the filters and keeping your email alive deal with on a regular basis. They deal with the utter dregs and horrors of society. They are the people dealing with unrelenting spam and virus and phishing attacks bad enough to threaten to take down their networks and the networks of everyone else. They also end up dealing with law enforcement to deal with criminals. Some of what they do is deal with is unspeakable, abuse and mistreatment of children and animals. These are the folks who stand in front of the rest of us, and make the world better for all of us.
They should be thanked for doing their job, not chastised because they’re doing what the people who pay them expect them to be doing.
Yes, recipients want the mail they want. But, y’know, I bet they really don’t want all the bad stuff that the ISPs protect against. Ken took offense at a statement that he really shouldn’t have. ISPs do check their false positive rates on filtering, and those rates are generally less than 1% of all the email that they filter. Marketers should be glad they’re such a small part of the problem. They really don’t want to be a bigger part.

Read More

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Read More

We're gonna party like it's 1996!

Over on deliverability.com Dela Quist has a long blog post up talking about how changes to Hotmail and Gmail’s priority inbox are a class action suit waiting to happen.
All I can say is that it’s all been tried before. Cyberpromotions v. AOL started the ball rolling when they tried to use the First Amendment to force AOL to accept their unsolicited email. The courts said No.
Time goes on and things change. No one argues Sanford wasn’t spamming, he even admitted as much in his court documents. He was attempting to force AOL to accept his unsolicited commercial email for their users. Dela’s arguments center around solicited mail, though.
Do I really think that minor difference in terminology going to change things?
No.
First off “solicited” has a very squishy meaning when looking at any company, particularly large national brands. “We bought a list” and “This person made a purchase from us” are more common than any email marketer wants to admit to. Buying, selling and assuming permission are par for the course in the “legitimate” email marketing world. Just because the marketer tells me that I solicited their email does not actually mean I solicited their email.
Secondly, email marketers don’t get to dictate what recipients do and do not want. Do ISPs occasionally make boneheaded filtering decisions? I’d be a fool to say no. But more often than not when an ISP blocks your mail or filters it into the bulk folder they are doing it because the recipients don’t want that mail and don’t care that it’s in the bulk folder. Sorry, much of the incredibly important marketing mail isn’t actually that important to the recipient.
Dela mentions things like bank statements and bills. Does he really think that recipients are too stupid to add the from address to their address books? Or create specific filters so they can get the mail they want? People do this regularly and if they really want mail they have the tools, provided by the ISP, to make the mail they want get to where they want it.
Finally, there is this little law that protects ISPs. 47 USC 230 states:

Read More