Email without filters

… or Find the False Positive.
Anyone sending a lot of email has complained about spam filters and false positives at some point. But most people haven’t run a mailbox with no spam filters in front of it in recent years, so don’t have much of a feel for what an unfiltered mailbox looks like, how important filters are and how difficult their job is.
I run no transaction level filters in front of my mailbox, just content filters that route mail to one of several inboxes or a junk folder, so if I want to I can look at what unfiltered email looks like. I took data from all mail that was sent to me yesterday, and put it in a format that really shows the problem filters face and especially the difficulty of spotting which mail in the junk folder is a false positive.
An inbox with no filters looks like this.

Running a spam filter against it, simply categorizing each email as spam (pink) or not-spam (green) looks like this.
 

Even with the messages categorized as spam vs not-spam it’s hard to work out which messages are important and which aren’t, let alone where the false positives might be.
If I sort the categories by hand you get this – where you can see that out of 1200 or so mails about three quarters were spam. Of the three false positives two were bulk email that I didn’t care that I didn’t receive and only one was email that I considered important.
 
 

Related Posts

Comments on Holomaxx post

I’m putting together a longer analysis of the Holomaxx case that will look at the claims against the various defendants. There’s some deep mis-understanding of how various things works (hint: wiretapping? not so much).
There was one comment from “The Other Barry” about complaints that I think bears highlighting.

Read More

Blocking of ESPs

There’s been quite a bit of discussion on my post about upcoming changes that ESPs will be facing in the future. One thing some people read into the post is the idea that ISPs will be blocking ESPs wholesale without any regard for the quality of the mail from that company.
The idea that ESPs are at risk for blocking simply because they are ESPs has been floating around the industry based on comments by an employee at a spam filter vendor at a recent industry conference.
I talked to the company to get some clarification on what that spam filtering company is doing and hopefully to calm some of the concerns that people have.
First off, and probably most important, is that the spam filtering company in question primarily targets their service to enterprises. Filtering is an important part of this service, but it also handles email archiving, URL filtering and employee monitoring. The target market for the company is very different than the ISP market.
The ISPs are not talking about blocking indiscriminately, they are talking about blocking based on bad behavior.
Secondly, this option was driven by customer request. The customers of the spam filtering appliance were complaining about “legitimate” mail from various ESPs. Despite being reasonable targeted the mail was unrequested by the recipient. While ESPs use FBLs and other sources of complaints to clean complainers off rented or epended lists at ISPs, the option is not available for mail sent to corporations. Enterprises don’t, nor should they have to, create and support FBLs. Nor should employees be expected to unsubscribe from mail they never requested.
This option is the direct result of ESPs allowing customers to send spam.
Thirdly, this option is offered to those customers who ask for it. It is not done automatically for everyone. The option is also configurable down to the end user.
While I haven’t seen the options, nor which ESPs are affected, I expect that the ones on the list are the ones that the filtering vendor receives complaints about. If you are not allowing your customers to send spam, and are stopping them from buying lists or epending, then you probably have not come to the attention of the filtering company and are not on the list of ESPs to block.

Read More

Change is required

I get a lot of calls from senders who tell me that they have not changed what they were doing, but all of a sudden their mail isn’t performing the way it used to. Sometimes it’s simply less effective marketing, but more often than not the issue is mail being blocked or filtered to the bulk folder.
What worked today won’t work tomorrow. Spammers are forever evolving new techniques to get past spam filters. ISPs are forever evolving new techniques to stop them.
One of the current driving forces for spam filter development is focused on the individual recipients. Recipient wants and needs are king in the world of ISP mail filtering. Much of that is driven by the underlying business models of the free ISPs. They are selling eyeballs to their advertisers and that relies on keeping as many eyeballs around for as long as possible.
An early version of the recipient driven filtering was “add to your address book” where individual users could over ride ISP delivery decisions by actively adding a From: address to their address book. The ISPs have been refining this over time. For instance, if you reply to an email in some clients, you are prompted to add that address to your address books. If you take an email out of your bulk folder and move it to your inbox then that address is automatically added to your address book.
But the refinements haven’t stopped there. ISPs are now making smart decisions about what emails a particular recipient will want to receive. This raises a number of challenges to senders. How do you send email to ten thousand or a hundred thousand or a million people and make it relevant to all of them?
Smart senders will take the individual delivery challenge in stride. They will change along with the ISPs, to send mail that their recipients want to receive. Change is inevitable and required.

Read More