Email without filters

… or Find the False Positive.
Anyone sending a lot of email has complained about spam filters and false positives at some point. But most people haven’t run a mailbox with no spam filters in front of it in recent years, so don’t have much of a feel for what an unfiltered mailbox looks like, how important filters are and how difficult their job is.
I run no transaction level filters in front of my mailbox, just content filters that route mail to one of several inboxes or a junk folder, so if I want to I can look at what unfiltered email looks like. I took data from all mail that was sent to me yesterday, and put it in a format that really shows the problem filters face and especially the difficulty of spotting which mail in the junk folder is a false positive.
An inbox with no filters looks like this.

Running a spam filter against it, simply categorizing each email as spam (pink) or not-spam (green) looks like this.
 

Even with the messages categorized as spam vs not-spam it’s hard to work out which messages are important and which aren’t, let alone where the false positives might be.
If I sort the categories by hand you get this – where you can see that out of 1200 or so mails about three quarters were spam. Of the three false positives two were bulk email that I didn’t care that I didn’t receive and only one was email that I considered important.
 
 

Industry Jargon
SORBS Progress

Related Posts

Controlling delivery

How much control over delivery do senders have? I have repeatedly said that senders control their delivery. This is mostly true. Senders control their side of the delivery chain, but there is a point where the recipient takes over and controls things.
As a recipient I can

Read More

Email marketing ulcers for the holiday

I’ve mentioned here before that I can usually tell when the big ISPs are making changes to their spam filtering as that ISP dominates my discussions with current and potential clients and many discussions on delivery mailing lists.
The last two weeks the culprit has been Yahoo. They seem to be making a lot of changes to their filtering schemes right at the busiest email marketing time of the year. Senders are increasing their volume trying to extract that last little bit of cash out of holiday shoppers, but they’re seeing unpredictable delivery results. What worked to get mail into the inbox a month ago isn’t working, or isn’t working as well, now.
Some of this could be holiday volume related. Many marketers have drastically increased their mail volume over the last few weeks. But I don’t think the whole issue is simply that there is more email marketing flowing into our mailboxes.
As I’ve been talking with folks, I have started to see a pattern and have some ideas of what may be happening. It seems a lot of the issue revolves around bulk foldering. Getting mail accepted by the MXs seems to be no different than it has been. The change seems to be based on the reputation of the URLs and domains in the email.
Have a domain with a poor reputation? Bulk. Have a URL seen in mail people aren’t interested in? Bulk. Have a URL pointing to a website with problematic content? Bulk.
In the past IPs that were whitelisted or had very good reputations could improve delivery of email with neutral or even borderline poor reputations. It seems that is no longer an effect senders can rely on. It may even be that Yahoo, and other ISPs, are going to start splitting IP reputation from content reputation. IP reputation is critical for getting mail in the door, and without a good IP reputation you’ll see slow delivery. But once the mail has been accepted, there’s a whole other level of filtering, most of it on the content and generally unaffected by the IP reputation.
I don’t think the changes are going to go away any time soon. I think they may be refined, but I do think that reputation on email content (particularly domains and URLs and target IP addresses) is going to play a bigger and bigger role in email delivery.
What, specifically, is going to happen at Yahoo? Only they can tell you and I’m not sure I have enough of a feel for the pattern to speculate about the future. I do think that it’s going to take a few weeks for things to settle down and be consistent enough that we can start to poke the black box and map how it works.

Read More

Change is required

I get a lot of calls from senders who tell me that they have not changed what they were doing, but all of a sudden their mail isn’t performing the way it used to. Sometimes it’s simply less effective marketing, but more often than not the issue is mail being blocked or filtered to the bulk folder.
What worked today won’t work tomorrow. Spammers are forever evolving new techniques to get past spam filters. ISPs are forever evolving new techniques to stop them.
One of the current driving forces for spam filter development is focused on the individual recipients. Recipient wants and needs are king in the world of ISP mail filtering. Much of that is driven by the underlying business models of the free ISPs. They are selling eyeballs to their advertisers and that relies on keeping as many eyeballs around for as long as possible.
An early version of the recipient driven filtering was “add to your address book” where individual users could over ride ISP delivery decisions by actively adding a From: address to their address book. The ISPs have been refining this over time. For instance, if you reply to an email in some clients, you are prompted to add that address to your address books. If you take an email out of your bulk folder and move it to your inbox then that address is automatically added to your address book.
But the refinements haven’t stopped there. ISPs are now making smart decisions about what emails a particular recipient will want to receive. This raises a number of challenges to senders. How do you send email to ten thousand or a hundred thousand or a million people and make it relevant to all of them?
Smart senders will take the individual delivery challenge in stride. They will change along with the ISPs, to send mail that their recipients want to receive. Change is inevitable and required.

Read More