Censoring email

It seems some mail to Apple’s iCloud has been caught in filters. Apparently, a few months ago someone sent a script to a iCloud user that contained the phrase “barely legal teen” and Apple’s filters ate it.
The amount of hysteria that I’ve seen in some places about this, though, seems excessive. One of my favorite quotes was from MacWorld and just tells me that many of the people reporting on filtering have no idea how filters really work.

And it’s not as if there’s a lack of good, free email providers with years of spam-blocking experience: Google, Yahoo, and Microsoft all spring immediately to mind. And—as far as we know, anyway—those services aren’t “helpfully” blocking any emails without telling their users.

“As far as [you] know” isn’t very far, actually. These services block email all the time and normally don’t tell users about it. Hotmail is notorious for accepting email and then just silently dropping it on the floor. Yahoo doesn’t usually drop mail after it’s been accepted, but is very picky about what mail it accepts. About the only company mentioned that accepts everything is Gmail. And even then I know Gmail does, very rarely, block at the IP level.
Filters are complex and filters are extensive. I hate it when filters are responsible for losing legitimate mail but it happens. I’m pretty sure, though, that outside of the testing for the phrase “barely legal teen” that this is a filter phrase that has an extremely low false positive rate.
That’s the crux of what’s useful in filters: how much bad mail does this stop while letting as much good mail as possible through. If a particular filter catches lots of spam, and blocks only a tiny bit of real mail, it will be a useful filter. If it doesn’t catch much spam but also doesn’t block much real mail, it might be a useful filter. If it catches too much real mail, it’s not a useful filter.
As it is, Apple and their filtering vendor have adjusted their filters such that mail with the phrase “barely legal teen” is again making it into the inbox.
I’m not really sure this is a win.

Related Posts

Amendment is futile, part 2

When Yahoo filed for dismissal of the Holomaxx complaint, they ended the motion with “Amendment would be futile in this case.” The judge granted Yahoo’s motion but did grant Holomaxx leave to amend. Holomaxx filed an amended complaint earlier this month.
The judge referenced a couple specific deficiencies of Holomaxx’s claims in his dismissal.

Read More

I'm on a blocklist! HELP!

Recently, an abuse desk rep asked what to do when customers were complaining about being assigned an IP address located on a blocklist. Because not every blocklist actually affects mail delivery it’s helpful to identify if the listing is causing a problem before diving in and trying to resolve the issue.

Read More

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Read More