Insight into Gmail filtering

Last week I posted a link to an article discussing how Gmail builds defenses to protect their users from malicious mail. One of the things I found very interesting in that article was the discussion about how Gmail deploys many changes at once, to prevent people from figuring out what the change was.
Let’s take a look at what Gmail said.

Make it hard for attackers to understand your defenses – Use overwhelming force and deploy many countermeasures at once
This is probably the most subtle of the lessons. Attackers constantly probe systems to find loopholes. For example, at some point one of Gmail’s spammers became very astute at finding bugs in our parsers and started to find very subtle bugs he could exploit. For example, he realized he could use the @ ambiguity (it is used in email addresses and in http links) to confuse our parsers and for a brief period of time he successfully evaded detection. This is why it is very important to make probing more difficult for attackers by rolling out multiple changes. That way they are overwhelmed by the number of things to test and can’t easily figure out what changed.
Bottom line: When rolling out change in your defenses, don’t rush (too much) and release multiple changes at once.

I cannot tell you the number of people who have approached me – in person at conferences, on twitter, through email, on LinkedIn – asking if I knew, “What gmail changed this week.” Now, at least, I have an answer. “Gmail changes a lot of things at once in order to stop people from figuring out the filters.”
I’ll be honest, I stopped trying to probe Gmail’s filters to identify ways around them a couple years ago. They are just too hard to evaluate. Sure, I can identify certain things to change that will get email into the inbox, briefly. But unless the underlying issues were fixed, the filters catch up and the mail will go back to the bulk folder. Sometimes it takes the filters days to catch up, sometimes it can take hours.
In any case, probing the filters to see what they’re doing is a very short term, limited fix to Gmail problems.
What I’ve focused on, with my clients, is getting the filters to work for them. We know that modern filters don’t treat all mail from a single company, IP or domain equally. Instead they make delivery decisions for each individual recipient of that mail. Those of you who have seen some of my talks may have seen this image before.
IndividualRecipPreferences
Things like IP reputation, domain reputation, content reputation and link reputation all contribute to the reputation of an email. If the reputation is very bad, the mail is bounced and no body receives it. But if the mail isn’t bounced, then they go through the individual recipient preferences. It is the combination of individual preferences and email reputation that determine where the mail ends up for each recipient. Different recipients may get mail differently.
This is why engagement is so important in email. Sending to people who want to receive the mail improves overall inbox delivery. If most of your recipients want your mail than chances are if you mail someone new, they’ll want your mail, too.
Gmail has a goal with their email delivery. You can make filters work for you by sending mail that users want and engage with. If you’re having problems with Gmail delivery focus on the recipients and making them happy. Don’t waste time trying to troubleshoot a filter change. Gmail isn’t going to make it easy for you.
 
 

Related Posts

Spamhaus rising?

Ken has a good article talking about how many ESPs have tightened their standards recently and are really hounding their customers to stop sending mail recipients don’t want and don’t like. Ken credits much of this change to Spamhaus and their new tools.

Read More

More from Gmail

Campaign Monitor has an interview with Gmail looking at how to get mail to the Gmail inbox. It’s a great article and I think everyone should go read it.
One of the most important things it talks about is how complex filters are.

Read More

One letter off…

I’m working on a blog post about the new Gmail tabbed inbox and the messages Gmail is inserting into the promotions tab. The messages aren’t showing up on most of my accounts, so I logged into an infrequently used account of mine. Ads are there, I got my screenshots and some data about the behaviour of the messages. So far so good.
I also discovered that at least two other women are using my address. One of them apparently ordered a bunch of wedding stuff from David’s Bridal shop using my email address. I hope Kirstie got her special order in time.
The other case is more interesting. I found dozens of emails in my inbox from what appeared to be friends including me in their email forward chain.
The Comic Sans. The FW:FW:FW:FW:FW subject lines. The horribly drawn cartoons. The inspirational messages. The prayer requests. The invites to bridge night. The followup demands that I reply to their invites for bridge night. The sad emails that I didn’t go to bridge night. There were emails from grandchildren. Questions about where I’d been and if I moved. Prayer chains. The messages go on and on.
Looking back through my inbox, this has been going on since sometime late in 2012. (Told you this was an infrequently used account). I looked and looked and I think I figured out what happened. A woman named Helen appears to to have an email address one letter off from mine (string@ vs stringsstring@) and one of her church friends tried to reply to her and dropped the ‘s’ from the email address. Once she did that, everyone else just kept hitting “reply all” and are including me in their forward chain.
It’s not commercial, it’s not spam. It’s just a bunch of people mistyping an email address and sending mail to someone they don’t know. I’m kinda glad it was a bunch of church ladies rather than Carlos Danger sending … well… Carlos Danger type messages.
People get email addresses wrong sometimes. It happens (ask me about the time I almost got my mailserver blocked because I mistyped an address while sending mail to a blocklist maintainer and hit a trap address by mistake…). The problem is that it can overwhelm an uninvolved person’s mailbox, even when it’s not commercial. Sure, if I was logging in to this account more often I’d probably have shut it down, but if they were paying attention they would have realized Helen is never replying to anything they send.
I kinda feel the same about commercial mailers that send me mail over and over and over again. I never open it, I never reply to it, I never respond to it. I wonder if there is actually anyone actually sending the mail, or if there’s just a lonely mailserver bricked up in a wall somewhere continually sending out spam.
Don’t be the bricked up server in the wall. Pay attention to what your recipients are doing.

Read More