Filtering

Misinformation on filters

I’ve seen reports that someone is asserting that utm=COVID19 in URLs results in all mail going to bulk at multiple ISPs. This is the type of thing that someone says is true and dozens of folks believe it and thus a “deliverability phact” is born. For a plethora of reasons, this doesn’t pass the sniff test. Don’t believe everything you read on the internet.

The variables are not independent

In my previous career I was a molecular biologist. Much of my work was done on bacteria but after I left grad school, I ended up working in a developmental biology lab. Bacteria were (mostly) simple: just about every trait was controlled by a single gene. We could study what that gene did by removing it from the bacteria or adding it to a well characterised bacteria.

Troubleshooting delivery problems

Everyone has their own way of troubleshooting problems. I thought I would list out the steps I take when I’m trying to troubleshoot them.

Spam is never timely nor relevant

One of the ongoing recommendations to improve deliverability is to send email that is timely and relevant to the recipient. The idea being that if you send mail a recipient wants, they’re more likely to interact with it in a way that signals to the mailbox provider that the message is wanted. The baseline for that, at least whenever I’ve talked about timely and relevant, is that the recipient asked for mail from you in the first place.

When you can’t get a response

I’ve seen a bunch of folks in different places looking for advice on what to do when they can’t get a response from a postmaster team, or a filtering company. I was all set to write yet another post about how silence is an answer. Digging through the archives, though, I see I’ve written about this twice already in the last 18 months.

Their network, their rules

Much of the equipment and wires that the internet runs on is privately owned, nor is it a public utility in the traditional sense. The owners of the property have a lot of leeway to do what they like with that property. Yes, there are standards, but the standards are about interoperability. They describe things you have to do in order to exchange traffic with other entities. They do not dictate internal policies or processes.

Rethinking public blocklists

Recently, a significant majority of discussions of email delivery problems mention that neither the IPs or domains in use are on any of the public blocklists. I was thinking about this recently and realised that, sometime in the past, I stopped using blocklists as a source of useful information about reputation.

Gmail suddenly puts mail in the bulk folder

One of the delivery challenges that regularly comes up in various delivery discussion spaces is the “Gmail suddenly put my mail in spam.” From my perspective, there is rarely a “suddenly” about Gmail’s decision making process.

Email filters and small sends

Have you heard about the Baader-Meinhoff effect?

The Baader-Meinhof effect, also known as frequency illusion, is the illusion in which a word, a name, or other thing that has recently come to one’s attention suddenly seems to appear with improbable frequency shortly afterwards (not to be confused with the recency illusion or selection bias). Baader–Meinhof effect at Wikipedia

There has to be an corollary for email. For instance, over the last week or so I’ve gotten an influx of questions about how to fix delivery for one to one email. Some have been from clients “Oh, while we’re at it… this happened.” Others have been from groups I’m associated with “I sent this message and it ended up in spam.”

Gmail, machine learning, filters

I’m sure by now readers have seen the article from Gmail “Spam does not bring us joy — ridding Gmail of 100 million more spam messages with TensorFlow.” If you haven’t seen it, go read it. It’s not often companies write about their filtering philosophy and what tools they’re using to manage incoming bad mail.

All filters are not equal

Many questions about delivery problems often assume that there is one standard email filter and the rules are the same across all of them. Unfortunately, this isn’t really the case.

What is spearphishing?

As I’m writing this, I’m watching Deputy Atty General Rod Rosenstein discuss the indictments of 12 Russian military officers for hacking activities during the 2016 election cycle. One of the methods used to gain access to systems was spearphishing.
I think most of us know what phishing is, sending lots of emails to a wide range of people in an attempt to collect some credentials. These credentials are usually passwords to bank or email accounts, but can also be things like amazon or other accounts.
Spearphishing is an attempt to collect credentials from a specific person. The net isn’t thrown wide, to collect any credentials, rather individuals are targeted and researched. These attacks are planned. The targets are carefully researched and observed. The emails are crafted specifically for that target. If one set of emails doesn’t work, then they try again.
In terms of email marketing and deliverability, phishing is something detectable by many anti-spam filters. They’re sent in bulk, and they all look similar or identical to the filters. Spearphising isn’t as simple to detect with standard tools. What many organizations have done is try and combat this with warnings in the client. Like this one from gmail:

Security is becoming a bigger and bigger part of email filtering. I expect that as filters start addressing security more, we’ll see increased warnings like the above.
What can senders do?

Back to the office!

I’m back in the office after a busy June. The 2 continent, 3 city tour was unexpectedly extended to a 4th city thus I was out most of last week as well.
What was I doing? We spent a week in Dublin, which is an awesome and amazing city and I love it a little bit more every time we visit. After Dublin I jetted off to Chicago, where I spoke at ActiveCampaign’s first user conference.
The talk I did for ActiveCampaign was about how we’re in the middle of a fundamental shift in how email is filtered, particularly at the consumer ISPs. In order reach the inbox. we need to think beyond IP or domain reputation. We need to stop thinking of filters as a way of sorting good mail from bad mail. I touched a little on these concepts in my What kind of mail do filters target? blog post.
The shift in filtering is changing how email reaches the inbox and what we can and should be monitoring. At the same time, the amount of data we can get back from the ISPs is decreasing. This means we’re looking at a situation when our primary delivery fixes can’t be based on feedback from the filters. This is, I think, going to be an ongoing theme of blog posts over the next few months.

The next trip was to spend 2 days onsite at a client’s office. These types of onsite training are intense but I do enjoy them. As this was mostly client specific, there isn’t much I can share. They did describe it as a masterclass in deliverability, so I think it was also intense for them.
That was the planned 2 continent, 3 city tour. The last city was a late addition of a more personal nature. We headed downstate to join my cousin and her family in saying goodbye to my uncle. He was an amazing man. A larger than life, literal hero (underwater EOD, awarded the silver star) whom I wish I had known better. Most of what I remember is how much he loved and adored my aunt.
I’ll be getting back into the swing of blogging over the next few days. It’s good to be back and not looking at traveling in the short term.

Updating the filtering model

One thing I really like about going to conferences is they’re often one of the few times I get to sit and think about the bigger email picture. Hearing other people talk about their marketing experiences, their email experiences, and their blocking experiences usually triggers big picture style thoughts.
Earlier this week I was at Activate18, hosted by Iterable. The sessions I attended were interesting and insightful. Of course, I went to the deliverability session. While listening to the presentation, I realized my previous model of email filtering needed to be updated.

AOL MX Change update

The AOL postmaster team posted some information about the upcoming MX transition on their blog.

Filtering by gestalt

One of those $5.00 words I learned in the lab was gestalt. We were studying fetal alcohol syndrome (FAS) and, at the time, there were no consistent measurements or numbers that would drive a diagnosis of FAS. Diagnosis was by gestalt – that is by the patient looking like someone who had FAS.
It’s a funny word to say, it’s a funny word to hear. But it’s a useful term to describe the future of spam filtering. And I think we need to get used to thinking about filtering acting on more than just the individual parts of an email.

Filtering is not just IP reputation or domain reputation. It’s about the whole message. It’s mail from this IP with this authentication containing these URLs. Earlier this year, I wrote an article about Gmail filtering. The quote demonstrates the sum of the parts, but I didn’t really call it out at the time.

Truth, myths and realities

For a long time it was a known fact that certain ISPs recycled abandoned addresses into spamtraps. There were long discussions by senders about this process and how it happened. Then at a conference a few years ago representatives of ISPs got up and announced that they do not recycle addresses. This led to quite a bit of consternation about how deliverability folks were making things up and were untrustworthy and deceptive.

In the early 2000’s ISPs were throwing a lot of things at the wall to deal with mail streams that were 80 – 90% bulk. They tried many different things to try and tame volumes that were overwhelming infrastructure. ISPs did try recycled traps. I know, absolutely know, two did. I am very sure that others did, too, but don’t have specific memories of talking to specific people about it.
At that time, a lot of deliverability knowledge was shared through word of mouth. That turned into a bit of an oral history. The problem with oral history is that context and details get lost. We can use the story of the ISP that did/did not recycle traps as an example.
Deliverability folks talk about an ISP that recycles traps. They don’t mention how often it happens. Some folks make the assumption that this is an ongoing process. It’s not, but anyone who knows it’s not risks violating confidences if they correct it. Besides, if senders believe it’s an ongoing process maybe they’ll be better behaved. Eventually, the story becomes all ISPs recycle traps all the time. This is our “fact” that’s actually a myth.
Then an ISP employee goes to a conference an definitively states they don’t recycle traps. I believe he stated the truth as he knows it to be. That ISP moved on from recycled traps to other kinds of traps because there were better ways to monitor spam.
We were talking about this on one of the deliverability lists and I told another story.
[ISP] recycled addresses once – back when JD was there which must have been, oh, around 2005/6 or so. I heard this directly from JD. It wasn’t done again, but a whole bunch of people just assumed it was an ongoing thing. Since my knowledge was a private conversation between JD and me, I never felt comfortable sharing the information. Given the circumstances, I’ve decided it’s OK to start sharing that end of the story a little more freely.
No one set out to create a myth, it just happened. No one intended to mislead. But sometimes it happens.

Gmail filtering in a nutshell

Gmail’s approach to filtering; as described by one of the old timers. This person was dealing with network abuse back when I was still slinging DNA around as my job and just reading headers as a hobby.

It depends… no more

The two most hated words in deliverability. Many people ask general questions about deliverability and most experts, including myself, answer, “It depends.”
There are a lot of problems with this answer. The biggest problem is that it’s led to the impression that there are no real answers about deliverability. That because we can’t answer hypothetical questions we are really just making the answers up.

The reason we use “it depends” is because the minute details matter when it comes to deliverability. Wether or not something will hurt or help deliverability depends on the specific implementation. Who’s doing the sending? What is their authentication setup? What IP are they using? How were the addresses collected? What is their frequency? What MTA is used? Are they linking to outside sites? Are they linking to outside services? Where are images hosted? The relevant questions go on and on and on.
I am going to stop saying it depends when answering generic deliverability questions. Instead I will be using the phrase “details matter.” Details do matter. Details are everything. Details drive deliverability.
Details Matter
The importance of details is why many deliverability people hedge their answers. The details do matter.
I will do my best to stop answering It Depends to deliverability questions. Instead, I’ll be answering with question and pointing out the details matter.

Ask Laura: Should we allow tagged addresses?

Use the form…

A lot of senders get frustrated with the time it can take to get a response from some ISPs. It’s totally understandable, for a lot of companies delivery problems are all hands on deck level problems. They want them fixed and they want them fixed IMMEDIATELY. They want feedback that their issue is being addressed. They want to know someone at the ISP knows there is a problem.
I’ve talked before about visiting my friend Anna and watching her laptop screen explode with IMs from senders who wanted help with an AOL issue. She’s awesome and conscientious and tried to address all of those issues as fast as she could. She did want senders to feel like their issues were important and that someone inside AOL cared about the mail blocks.

I was always a strong advocate for following the official pathways for addressing problems. That was the whole point of the 2009 blog post. These days it’s easier to do than it ever was. Many ISPs have forms and process around handling delivery issues. This is good! In the past getting an answer to “why is my mail blocked” required knowing the right people. Now, it’s not about who you know. The ISPs and filtering companies who are open to senders have postmaster pages, unblock forms and official request channels. Those that don’t have those channels have made certain business decisions to not provide support for senders.
Despite the availability of webforms and knowledge bases and detailed information, a lot of people still think that the only way to get attention or get an issue addressed is to get someone on the phone. It’s not, though.
ISPs have their processes. If you want things handled quickly use those processes. Even in the places where very helpful reps are, they can’t (on order of lawyers and executives) help people unless there is a ticket already open.
Always, always use the recommended processes before trying to find “a real person.” Most of the time your issue can be solved faster if you fill out the form than if you hunt around for a person. In the worst case, all that time will be wasted as the person in question will tell you to fill out the form.

Google takes on intrusive interstitials

Starting next January, Google will be modifying its mobile search results to lower the ranking of sites that use interstitials that interfere with the users experience. In a blog post announcing the change they explain:

Things to read: March 9, 2016

It’s sometimes hard for me to keep up with what other people are saying and discussing about email marketing. I’ve been trying to be more active on LinkedIn, but there are just so many good marketing and delivery blogs out there I can’t keep up with all of them.

Here are a couple interesting things I’ve read in the last week.
Five Steps to Stay Out of the Spam Folder. Conceptually easy, sometimes hard to pull off in practice, these recommendations mirror many things I say here and tell my clients about delivery. The audience is in charge and your recipients are the best ally you can have when it comes to getting into the inbox.
Which states are the biggest sources of spam?. California and New York top the list, but the next two states are a little surprising. Over on Spamresource, Al points out the two next states have some unique laws that may affect the data. I just remember back in the day there were a lot of spammers in Michigan, I’m surprised there’s still a significant volume from there.
CASL didn’t destroy Canadian email. Despite concerns that CASL would destroy the Canadian email marketing industry, the industry is going strong and expanding. In fact, spending on email marketing in Canada was up more than 14% in 2015 and is on track to be up another 10% this year. Additionally, according to eMarketer lists are performing better because they’re cleaner.
A brief history of email. Part of the Guardian’s tribute to Ray Tomlinson, the person who sent the first email. Ray’s work literally changed lives. I know my life would be significantly different if there wasn’t email. Can you imagine trying to be a deliverability consultant without email? 🙂

Port25 blocking

A number of hosting providers are blocking outgoing port25. This has implications for a lot of smaller senders who either want to run their own mail server or who use SMTP to send mail to their ESP.

Trawling through the junk folder

As a break from writing unit tests this morning I took a few minutes to go through my Mail.app junk folder, looking for false positives for mail delivered over the past six weeks.
trashcans
We don’t do any connection level rejection here, so any mail sent to me gets delivered somewhere. Anything that looks like malware gets dumped in one folder and never read, anything that scores a ridiculously high spamassassin score gets dumped in another folder and never read, mailing lists get handled specially and everything else gets delivered to Mail.app to deal with. That means that Mail.app sees less of the ridiculously obvious spam and is mostly left to do bayesian filtering, and whatever other magic Apple implemented.
There were about thirty false positives, and they were all B2C bulk advertising mail. I receive a lot of 1:1 mail, transactional mail and B2B marketing mail and there were no false positives at all for any of those.
All the false positives were authenticated with both SPF and DKIM. All of them were for marketing lists I’d signed up for while making a purchase. All of them were “greymail” – mail that I’d agreed to receive, and that was inoffensive but not compelling. While I easily spotted all of them as false positives via the from address and subject, none of them were content I’d particularly missed.
Almost all of the false positives were sent through ESPs I recognized the name of, and about 80% of them were sent through just two ESPs (though that wasn’t immediately obvious, as one of them not only uses random four character domain names, it uses several different ones – stop doing that).
If you’d asked me to name two large, legitimate ESPs from whom I recalled receiving blatant, blatant spam recently, it would be those same two ESPs. Is Mail.app is picking up on my opinions of the mail those ESPs are sending? It’s possible – details specific to a particular ESPs mail composition and delivery pipelines are details that a bayesian learning filter may well recognize as efficient tokens.

Confusing the engineers

We went camping last weekend with a bunch of friends. Had a great time relaxing on the banks of the Tuolumne River, eating way too much and visiting.
On Saturday I was wearing a somewhat geeky t-shirt. It said 554: abort mission. (Thank you MessageSystems). At some point on Saturday every engineer came up to me, read my shirt and then looked at me and said “That’s not HTTP.”
That lead to various discussions about how their junior engineers don’t actually know SMTP at all. Why? Because the SMTP libraries just work. Apparently the HTTP libraries aren’t that great, so folks have to learn more about HTTP to troubleshoot and use them.
I’m sure there’s a joke in there somewhere: A Kindle engineer, an Android engineer and a robot engineer walk into a campsite…
It did leave me thinking, though, about how it’s not that easy to run your own mail server these days. Gone are the days when running your own server was cost effective and easy. These days, there is just too much spam coming in. Crafting filters is a skilled job. It’s not that hard to run good filters. But to run good filters takes time to do well.
There are also a lot of challenges to sending mail. One of the discussions I had at the campsite was how hard it was to configure outbound mail. The engineer was helping a friend set up a website and trying to get the website to send notifications to the friend. But without setting up authentication the mail kept silently failing.
Of course, we do run our own mail server. But it’s our job and, in many ways, it keeps us honest. We don’t run many filters meaning we see what spammers are doing and can use our own experiences to better understand what commercial filters are dealing with.
For most people, though, I really think using a service is the right solution. Find one with filters that meet your needs and just pay them to deal with the headache.

Do system administrators have too much power?

Yesterday, Laura brought a thread from last week to my attention, and the old-school ISP admin and mail geek in me felt the need to jump up and say something in response to Paul’s comment. My text here is all my own, and is based upon personal experience as well as those of my friends. That said, I’m not speaking on their behalf, either. 🙂
I found Paul’s use of the word ‘SysAdmin’ to be a mighty wide (and — in my experience — probably incorrect) brush to be painting with, particularly when referring to operations at ISPs with any significant number of mailboxes. My fundamental opposition to use of the term comes down to this: It’s no longer 1998.
The sort of rogue (or perhaps ‘maverick’) behavior to which you refer absolutely used to be a thing, back when a clean 56k dial-up connection was the stuff of dreams and any ISP that had gone through the trouble to figure out how to get past the 64k user limit in the UNIX password file was considered both large and technically competent. Outside of a few edge cases, I don’t know many system administrators these days who are able to (whether by policy or by access controls) — much less want to — make such unilateral deliverability decisions.
While specialization may be for insects, it’s also inevitable whenever a system grows past a certain point. When I started in the field, there were entire ISPs that were one-man shows (at least on the technical side). This simply doesn’t scale. Eventually, you start breaking things up into departments, then into services, then teams assigned to services, then parts of services assigned to teams, and back up the other side of the mountain, until you end up with a whole department whose job it is to run one component of one service.
For instance, let’s take inbound (just inbound) email. It’s not uncommon for a large ISP to have several technical teams responsible for the processing of mail being sent to their users:

Politics and Delivery

Last week I posted some deliverability advice for the DNC based on their acquisition of President Obama’s 2012 campaign database. Paul asked a question on that post that I think is worth some attention.

June 2015: the Month in Email

Happy July! We are back from another wonderful M3AAWG conference and enjoyed seeing many of you in Dublin. It’s always so great for us to connect with our friends, colleagues, and readers in person. I took a few notes on Michel van Eeten’s keynote on botnets, and congratulated our friend Rodney Joffe on winning the prestigious Mary Litynski Award.
In anti-spam news, June brought announcements of three ISP-initiated CAN-SPAM cases, as well as a significant fine leveled by the Canadian Radio-television and Telecommunications Commission (CRTC) against Porter Airlines. In other legal news, a UK case against Spamhaus has been settled, which continues the precedent we’ve observed that documenting a company’s practice of sending unsolicited email does not constitute libel.
In industry news, AOL started using Sender Score Certification, and Yahoo announced (and then implemented) a change to how they handle their Complaint Feedback Loop (CFL). Anyone have anything to report on how that’s working? We also noted that Google has discontinued the Google Apps for ISPs program, so we expect we might see some migration challenges along the way. I wrote a bit about some trends I’m seeing in how email programs are starting to use filtering technologies for email organization as well as fighting spam.
Steve, Josh and I all contributed some “best practices” posts this month on both technical issues and program management issues. Steve reminded us that what might seem like a universal celebration might not be a happy time for everyone, and marketers should consider more thoughtful strategies to respect that. I wrote a bit about privacy protection (and pointed to Al Iverson’s post on the topic), and Josh wrote about when senders should include a physical address, what PTR (or Reverse DNS) records are and how to use them, testing your opt-out process (do it regularly!), and advice on how to use images when many recipients view email with images blocked.

Filtering more than spam

The obvious application of machine learning for email is to send spam to the junk/bulk folder. Most services use some level of machine learning for filters. Places like Gmail have extensive machine learning filters to filter spam and unwanted mail away from their users.
Some organizations are taking the filtering process a step further. Almost every mail client more advanced than PINE has the ability for users to create rules to sort mail into folders. Late last year, Office 365 rolled out a feature, Clutter that tracks how a user interacts with mail and filters unimportant mail. This allows each user to have their own filters, but without the overhead of having to create the filters.
The Clutter engine looks at both how the user interacts with mail and things it knows about the organization. For example, if Exchange is tied into Active Directory, then mail from a manager will be prioritized while mail from a co-worker may end up in the clutter folder.
Email is a critical business tool. A significant number of companies rely on email for internal and external communication. Many users treat their inbox as a todo list, prioritizing what they work on based on what’s in their mail box. Despite the needs of users, the mail client hasn’t really changed.
Over the last few years, we’ve seen different online services attempt to build a more effective email client. Some of these features were things like tabs and priority inbox at Gmail. Microsoft created the “sweep” feature for Outlook/Hotmail users to manage inbox clutter. Third parties have created services to try and improve the mailbox experience for their users.
Many of the email filters, up to this point, have really been focused on protecting users from spam and malicious emails. Applying that filtering knowledge to more than just spam, but to the different kinds of emails makes sense to me. I’ve always had a fairly extensive set of filters, initially procmail but now sieve, to process and organize incoming mail. But I kinda like the idea that my mail client learns how I filter messages and do the right thing on its own.
I’d love to see some improvements in the mail client, that make it easier to manage and organize incoming email. It remains to be seen if this is a feature that takes off and makes its way to other clients or not.

Deliverability and IP addresses

Almost 2 years ago I wrote a blog post titled The Death of IP Based Reputation. These days I’m even more sure that IP based reputation is well and truly dead for legitimate senders.
There are a lot of reasons for this continued change.

January 2015 – The Month in Email

It’s February already! January went fast, right? At WttW, we are gearing up for MAAWG SF later this month — will we see you there?
We started the year with a set of predictions about email. Mostly we think email will continue to be great at some things and not-so-great at other things, and we’ll keep fighting the good fight to make it better.
As always, I’m interested in filters and how spammers continue to work around them to reach the inbox. I also wrote about how the language of an email impacts delivery, and wrote an expanded response to a comment suggesting email filters should be illegal. You can guess where I stand on that (and if you can’t, perhaps you might read more about how email is an inherently malicious traffic stream…)
I also took a moment to point out a trend I’m really enjoying, which is the rise of content marketing (a.k.a. giving customers useful and interesting information they can’t find elsewhere). As I said in the post, I’ll be curious to see how ROI plays out with this strategy.
We also talked about some of the less exciting content we see in email, notably the infamous Murkowski Statement, by which a spammer declares “Nope! Nothing to see over here!”
Steve also pointed out some content shenanigans in the form of hidden preview text, with some additional clarification from the original marketer in the comments.
In industry news, the big story was that Microsoft has partially implemented DMARC for Office365, and was the first to make a public statement about the specific ways they’ve chosen to implement. In my post, I did a walkthrough of a message to illustrate a bit about how this works, which might be useful if you’re trying to wrap your head around DMARC implementations.
We also talked about consolidation in the ESP space, and got a number of comments from readers about who they think might be next. Shortly thereafter, Listcast was acquired by MailerMailer.
Josh noted a few major shutdowns: Yahoo China email services and the AHBL list. The latter explores the challenges inherent in decommissioning a blacklist, and there’s a good discussion in the comments, so you might check it out if you missed that earlier this month.
Josh also pointed to the Salesforce State of Marketing report, which is always a useful set of metrics about how marketers are using email and other channels. It’s definitely worth a read.

Email filtering: not going away.

I don’t do a whole lot of filtering of comments here. There are a couple people who are moderated, but generally if the comments contribute to a discussion they get to be posted. I do get the occasional angry or incoherent comment. And sometimes I get a comment that is triggers me to write an entire blog post pointing out the problems with the comment.
Today a comment from Joe King showed up for The Myth of the Low Complaint Rate.

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays. By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list. If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list. Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list. RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list. You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,

URL reputation and shorteners

A bit of a throwback post from Steve a few years ago. The problem has gotten a little better as some shortening companies are actually disabling spammed URLs, and blocking URLs with problematic content. I still don’t recommend using a public URL shortener in email messages, though.
Any time you put a URL in mail you send out, you’re sharing the reputation of everyone who uses URLs with that hostname. So if other people send unwanted email that has the same URL in it that can cause your mail to be blocked or sent to the bulk folder.
That has a bunch of implications. If you run an affiliate programme where your affiliates use your URLs then spam sent by your affiliates can cause your (clean, opt-in, transactional) email to be treated as spam. If you send a newsletter with advertisers URLs in it then bad behaviour by other senders with the same advertisers can cause your email to be spam foldered. And, as we discussed yesterday, if spammers use the same URL shortener you do, that can cause your mail to be marked as spam.
Even if the hostname you use for your URLs is unique to you, if it resolves to the same IP address as a URL that’s being used in spam, that can cause delivery problems for you.
What does this mean when it comes to using URL shorteners (such as bit.ly, tinyurl.com, etc.) in email you send out? That depends on why you’re using those URL shorteners.
The URLs in the text/html parts of my message are big and ugly
Unless the URL you’re using is, itself, part of your brand identity then you really don’t need to make the URL in the HTML part of the message visible at all. Instead of using ‘<a href=”long_ugly_url”> long_ugly_url </a>’ or ‘<a href=”shortened_url”> shortened_url </a>’ use ‘<a href=”long_ugly_url”> friendly phrase </a>’.
(Whatever you do, don’t use ‘<a href=”long_ugly_url”> different_url </a>’, though – that leads to you falling foul of phishing filters).
The URLs in the text/plain parts of my message are big and ugly
The best solution is to fix your web application so that the URLs are smaller and prettier. That will make you seem less dated and clunky both when you send email, and when your users copy and paste links to your site via email or IM or twitter or whatever. “Cool” or “friendly” URLs are great for a lot of reasons, and this is just one. Tim Berners-Lee has some good thoughts on this, and AListApart has two good articles on how to implement them.
If you can’t do that, then using your own, branded URL shortener is the next best thing. Your domain is part of your brand – you don’t want to hide it.
I want to use a catchy URL shortener to enhance my brand
That’s quite a good reason. But if you’re doing that, you’re probably planning to use your own domain for your URL shortener (Google uses goo.gl, Word to the Wise use wttw.me, etc). That will avoid many of the problems with using a generic URL shortener, whether you implement it yourself or use a third party service to run it.
I want to hide the destination URL from recipients and spam filters
Then you’re probably spamming. Stop doing that.
I want to be able to track clicks on the link, using bit.ly’s neat click track reporting
Bit.ly does have pretty slick reporting. But it’s very weak compared to even the most basic clickthrough reporting an ESP offers. An ESP can tell you not just how many clicks you got on a link, but also which recipients clicked and how many clicks there were for all the links in a particular email or email campaign, and how that correlates with “opens” (however you define that).
So bit.ly’s tracking is great if you’re doing ad-hoc posts to twitter, but if you’re sending bulk email you (or your ESP) can do so much better.
I want people to have a short URL to share on twitter
Almost all twitter clients will abbreviate a URL using some URL shortener automatically if it’s long. Unless you’re planning on using your own branded URL shortener, using someone else’s will just hide your brand. It’s all probably going to get rewritten as t.co/UgLy in the tweet itself anyway.
If your ESP offers their own URL shortener, integrating into their reporting system for URLs in email or on twitter that’s great – they’ll be policing users of that just the same as users of their email service, so you’re unlikely to be sharing it with bad spammers for long enough to matter.
All the cool kids are using bit.ly, so I need to to look cool
This one I can’t help with. You’ll need to decide whether bit.ly links really look cool to your recipient demographic (Spoiler: probably not) and, if so, whether it’s worth the delivery problems they risk causing.
And, remember, your domain is part of your brand. If you’re hiding your domain, you’re hiding your branding.
So… I really do need a URL shortener. Now what?
It’s cheap and easy to register a domain for just your own use as a URL shortener. Simply by having your own domain, you avoid most of the problems. You can run a URL shortener yourself – there are a bunch of freely available packages to do it, or it’s only a few hours work for a developer to create from scratch.
Or you can use a third-party provider to run it for you. (Using a third-party provider does mean that you’re sharing the same IP address as other URL shorteners – but everyone you’re sharing with are probably people like you, running a private URL shortener, so the risk is much, much smaller than using a freely available public URL shortener service.)
These are fairly simple fixes for a problem that’s here today, and is going to get worse in the future.

The long tail of domains

I frequently get clients telling me that they have about 15 (20, 30) major domains on their list, and then a long tail of domains with only a couple of recipients. If you sort simply by the left hand side of the @, that’s true.
When you’re sending email, it’s not just the domain in the email address that is important. Of equal importance is the MX. The MX is what actually handles the mail and where many filters are applied. Sorting by MX, instead of simply recipient domain, can identify that most of your small business clients are hosted at a particular provider. The number of subscribers behind that filter may be enough to push that filter into your top 10 or even top 5 recipient domains.
There’s a much smaller tail when grouping recipients by MX domain. It makes it much easier to understand where blocks are happening. I have even seen cases where clients didn’t realize they were blocked at a commercial provider because they only saw the “onesie twosie” domains as undeliverable. They missed a real problem with blocking because they were looking at the wrong data.
I sometimes get the side eye from some ISP folks if I use the term receiver (because, well, they’re senders as much as they are receivers). But I use receiver to help distinguish between the recipient domain and the actual domain handling the email.
When was the last time you looked at your delivery by filter or MX rather than by recipient domain? What did you find?

Superstition, correlation and reality

I’m not a huge baseball fan, probably a side effect of growing up in a city with no MLB team. But I do enjoy the social aspects of rooting for local teams when they’re winning big games. Last night I was following the World Series score online and switched over to watch the last inning. I posted something about the game on FB just about 30 seconds before the Giant’s outfield bobbled what should have been a single (at best). I immediately posted an apology, “Sorry about that, shouldn’t have said anything!”
Do I really think that my post somehow cursed two outfielders and caused them to bobble a simple play? No, of course not. But it is a very human response. In fact, there’s an entire advertising campaign centered around the the weird things people do while watching sports.
There is a lot of superstition in email delivery, too. I think that’s a combination of filtering necessarily being a black box, human’s built in tendency to see patterns in random data, and a need to be able to control and affect outcomes.
Figuring out cause and effect in the real world is not trivial. In my research days we set out to control as many confounding factors as possible so we could demonstrate the cause and the effect. That’s really hard to do when you’re not at a lab bench. In the real world, we can’t always control things directly. Instead, we have to rely on statistics and representative (or non-representative) samples.
Delivery isn’t even close to a science and one of the major issues is that filters are always changing. I’ve certainly seen occasions where multiple clients, or colleagues, were having problems delivering to one ISP or another. One of my clients made a change and saw their delivery improve. They patted themselves on the back for figuring out the problem. At the same time, though, other folks saw their delivery improve without making any changes. I can’t always convince people that whatever they did had nothing to do with their delivery improving.
The flip side is I can’t always convince people to stop doing somethings that they don’t need to do. I see a lot of mail with both DomainKeys and DKIM signatures. In most cases both signatures have the same selectors. DomainKeys is deprecated. No one, and I mean no one with a modern email system, is checking DomainKeys without checking DKIM. Senders can safely stop signing with DomainKeys and have nothing happen. It doesn’t matter, lots of ESPs and sender sign with both. They’re not going to change it. I’ve had multiple groups tell me they’re afraid to stop signing because it might hurt their delivery.
The reality is I didn’t make the Giant’s outfield bobble the ball because I posted to FB that I was watching the bottom of the 9th inning. The reality is that DomainKeys is deprecated and there’s no benefit to signing with both DomainKeys and DKIM. The reality is we are humans and we are inherently superstitious. Most of the times our superstitions are harmless. But sometimes they cause us more work than we need to do and provide no tangible benefits.

Content based filtering

Content filtering is often hard to explain to people, and I’m not sure I’ve yet come up with a good way to explain it.
A lot of people think content reputation is about specific words in the message. The traditional content explanation is that words like “Free” or too many exclamation points in the subject line are bad and will be filtered. But it’s not the words that are the issue it’s that the words are often found in spam. These days filters are a lot smarter than to just look at individual words, they look at the overall context of the message.
ISP_tolerances
Even when we’re talking content filters, the content is just a way to identify mail that might cause problems. Those problems are evaluated the same way IP reputation is measured: complaints, engagement, bad addresses. But there’s a lot more to content filtering than just the engagement piece. What else is part of content evaluation?

Typo traps

People make all sorts of claims about typo traps. One claim that showed up recently was that Spamhaus has just started using typo traps. I asked my Facebook network when people started using typos to detect incoming spam.
Two different colleagues mentioned using typos, both on the left hand side and the right hand side, back in ’98 and ’99.
The point is, typo traps are absolutely nothing new. They are, in fact, as old as spam filtering itself. And as one of trap maintainers remind me, not all of them even look like typos. It’s not as simple as hotmial.com or gmial.com.
I really think that focusing on traps is paying attention to the wrong thing.
The traps are not the issue. The underlying issue is that people are signing up addresses that don’t belong to them. Sometimes those are addresses that are spamtraps. Sometimes those are simply addresses that belong to someone else. Those addresses don’t belong to customers, they belong to random people who may never have heard of the sender. Sending mail to those people is sending spam.
Just trying to remove traps from your address lists isn’t going to solve the underlying problem. Instead, focus on improving your data process to keep from sending mail to random strangers.

Don't wait to address delivery problems

One of the worst ways to deal with blocking issues is to ignore them and hope your mail magically moves from the bulk folder back into the inbox. While this does happen as ISPs and filter companies update their filters, it’s not that common and it’s usually the result of a sender actually cleaning up their sending processes and improving the quality of the mail they send.
Do not ignore blocks. What I generally tell people is that it takes at least as long to repair a bad reputation as it took to get that bad reputation in the first place. If you wait months before actually addressing delivery problems, you’re not going to make a change and have the filters react in hours.
This doesn’t mean that every block is a business crisis. Blocks happen and they do go up and down based on thresholds and automatic monitoring scripts and content. But if a block happens consistently for 4 or 5 days in a row it is time to look at what you’re doing. Don’t just focus on the sidelines and little stuff, either. Look at your marketing program and the mail you’re sending.

Incorrect rejection messages

At least one ESP and Spamhaus are currently investigating bounce messages at a couple ISPs incorrectly pointing to Spamhaus as the reason for the block. The bounce messages are taking the form:

Ever changing filtering

One of the ongoing challenges sending email, and managing a high volume outbound mail server is dealing with the ongoing changes in filtering. Filters are not static, nor can they be. As ISPs and filtering companies identify new ways to separate out wanted email from unwanted email, spammers find new ways to make their mail look more like wanted mail.
This is one reason traps are useful to filtering companies. With traps there is no discussion about whether or not the mail was requested. No one with any connection to the email address opted in to receive mail. The mail was never requested. While it is possible for trap addresses to get on any list monitoring mail to spam traps is a way to monitor which senders don’t have good practices.
New filtering techniques are always evolving. I mentioned yesterday that Gmail was making filtering changes, and that this was causing a lot of delivery issues for senders. The other major challenge for Gmail is the personalized delivery they are doing. It’s harder and harder for senders to monitor their inbox delivery because almost every inbox is different at Gmail. I’ve seen different delivery in some of my own mailboxes at Gmail.
All of this makes email delivery an ongoing challenge.

Role accounts, ESPs and commercial email

There was a discussion today on a marketing list about role accounts and marketing lists. Some ESPs block mail to role accounts, and the discussion was about why and if this is a good practice. In order to answer that question, we really need to understand role accounts a little more.

Filtering secret sauce

It seems one of the most asked questions I hear from people is about filters and what the secret sauce is.

SMTP Level Rejections

While discussing a draft of a Deliverability BCP document the issue came up of what rejections at different phases of the email delivery transaction can mean. That’s quite a big subject, but here’s a quick cheat sheet.
At initial connection
Dropped or failed connection:

IP reputation and email delivery

IP reputation is a measure of how much wanted mail a particular IP address sends. This wanted mail is measured as a portion of the total email sent from that IP. Initially IP reputation was really the be all and end all of reputation, there was no real good way to authenticate a domain or a from address. Many ISPs built complex IP reputation models to evaluate mail based on the IP that sent the mail.
These IP reputation models were the best we had, but there were a lot of ways for spammers to game the system. Some spammers would create lots of accounts at ISPs and use them to open and interact with mail. Other spammers would trickle their mail out over hundreds or thousands of IPs in the hopes of diluting the badness enough to get to the inbox. Through it all they kept trying to get mail out through reputable ESPs, either by posing as legitimate customers or compromising servers.
These things worked for a while, but the ISPs started looking harder at the recipient pool in order to figure out if the interactions were real or not. They started looking at the total amount of identical mail coming from multiple IP addresses. The ISPs couldn’t rely on IP reputation so they started to dig down and get into content based filtering.
As the ISPs got better at identifying content and filtering on factors other than source IP, the importance of the IP address on inbox delivery changed. No longer was it good enough to have a high reputation IP sending mail.
These days your IP reputation dictates how fast you can send mail to a particular ISP. But a high reputation IP isn’t sufficient to get all the mail in the inbox. It’s really content that drives the inbox / bulk folder decisions these days.

Generally IPs that the ISP has not seen email traffic from before start out with a slight negative reputation. This is because most new IPs are actually infected machines. The negative reputation translates to rate limiting. The rate limiting minimizes people getting spam while the ISP works out if this is a real sender or a spammer.
Some ISPs put mail in the inbox and bulk foldering during the whitelisting process. In this case what they’re doing is seeing if your recipients care enough about your mail to look for it in the bulk folder. If they do, and they mark the mail as “not spam” then this feeds back to the sender reputation and the IP reputation.
If you’re seeing a lot of bulk foldering of mail, it’s unlikely there’s anything IP reputation based to do. Instead of worrying about IP reputation, focus instead on the content of the mail and see what you may need to do to improve the reputation of the domains and URLs (or landing pages) in the emails.

Thoughts on Gmail and the inbox

Over the last few months more and more marketers are finding their primary delivery challenge is the Gmail inbox. I’ve been thinking about why Gmail might be such a challenge for marketers. Certainly I have gotten a lot of calls from people struggling to figure out how to get into the Gmail inbox. I’ve also seen aggressive domain based filtering from Gmail, where any mention of a particular domain results in mail going to the bulk folder.
It’s one of those things that’s a challenge, because in most of these cases there isn’t one cause for bulk foldering. Instead there’s a whole host of things that are individually very small but taken together convince Gmail that the mail doesn’t need to be in the inbox.
A pattern that I’m starting to see is that Gmail is taking a more holistic look at all the mail from a sender. If the mail is connected to an organization, all that mail is measured as part of their delivery decision making. This is hurting some ESPs and bulk senders. I’ve had multiple ESPs contact me in the last 6 months looking for help because all their customer emails are going to bulk folder.
Gmail’s filtering is extremely aggressive. From my perspective it always has been. I did get an invite for a Gmail account way back in the day. I moved a couple mailing lists over to that account to test it with some volume and discussion lists. I gave up not long after because no matter what I did I couldn’t get gmail to put all the mail from that list into the tag I had set up for it. Inevitably some mail from some certain people would end up in my spam folder.
Gmail has gotten better, now they will let you override their filters but give you a big warning that the message would have been delivered to spam otherwise.

What are mailers to do? Right now I don’t have a good answer. Sending mail people want is still good advice for individual senders. But I am not sure what can be done about this ESP wide filtering that I’m starting to see. It’s possible Gmail is monitoring all the mail from a particular sender or ESP and applying a “source network” score. Networks letting customers send mail Gmail doesn’t like (such as affiliate mail or payday mail, things they mentioned specifically at M³AAWG) are having all their customers affected.
I suspect this means that ESPs seeing problems across their customer base are going to have to work harder to police their customers and remove problematic mail streams completely. Hopefully, ESPs that can get on the Gmail FBL can identify the problem customers faster before those customers tank mail for all their senders.

ISPs speak at M3AAWG

Last week at M3AAWG representatives from AOL, Yahoo, Gmail and Outlook spoke about their anti-spam technologies and what the organizations were looking for in email.
This session was question and answers, with the moderator asking the majority of the questions. These answers are paraphrased from my notes or the MAAWG twitter stream from the session.
What are your biggest frustrations?
AOL: When senders complain they can’t get mail in and we go look at their stats and complaints are high. Users just don’t love that mail. If complaints are high look at what you may have done differently, content does have an effect on complaints.
Outlook: When we tightened down filters 8 years ago we had to do it. Half of the mail in our users inbox was spam and we were losing a steady number of customers. The filter changes disrupted a lot of senders and caused a lot of pain. But these days only 0.5% of mail in the inbox is spam. Things happen so fast, though, that the stress can frustrate the team.
Gmail: Good senders do email badly sometimes and their mail gets bulked. Senders have to get the basic email hygiene practices right. Love your users and they’ll love you back.
What’s your philosophy and approach towards mail?
AOL: There is a balance that needs to be struck between good and bad mail. The postmaster team reminds the blocking team that not all mail is bad or malicious. They are the sender advocates inside AOL. But the blocking team deals with so much bad mail, they sometimes forget that some mail is good.
Yahoo: User experience. The user always comes first. We strive to protect them from malicious mail and provide them with the emails they want to see. Everything else is secondary.
Gmail: The faster we stop spam the less spam that gets sent overall. We have highly adaptive filters that can react extremely quickly to spam. This frustrates the spammers and they will give up.
Outlook: The core customer is the mailbox user and they are a priority. We think we have most of the hardcore spam under control, and now we’re focused on personalizing the inbox for each user. Everyone online should hold partners accountable and they should expect to be held accountable in turn. This isn’t just a sender / ESP thing, ISPs block each other if there are spam problems.
What are some of your most outrageous requests?
We’ve been threatened with lawsuits because senders just don’t want to do the work to fix things. Some senders try to extort us. Other senders go to the advertising execs and get the execs to yell at the filtering team.
Coming to MAAWG and getting cornered to talk about a particular sender problem. Some senders have even offered money just to get mail to the spam folder.
Senders who escalate through the wrong channels. We spent all this money and time creating channels where you can contact us, and then senders don’t use them.
Confusing business interests with product interests. These are separate things and we can’t change the product to match your business interest.
What are your recommendations for changing behaviors?
Outlook: We provide lots of tools to let you see what your recipients are doing. USE THE TOOLS. Pay attention to your recipient interaction with mail. Re-opt-in recipients periodically. Think about that mail that is never opened. Monitor how people interact with your mail. When you have a problem, use our webpages and our forms. Standard delivery problems have a play book. We’re going to follow that playbook and if you try to get personal attention it’s going to slow things down. If there’s a process problem, we are reachable and can handle them personally. But use the postmaster page for most things.
Gmail: Get your hygiene right. If you get your hygiene right, deliverability just works. If you’re seeing blocking, that’s because users are marking your mail as spam. Pay attention to what the major receivers publish on their postmaster pages. Don’t just follow the letter of the law, follow the spirit as well. Our responsibility, as an ISP, is to detect spam and not spam. Good mailers make that harder on us because they do thinks that look like spammers. This doesn’t get spammer mail in more, it gets legitimate mail in less. Use a real opt-in system, don’t just rely on an implied opt-in because someone made a purchase or something.
Yahoo: ESPs are pretty good about screening their customers, so pay attention to what your ESPs are saying. Send mail people want. Verify that the email addresses given to you actually belong to people who want your mail. Have better sender practices.
What do you think about seed accounts?
The panel wasn’t very happy about the use of seed accounts. Seeds are not that useful any longer, as the ISPs move to more and more personalized delivery. Too much time and too many cycles are used debugging seed accounts. The dynamic delivery works all ways.
When things go wrong what should we do?
AOL: Open a ticket. We know we’ve been lax recently, but have worked out of our backlog and are caught up to date. Using the ticketing system also justifies us getting more headcount and makes everyone’s experience better. Also, don’t continue what you’re doing. Pausing sending while you’re troubleshooting the issue. We won’t adjust a rep for you, but we may be able to help you.
Gmail: Do not jump the gun and open a ticket on the first mail to the spam folder. Our filters are so dynamic, they update every few minutes in some cases. Be sure there is a problem. If you are sure you’re following the spirit and letter of the sender guidelines you can submit a ticket. We don’t respond to tickets, but we work every single one. When you’re opening a ticket provide complete information and full headers, and use the headers from your own email address not headers from a seed account. Give us a clear and concise description of the problem. Also, use the gmail product forum, it is monitored by employees and it’s our preferred way of getting information to the anti-abuse team. Common issues lots of senders are having will get addressed faster.
Outlook: Dig in and do your own troubleshooting, don’t rely on us to tell you what to fix. The support teams don’t have a lot of resources so use our public information. If you make our job harder, then it takes longer to get things done. But tell us what changes you’ve made. If you’ve fixed something, and tell us, our process is different than if you’re just asking for a delisting or asking for information. When you’ve fixed things we will respond faster.
How fast should users expect filters to respond after making changes?
Filters update continually so they should start seeing delivery changes almost immediately. What we find is people tell us they’ve made changes, but they haven’t made enough or made the right ones. If the filters don’t update, then you’ve not fixed the problem.

Engagement, it's not what you might think

Most delivery experts will tell you that ISPs measure recipient engagement as a part of their delivery. That’s absolutely true, but I think there’s a language difference that makes it hard for senders to understand what we mean by engagement.
ISPs, and other filtering companies, profile their user base. They know, for instance, who logs in and checks mail every day. They know who checks mail every 20 seconds. They know who gets a lot of spam. They know who hasn’t logged in for months. They know who accurately marks mail as spam and who is sloppy with the this-is-spam button. They know if certain recipients get the same mail, it’s likely to be spam.
Engagement at the ISPs is more about the recipient engaging with their email address and the mail in their mailbox then it is about the recipient engaging with specific emails.

Abuse it and lose it

Last week I blogged about the changes at ISPs that make “ISP Relations” harder for many senders. But it’s not just ISPs that are making it a little more difficult to get answers to questions, some spam filtering companies are pulling back on offering support to senders.
For instance, Cloudmark sent out an email to some ESPs late last week informing them that Cloudmark was changing their sender support policies. It’s not that they’re overwhelmed with delisting requests, but rather that many ESPs are asking for specific data about why the mail was blocked. In December, Spamcop informed some ESPs that they would stop providing data to those ESPs about specific blocks and spam trap hits.
These decisions make it harder for ESPs to identify specific customers and lists causing them to get blocked. But I understand why the filtering companies have had to take such a radical step.
Support for senders by filtering companies is a side issue. Their customers are the users of the filtering service and support teams are there to help paying customers. Many of the folks at the filtering companies are good people, though, and they’re willing to help blocked senders and ESPs to figure out the problem.
For them, providing information that helps a company clean up is a win. If an ESP has a spamming customer and the information from the filtering company is helping the ESP force the customer to stop spamming that’s a win and that’s why the filtering companies started providing that data to ESPs.
Unfortunately, there are people who take advantage of the filtering companies. I have dozens of stories about how people are taking advantage of the filtering companies. I won’t share specifics, but the summary is that some people and ESPs ask for the same data over and over and over again. The filtering company rep, in an effort to be helpful and improve the overall email ecosystem, answers their questions and sends the data. In some cases, the ESP acts on the data, the mail stream improves and everyone is happy (except maybe the spammer). In other cases, though, the filtering company sees no change in the mail stream. All the filtering company person gets is yet another request for the same data they sent yesterday.
Repetition is tedious. Repetition is frustrating. Repetition is disheartening. Repetition is annoying.
What we’re seeing from both Spamcop and Cloudmark is the logical result from their reps being tired of dealing with ESPs that aren’t visibly fixing their customer spam problems. Both companies are sending some ESPs to the back of the line when it comes to handling information requests, whether or not those ESPs have actually been part of the problem previously.
The Cloudmark letter makes it clear what they’re frustrated about.

Holiday mailing advice from mailbox providers

Christine Borgia has a post on the Return Path blog where she interviews a number of different groups (spamfilters, DNSBLs, mailbox providers) about their filtering strategy for the holidays. Overall, no one changes their filtering during the Holiday Mailing Season. On the other hand, many marketers do change their marketing strategies in ways that trigger more filtering and blocking.
The take home message? Pay attention to what is being sent and who it is being sent to. This is nothing new, but many marketers seem to forget it in the effort to get into their customers’ inboxes.

Looking for message labs help?

There’s a common bounce error from the Message Labs’ filtering appliance that goes no where.

ISP Relationships

Delivra has a new whitepaper written by Ken Magill talking about the value (or lack thereof) of relationships with ISPs. In Ken’s understated way, he calls baloney on ESPs that claim they have great delivery because they have good relationships with ISPs.
He’s right.
I get a lot of calls from potential clients and some calls from current clients asking me if I can contact an ISP on their behalf and “tell the ISP we’re really not a spammer”. My normal answer is that I can, but that there isn’t a place in the spam filtering process for “sender has hired Laura and she says they’re not a spammer.” I mean, it would be totally awesome if that was the case. But it’s not. It’s even the case where I’m close friends with folks inside the ISPs.
I’m pretty sure I’ve told the story before about being at a party with one of the Hotmail ISP folks. There was a sender that had hired me to deal with some Hotmail issues and I’d been working with Barry H. (name changed, and he’s not at Hotmail any more) to resolve it. During the course of the party, we started talking shop. Barry told me that he was sure that my client was sending opt-in mail, but that his users were not reacting well for it. He also told me there was no way he could override the filters because there wasn’t really a place for him to interfere in the filtering.
Even when folks inside the ISPs were willing and able to help me, they usually wouldn’t do so just because I asked. They might look at a sender on my request, but they wouldn’t adjust filters unless the sender met their standards.
These days? ISPs are cutting their non-income producing departments to the bone, and “sender services” is high up the list of departments to cut. Most of the folks I know have moved on from the ISP to the ESP side. Ken mentions one ISP rep that is now working for a sender. I actually know of 3, and those are just employees from the top few ISPs who are now at fairly major ESPs. I’m sure there are a lot more than that.
The reality is, you can have the best relationships in the world with ISPs, but that won’t get bad mail into the inbox. Filters don’t work that way anymore. That doesn’t mean relationships are useless, though. Having relationships at ISPs can get information that can shorten the process of fixing the issue. If an ISP says “you are blocked because you’re hitting spam traps” then we do data hygiene. If the ISP says “you’re sending mail linking to a blocked website” then we stop linking to that website.
I have a very minor quibble with one thing Ken said, though. He says “no one has a relationship with Spamhaus volunteer, they’re all anonymous.” This isn’t exactly true. Spamhaus volunteers do reveal themselves. Some of them go around openly at MAAWG with nametags and affiliations. A couple of them are colleagues from my early MAPS days. Other do keep their identities secret, but will reveal them to people they trust to keep those identities secret. Or who they think have already figured it out. There was one drunken evening at MAAWG where the nice gentleman I was joking with leaned over and says “You know I am elided from Spamhaus, right?” Uh. No? I didn’t. I do now!
But even though I have the semi-mythical personal relationship with folks from Spamhaus, it doesn’t mean my clients get preferential treatment. My clients get good advice, because I know what Spamhaus is looking for and can translate their requirements into solid action steps for the client to perform. But I can think of half a dozen ESP delivery folks that have the same sorts of relationships with Spamhaus volunteers.
Overall, relationships are valuable, but they are not sufficient to fix inbox delivery problems.

Questions on Google lawsuit post

A couple questions in the previous discussion thread about the Google privacy case. Both concern permission granted to Google to scan emails.
Google’s stance about this is fairly simple.
Gmail users give explicit permission for their mail to be scanned.
People who send mail to Gmail users give implicit permission for their mail to be scanned.
The plaintiff’s lawyers are alleging that some subset of gmail users – specifically those at Universities that use Google apps and ISP customers like CableOne – did not give explicit permission for their mail to be scanned by Google. They’re also arguing no senders give permission.
In addition to the lack of permission, the plaintiffs lawyers are arguing that Google’s behaviour is in violation of Google’s own policies.
Google thinks scanning is part of the ordinary course of business and they’re doing nothing wrong.
This is an interesting case. I think anyone who knows about email understands that the people who run the mail server have the ability to read anything that goes through. But a lot of us trust that most postmaster and admin types consider it unprofessional to look at mail without a decent reason. There are good reasons an admin might need to go into a mail spool.
Automated filtering is simply a part of life on the internet these days. Mails have to be scanned for viruses, spam and, yes, they are scanned for targeted advertising. I’m not convinced Google is outside the norm when they say that any emails sent through Google is personal information given too Google and therefore Google can use that information in accordance with their policies.

Inbox challenges and dull email in the tabbed inbox

Getting to the inbox is becoming a greater and greater challenge for many marketers. According to Return Path, 22% of opt in mail doesn’t make it to the inbox.
The challenge to marketers is that a lot of opt in mail isn’t important to the recipient. Sure, they’re happy enough to get it if they notice it, but if it’s not there then they don’t care. They’ll buy from an email ad, but it might not be something they’ll seek out. Recipient behaviour tells the ISPs that the mail isn’t all that important, and a lot of it is just background noise so the ISP not delivering it to the inbox doesn’t matter.
Email marketing is like the Girl Scout of the Internet. If the Girl Scout shows up at your doorstep, you’re probably going to buy those 3 boxes of thin mints. But if she doesn’t, that’s OK. If you really want the cookies, you’ll find the co-worker who is taking orders for his daughter. Or you’ll find the table outside the local coffee shop. The Girl Scout showing up on your doorstep makes it more convenient, but she’s not critical to get your fix. Of course, the bonus of the Girl Scout on the doorstep is that a lot of people who won’t go find the cookies will buy when she’s on the doorstep.
A lot of email marketing triggers purchases that recipients would make anyway. They think they might want a particular product, and when they get that coupon or discount or even just a reminder they make the purchase. The email triggers the purchase of a product the buyer intends to purchase anyway. Some email marketing trigger purchases of things the recipient didn’t know existed, but is so enticing after one email they can’t live without. Some email marketing triggers an impulse purchase. In most of these categories, if mail doesn’t show up in the inbox, the recipient really doesn’t miss it.
Many marketers, despite loud protests that all their mail is important and wanted, know this. That’s why so many marketers are having conniptions about the new Gmail tabbed inbox. They’re losing access to the impulse.
From the data I’ve seen, tabs are effecting email marketing programs. Some programs are seeing more revenue, some are seeing less. I think it really remains to be seen what the long term effects are. For many recipients the new tabbed inbox is a new way to interact with their email. Change is hard, and there is a period of adaptation whenever an interface changes. We really don’t know what the long term effect of tabs on sales will be. Sales may go back to previous levels, sales may increase over previous levels, sales may decrease from current levels or sales may stay at their current levels. The full effect isn’t going to be obvious for a while.
It does mean, though, that email marketers need to step up their game. Email marketing in the age of a tabbed inbox might be less about the impulse purchase and more about cultivation and long term branding.

The 5 stages of a Spamhaus listing

Courtesy of Spencer over at Experian.
The 5 Stages of Recovery

Questions about Spamhaus

I have gotten a lot of questions about Spamhaus since I’ve been talking about them on the blog and on various mailing lists. Those questions can be condensed and summed up into a single thought.

Filtering is not just about spam

A lot of filters started out just as filters against spam. But over the years they’ve morphed into more general blocks against dangerous or problematic email. There’s a lot of crime and bad behavior on the internet, much of it using email as a conduit or vector. Filtering is so much more than stopping spam now. It’s as much, or more, about stopping crime.
Email filters are essential to protect us from scammers. Sometimes I forget this, and then I read about a grandmother getting swindled by a Nigerian scammer and ending up dead.
There are real consequences to poor filtering and there is real crime facilitated by email. It’s easy to forget this as we deal with the email that gets caught in filters when they shouldn’t.
Filters are one of the first lines of defense against online crime.
Not only does filtering stop crime, but they also keep email working. An unfiltered mail stream is an ugly, unreadable, unworkable mess.

Censoring email

It seems some mail to Apple’s iCloud has been caught in filters. Apparently, a few months ago someone sent a script to a iCloud user that contained the phrase “barely legal teen” and Apple’s filters ate it.
The amount of hysteria that I’ve seen in some places about this, though, seems excessive. One of my favorite quotes was from MacWorld and just tells me that many of the people reporting on filtering have no idea how filters really work.

There is no bat phone

I don’t have much to add to Al’s post about the lack of people to call at different ISPs to get mail delivered. I will say there was a time some ISPs had staff that would deal with senders and blocking problems. But those positions have gradually been eliminated over the last 2 or 3 years. In some cases the employees left for greener pastures, in others they were subject to layoffs and budget cuts. In most cases, though, the employees were not replaced.
ISPs have moved to complex and multi-tired spam filtering. They’ve removed the ability of most employees to actually interrupt the filtering and special case a sender. Getting mail delivered is about sending mail that recipients want. It’s not about who you know. It’s about how much recipients like your mail.

Facebook blocking spam: parallels to email filtering

Last month a Dangerous Minds posted numbers that indicated their Facebook posts were reaching fewer users. They suggested that this was a conspiracy by Facebook to make more money and soak small publishers with “exorbitant” advertising fees. I didn’t pay that much attention to it. I use Facebook to communicate with friends. The only commercial entities I “like” or are “friends” with are small local businesses that I shop at.
Today, I see a tweet from Ben Chestnut that looked intriguing.

Now, in a new paper in the Journal of Economic Perspectives, Justin Rao of Microsoft and David Reiley of Google (who met working at Yahoo) have teamed up to estimate the cost of spam to society relative to its worldwide revenues. The societal price tag comes to $20 billion. The revenue? A mere $200 million. As they note, that means that the “‘externality ratio’ of external costs to internal benefits for spam is around 100:1. Spammers are dumping a lot on society and reaping fairly little in return.” In case it’s not clear, this is a suboptimal situation. The Atlantic
Read More

Return Path on Content Filtering

Return Path have an interesting post up about content filtering. I like the model of 3 different kinds of filters, in fact it’s one I’ve been using with clients for over 18 months. Spamfiltering isn’t really about one number or one filter result, it’s a complex interaction of lots of different heuristics designed to answer the question: do recipients want this kind of mail?

First step in delivery

Ever trawl through your logs and notice that there is a delivery problem somewhere? I’m sure everyone sending email in any volume has.
What’s the first thing you do when you discover a block?

Filters and windmills

A colleague of mine was dealing with a client who is experiencing some difficulty delivering to the bulk folder. Said client spent much of a one hour phone call repeating “This is not how a free society works!!”
After the call my colleague commented, “I refuse to get ranty about filter systems.”
I know that filters, and the people who write and maintain them, are a frequent scapegoat for senders. The filters are always the problem, not anything the senders do.
Now, I’ll be the last person who will claim spam filters are perfect, they’re not. Filters sometimes do unexpected things, sometimes they do boneheaded things, sometimes they are broken.
We can’t forget, though, that filters perform a vital role in protecting users from malicious emails. Phishing emails, scams, fake products, viruses are a constant threat. Many end users don’t need to worry about this because filters are so good. But an unfiltered account can get thousands of scams and spams a day (ask me how I know).
Most of us in the delivery space can tell when a filter is working as intended and when there’s an underlying problem. And when the filter is working as intended there’s not a lot of use complaining about them. Ranting about filtering systems often delays a resolution. Senders that focus on what they can control tend to have more success reaching the inbox than those senders that focus on ranting about filtering systems.
Tilting at windmills doesn’t get the mail through.

Don't spam filter your role accounts

A variety of “amazon.com order confirmations” showed up in my inbox this morning. They were quite well done, looking pretty close to real Amazon branding, so quite a few people will click on them. And they funnel people who do click to websites that contain hostile flash apps that’ll compromise their machines (and steal their private data, login and banking credentials then add them to botnets to attack other sites and so on).
Not good. Just the sort of urgent, high-risk issue that ISP abuse desks really want to hear about. I sent email about it to the ISPs involved, including a copy of the original email. One of them went to iWeb, a big (tens of thousands of servers) hosting company.
This was the response:

Email marketing OF THE FUTURE!

ISPs are continually developing tools for their users. Some of the newer tools are automatic filters that help users organize the volumes of mail they’re getting. Gmail released Priority Inbox over a year ago. Hotmail announced new filters as part of Wave 5 back in October.
All of these announcements cause much consternation in the email marketing industry. Just today there was a long discussion on the Only Influencers list about the new Hotmail filtering. There was even some discussion about why the ISPs were doing this.
I think it’s pretty simple why they’re creating new tools: users are asking for them. The core of these new filters is ISPs reacting to consumer demand. They wouldn’t put the energy into development if their users didn’t want it. And many users do and will use priority inbox or the new Hotmail filtering.
Some people are concerned that marketing email will be less effective if mail is not in the inbox.

Email filters

What makes the best email filter? There isn’t really a single answer to that question. Different people and different organizations have different tolerances for how false positives versus false negatives. For instance, we’re quite sensitive to false positives here, so we run extremely conservative filtering and don’t block very much at the MTA level. Other people I know are very sensitive to false negatives and run more aggressive filtering and block quite a bit of mail at the MTA level.
For the major ISPs, the people who plan, approve, design and monitor the filters usually want to maximize customer happiness. They want to deliver as much real mail as possible while blocking as much bad mail. Blocking real mail and letting through bad mail both result in unhappy customers and increase the ISP’s costs, either through customer churn or through support calls. And this is a process, filters are not static. ISPs roll out new filters all the time, sometimes they are an improvement and sometimes they’re not. When they’re not, they’re pulled out of production. This works both for positive filters like Return Path and negative filters like blocklists.
Then there is mail filtering that doesn’t have to do with spam. Business filters, for instance, often block non-business mail. Permission of the recipient often isn’t even a factor. Companies don’t often go out of their way to block personal mail, but if personal mail gets blocked (say the vacation plane ticket or the amazon receipt) they don’t often unblock it. But when you think about why a business provides email, it makes perfect sense. The business provides email to further its own business goals. Some personal usage is usually OK, but if someone notices and blocks personal email then it’s unlikely the business will unblock it, even if the employee opted in.
In the case of email filters, the free market does work. Different ISPs filter mail differently. Some people love Gmail’s filters. Other people think Hotmail has the best filtering. There are different standards for filtering, and that makes email stronger and more robust. Consumers have choices in their mail provider and spamfiltering.

Changes at Gmail

As I’ve said before, I can usually tell when some ISP changes their filtering algorithm because I start getting tons and tons of calls about delivery problems at that ISP. This past month it’s been Gmail.
There have been two symptoms I’ve been hearing about. One is an increase in bulk folder delivery for mail that previously was reliably hitting the inbox. The other is a bit more interesting. I’ve heard of 3 different mailers, with good reputations and very clean lists, that are seeing 4xx delays on some of their mail. The only consistency I, and my colleagues at some ESPs, have identified is that the mail is “bursty.”
The senders affected by this do send out mail daily, but the daily mail is primarily order confirmations or receipts or other transactional mails. They send bi-weekly newsletters, though, exploding their volume from a few tens of thousands up to hundreds of thousands. This seems to trigger Gmail to defer mail. It does get delivered eventually. It’s frustrating to try and deal with because neither side is really doing anything wrong, but good senders are seeing delivery delays.
For the bulk foldering, Bronto has a good blog post talking about the changes and offering some solid suggestions for how to deal with them. I’m also hearing from some folks who are reliable that Gmail may be rolling back some of the bulk foldering changes based on feedback from their users.
So if you’re seeing changes at Gmail, it’s not just you.

Amendment is futile, part 2

When Yahoo filed for dismissal of the Holomaxx complaint, they ended the motion with “Amendment would be futile in this case.” The judge granted Yahoo’s motion but did grant Holomaxx leave to amend. Holomaxx filed an amended complaint earlier this month.
The judge referenced a couple specific deficiencies of Holomaxx’s claims in his dismissal.

Email without filters

… or Find the False Positive.
Anyone sending a lot of email has complained about spam filters and false positives at some point. But most people haven’t run a mailbox with no spam filters in front of it in recent years, so don’t have much of a feel for what an unfiltered mailbox looks like, how important filters are and how difficult their job is.
I run no transaction level filters in front of my mailbox, just content filters that route mail to one of several inboxes or a junk folder, so if I want to I can look at what unfiltered email looks like. I took data from all mail that was sent to me yesterday, and put it in a format that really shows the problem filters face and especially the difficulty of spotting which mail in the junk folder is a false positive.
An inbox with no filters looks like this.

Running a spam filter against it, simply categorizing each email as spam (pink) or not-spam (green) looks like this.

Even with the messages categorized as spam vs not-spam it’s hard to work out which messages are important and which aren’t, let alone where the false positives might be.
If I sort the categories by hand you get this – where you can see that out of 1200 or so mails about three quarters were spam. Of the three false positives two were bulk email that I didn’t care that I didn’t receive and only one was email that I considered important.

Turn it all the way up to 11

I made that joke the other night and most of the folks who heard it didn’t get the reference. It made me feel just a little bit old.
Anyhow, Mickey beat me to it and posted much of what I was going to say about Ken Magill’s response to a very small quote from Neil’s guest post on expiring email headers last week.
I, too, was at that meeting, and at many other meetings where marketers and the folks that run the ISP spam filters end up in the same room. I don’t think the marketers always understand what is happening inside the postmaster and filtering desks on a day to day basis at the ISPs. Legitimate marketing? It’s a small fraction of the mail they deal with. Ken claims that marketing pays the salaries of these employees and they’d be out of a job if marketing didn’t exist. Possibly, but only in the context that they are paid to keep their employers servers up and running so that the giant promises made by the marketing team of faster downloads and better online experiences actually happen.
If there wasn’t an internet and there weren’t servers to maintain, they’d have good jobs elsewhere. They’d be building trains or designing buildings or any of the thousands of other jobs that require smart technical people.
Ken has no idea what these folks running the filters and keeping your email alive deal with on a regular basis. They deal with the utter dregs and horrors of society. They are the people dealing with unrelenting spam and virus and phishing attacks bad enough to threaten to take down their networks and the networks of everyone else. They also end up dealing with law enforcement to deal with criminals. Some of what they do is deal with is unspeakable, abuse and mistreatment of children and animals. These are the folks who stand in front of the rest of us, and make the world better for all of us.
They should be thanked for doing their job, not chastised because they’re doing what the people who pay them expect them to be doing.
Yes, recipients want the mail they want. But, y’know, I bet they really don’t want all the bad stuff that the ISPs protect against. Ken took offense at a statement that he really shouldn’t have. ISPs do check their false positive rates on filtering, and those rates are generally less than 1% of all the email that they filter. Marketers should be glad they’re such a small part of the problem. They really don’t want to be a bigger part.

Change is required

I get a lot of calls from senders who tell me that they have not changed what they were doing, but all of a sudden their mail isn’t performing the way it used to. Sometimes it’s simply less effective marketing, but more often than not the issue is mail being blocked or filtered to the bulk folder.
What worked today won’t work tomorrow. Spammers are forever evolving new techniques to get past spam filters. ISPs are forever evolving new techniques to stop them.
One of the current driving forces for spam filter development is focused on the individual recipients. Recipient wants and needs are king in the world of ISP mail filtering. Much of that is driven by the underlying business models of the free ISPs. They are selling eyeballs to their advertisers and that relies on keeping as many eyeballs around for as long as possible.
An early version of the recipient driven filtering was “add to your address book” where individual users could over ride ISP delivery decisions by actively adding a From: address to their address book. The ISPs have been refining this over time. For instance, if you reply to an email in some clients, you are prompted to add that address to your address books. If you take an email out of your bulk folder and move it to your inbox then that address is automatically added to your address book.
But the refinements haven’t stopped there. ISPs are now making smart decisions about what emails a particular recipient will want to receive. This raises a number of challenges to senders. How do you send email to ten thousand or a hundred thousand or a million people and make it relevant to all of them?
Smart senders will take the individual delivery challenge in stride. They will change along with the ISPs, to send mail that their recipients want to receive. Change is inevitable and required.

Email marketing ulcers for the holiday

I’ve mentioned here before that I can usually tell when the big ISPs are making changes to their spam filtering as that ISP dominates my discussions with current and potential clients and many discussions on delivery mailing lists.
The last two weeks the culprit has been Yahoo. They seem to be making a lot of changes to their filtering schemes right at the busiest email marketing time of the year. Senders are increasing their volume trying to extract that last little bit of cash out of holiday shoppers, but they’re seeing unpredictable delivery results. What worked to get mail into the inbox a month ago isn’t working, or isn’t working as well, now.
Some of this could be holiday volume related. Many marketers have drastically increased their mail volume over the last few weeks. But I don’t think the whole issue is simply that there is more email marketing flowing into our mailboxes.
As I’ve been talking with folks, I have started to see a pattern and have some ideas of what may be happening. It seems a lot of the issue revolves around bulk foldering. Getting mail accepted by the MXs seems to be no different than it has been. The change seems to be based on the reputation of the URLs and domains in the email.
Have a domain with a poor reputation? Bulk. Have a URL seen in mail people aren’t interested in? Bulk. Have a URL pointing to a website with problematic content? Bulk.
In the past IPs that were whitelisted or had very good reputations could improve delivery of email with neutral or even borderline poor reputations. It seems that is no longer an effect senders can rely on. It may even be that Yahoo, and other ISPs, are going to start splitting IP reputation from content reputation. IP reputation is critical for getting mail in the door, and without a good IP reputation you’ll see slow delivery. But once the mail has been accepted, there’s a whole other level of filtering, most of it on the content and generally unaffected by the IP reputation.
I don’t think the changes are going to go away any time soon. I think they may be refined, but I do think that reputation on email content (particularly domains and URLs and target IP addresses) is going to play a bigger and bigger role in email delivery.
What, specifically, is going to happen at Yahoo? Only they can tell you and I’m not sure I have enough of a feel for the pattern to speculate about the future. I do think that it’s going to take a few weeks for things to settle down and be consistent enough that we can start to poke the black box and map how it works.

Comments on Holomaxx post

I’m putting together a longer analysis of the Holomaxx case that will look at the claims against the various defendants. There’s some deep mis-understanding of how various things works (hint: wiretapping? not so much).
There was one comment from “The Other Barry” about complaints that I think bears highlighting.

We're gonna party like it's 1996!

Over on deliverability.com Dela Quist has a long blog post up talking about how changes to Hotmail and Gmail’s priority inbox are a class action suit waiting to happen.
All I can say is that it’s all been tried before. Cyberpromotions v. AOL started the ball rolling when they tried to use the First Amendment to force AOL to accept their unsolicited email. The courts said No.
Time goes on and things change. No one argues Sanford wasn’t spamming, he even admitted as much in his court documents. He was attempting to force AOL to accept his unsolicited commercial email for their users. Dela’s arguments center around solicited mail, though.
Do I really think that minor difference in terminology going to change things?
No.
First off “solicited” has a very squishy meaning when looking at any company, particularly large national brands. “We bought a list” and “This person made a purchase from us” are more common than any email marketer wants to admit to. Buying, selling and assuming permission are par for the course in the “legitimate” email marketing world. Just because the marketer tells me that I solicited their email does not actually mean I solicited their email.
Secondly, email marketers don’t get to dictate what recipients do and do not want. Do ISPs occasionally make boneheaded filtering decisions? I’d be a fool to say no. But more often than not when an ISP blocks your mail or filters it into the bulk folder they are doing it because the recipients don’t want that mail and don’t care that it’s in the bulk folder. Sorry, much of the incredibly important marketing mail isn’t actually that important to the recipient.
Dela mentions things like bank statements and bills. Does he really think that recipients are too stupid to add the from address to their address books? Or create specific filters so they can get the mail they want? People do this regularly and if they really want mail they have the tools, provided by the ISP, to make the mail they want get to where they want it.
Finally, there is this little law that protects ISPs. 47 USC 230 states:

Gmail and the PBL

Yesterday I wrote about the underlying philosophy of spam filtering and how different places have different philosophies that drive their filtering decisions. That post was actually triggered by a blog post I read where the author was asking why Gmail was using the PBL but instead of rejecting mail from PBL listed hosts they instead accepted and bulkfoldered the mail.
The blog post ends with a question:

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Spamtraps

There is a lot of mythology surrounding spamtraps, what they are, what they mean, how they’re used and how they get on lists.
Spamtraps are very simply unused addresses that receive spam. They come from a number of places, but the most common spamtraps can be classified in a few ways.

Blocking of ESPs

There’s been quite a bit of discussion on my post about upcoming changes that ESPs will be facing in the future. One thing some people read into the post is the idea that ISPs will be blocking ESPs wholesale without any regard for the quality of the mail from that company.
The idea that ESPs are at risk for blocking simply because they are ESPs has been floating around the industry based on comments by an employee at a spam filter vendor at a recent industry conference.
I talked to the company to get some clarification on what that spam filtering company is doing and hopefully to calm some of the concerns that people have.
First off, and probably most important, is that the spam filtering company in question primarily targets their service to enterprises. Filtering is an important part of this service, but it also handles email archiving, URL filtering and employee monitoring. The target market for the company is very different than the ISP market.
The ISPs are not talking about blocking indiscriminately, they are talking about blocking based on bad behavior.
Secondly, this option was driven by customer request. The customers of the spam filtering appliance were complaining about “legitimate” mail from various ESPs. Despite being reasonable targeted the mail was unrequested by the recipient. While ESPs use FBLs and other sources of complaints to clean complainers off rented or epended lists at ISPs, the option is not available for mail sent to corporations. Enterprises don’t, nor should they have to, create and support FBLs. Nor should employees be expected to unsubscribe from mail they never requested.
This option is the direct result of ESPs allowing customers to send spam.
Thirdly, this option is offered to those customers who ask for it. It is not done automatically for everyone. The option is also configurable down to the end user.
While I haven’t seen the options, nor which ESPs are affected, I expect that the ones on the list are the ones that the filtering vendor receives complaints about. If you are not allowing your customers to send spam, and are stopping them from buying lists or epending, then you probably have not come to the attention of the filtering company and are not on the list of ESPs to block.

Controlling delivery

How much control over delivery do senders have? I have repeatedly said that senders control their delivery. This is mostly true. Senders control their side of the delivery chain, but there is a point where the recipient takes over and controls things.
As a recipient I can

The secret to dealing with ISPs

What is the secret to dealing with ISPs?
The short answer is: Don’t do it if at all possible. Talking to ISP reps generally isn’t going to magically improve your reptuation. There is no place in the reputation systems where delivery can be modified because the delivery specialist knows or is liked by the postmaster at an ISP.
With my clients, I work through delivery issues and can solve 80 – 90% of the issues without ever having to contact anyone at the ISPs. 90% of the remaining issues can be handled using the publicly available contacts and websites provided by the ISPs.
In the remaining cases, the “secret” to getting useful and prompt replies is to: