Spam Filters

Why Deliverability Matters to Me

Welcome to deliverability week. I want to especially thank Al for doing a lot of work behind the scenes herding this group of cats. He’s an invaluable asset to the community.

Troubleshooting: part 3

As I continue to think about how people troubleshoot email delivery I keep finding other things to talk about. Today we’re going to talk about the question most folks start with when troubleshooting delivery. “Did ISP change something?”

Tulsi Gabbard Sues Google

Today Tulsi Gabbard’s campaign sued Google for $50 million. Why? Because during the night of the first debate Google disabled her “advertising account” (I’m assuming she means adwords) preventing her from being able to purchase ads to direct searchers to her website. There’s also a paragraph in there that they’re “disproportionally putting her email into the spam folder.”

Who are mimecast?

Mimecast is a filter primarily used by businesses. They’re fairly widely used. In some of the data analysis I’ve done for clients, they’re a top 10 or top 20 filter.
Earlier today someone asked on Facebook if mimecast may be blocking emails based on the TLD. The short answer is it’s unlikely. I’ve not seen huge issues with them blocking based on TLD of the domain. They’re generally more selective than that.

The good news is mimecast is really pretty good about giving you explanations for why they’re blocking. They’ll even tell you if it’s mimecast related or if it’s a specific user / user-company block.
Some example rejection messages from a recent dive into some bounce logs.

AHBL Wildcards the Internet

AHBL (Abusive Host Blocking List) is a DNSBL (Domain Name Service Blacklist) that has been available since 2003 and is used by administrators to crowd-source spam sources, open proxies, and open relays. By collecting the data into a single list, an email system can check this blacklist to determine if a message should be accepted or rejected. AHBL is managed by The Summit Open Source Development Group and they have decided after 11 years they no longer wish to maintain the blacklist.
A DNSBL works like this, a mail server checks the sender’s IP address of every inbound email against a blacklist and the blacklist responses with either, yes that IP address is on the blacklist or no I did not find that IP address on the list. If an IP address is found on the list, the email administrator, based on the policies setup on their server, can take a number of actions such as rejecting the message, quarantining the message, or increasing the spam score of the email.
The administrators of AHBL have chosen to list the world as their shutdown strategy. The DNSBL now answers ‘yes’ to every query. The theory behind this strategy is that users of the list will discover that their mail is all being blocked and stop querying the list causing this. In principle, this should work. But in practice it really does not because many people querying lists are not doing it as part of a pass/fail delivery system. Many lists are queried as part of a scoring system.
Maintaining a DNSBL is a lot of work and after years of providing a valuable service, you are thanked with the difficulties with decommissioning the list. Popular DNSBLs like the AHBL list are used by thousands of administrators and it is a tough task to get them to all stop using the list. RFC6471 has a number of recommendations such as increasing the delay in how long it takes to respond to a query but this does not stop people from using the list. You could change the page responding to the site to advise people the list is no longer valid, but unlike when you surf the web and come across a 404 page, a computer does not mind checking the same 404 page over and over.
Many mailservers, particularly those only serving a small number of users, are running spam filters in fire-and-forget mode, unmaintained, unmonitored, and seldom upgraded until the hardware they are running on dies and is replaced. Unless they do proper liveness detection on the blacklists they are using (and they basically never do) they will keep querying a list forever, unless it breaks something so spectacularly that the admin notices it.
So spread the word,

Spam is not a moral judgement

Mention an email is spam to some senders and watch them dance around trying to explain all the ways they aren’t spammers. At some point, calling an email spam seems to have gone from a statement of fact into some sort of moral judgement on the sender. But calling an email spam is not a moral judgement. It’s just a statement of what a particular recipient thinks of an email.
There are lots of reasons mail can be blocked and not all those reasons are spam related. Sometimes it’s a policy based rejection. Mailbox providers publishing a DMARC record with a reject policy caused a lot of mail to bounce, but none of that was because that user (or that mailing list) was sending spam. Most cable companies prohibit customers from running mail servers on their cable connection and mail from those companies is widely rejected, but that doesn’t mean the mail is spam.
Sometimes a block is because some of the mail is being sent to people who didn’t ask for it or are complaining about it. This doesn’t make the sender a bad person. It doesn’t make the sending company bad. It just means that there is some issue with a part of the marketing program that need to be addressed.
The biggest problem I see is some senders get so invested in convincing receivers, delivery experts and filtering companies that they’re not spammers, that they miss actually fixing the problem. They are so worried that someone might think they’re spammers, they don’t actually listen to what’s being said by the blocking organization, or by their ISP or by their ESP.
Calling email spam isn’t a moral judgement. But, if too many people call a particular email spam, it’s going to be challenging to get that mail to the inbox. Instead of arguing with those people, and the filters that listen to them, a better use of time and energy is fixing the reasons people aren’t liking your email.

Spamhaus Speaks

There’s been a lot of discussion about Spamhaus, spam traps, and blocking. Today, Spamhaus rep Denny Watson posted on the Spamhaus blog about some of the recent large retailer listings. He provides us with some very useful information about how Spamhaus works, and gives 3 case studies of recent listings specifically for transactional messages to traps.
The whole thing is well worth a read, and I strongly encourage you to check out the whole thing.
There are a couple things mentioned in the blog that I think deserve some special attention, though.
Not all spam traps actually accept mail. In fact, in all of the 3 case studies, mail was rejected during the SMTP transaction. This did not stop the senders from continuing to attempt to mail to that address, though. I’ve heard over and over again from senders that the “problem” is that spamtrap addresses actually accept mail. If they would just bounce the messages then there would be no problem. This is clearly untrue when we actually look at the data. All of the companies mentioned are large brick and mortar retailers in the Fortune 200. These are not small or dumb outfits. Still, they have massive problems in their mail programs that mean they continue to send to addresses that bounce and have always bounced.
Listings require multiple hits and ongoing evidence of problems. None of the retailers mentioned in the case studies had a single trap hit. No, they had ongoing and repeated trap hits even after mail was rejected. Another thing senders tell me is that it’s unfair that they’re listed because of “one mistake” or “one trap hit.” The reality is a little different, though. These retailers are listed because they have horrible data hygiene and continually mail to addresses that simply don’t exist. If these retailers were to do one-and-out or even three-and-out then they wouldn’t be listed on the SBL. Denny even says that in the blog post.

Censoring email

It seems some mail to Apple’s iCloud has been caught in filters. Apparently, a few months ago someone sent a script to a iCloud user that contained the phrase “barely legal teen” and Apple’s filters ate it.
The amount of hysteria that I’ve seen in some places about this, though, seems excessive. One of my favorite quotes was from MacWorld and just tells me that many of the people reporting on filtering have no idea how filters really work.

8 things that make your mail look like spam

In the comments of last week’s Wednesday question John B. asked

Things Spammers Do

Much like every other day, I got some spam today. Here’s a lightly edited copy of it.
Let’s go through it and see what they did that makes it clear that it’s spam, which companies helped them out, and what you should avoid doing to avoid looking like these spammers…

Hunting the Human Representative

Yesterday’s post was inspired by a number of questions I’ve fielded recently from people in the email industry. Some were clients, some were colleagues on mailing lists, but in most cases they’d found a delivery issue that they couldn’t solve and were looking for the elusive Human Representative of an ISP.
There was a time when having a contact inside an ISP was almost required to have good delivery. ISPs didn’t have very transparent systems and SMTP rejection messages weren’t very helpful to a sender. Only a very few ISPs even had postmaster pages, and the information there wasn’t always helpful.
More recently that’s changed. It’s no longer required to have a good relationship at the ISPs to get inbox delivery. I can point to a number of reasons this is the case.
ISPs have figured out that providing postmaster pages and more information in rejection messages lowers the cost of dealing with senders. As the economy has struggled ISPs have had to cut back on staff, much like every other business out there. Supporting senders turned into a money and personnel sink that they just couldn’t afford any longer.
Another big issue is the improvement in filters and processing power. Filters that relied on IP addresses and IP reputation did so for mostly technical reasons. IP addresses are the one thing that spammers couldn’t forge (mostly) and checking them could be done quickly so as not to bottleneck mail delivery. But modern fast processors allow more complex information analysis in short periods of time. Not only does this mean more granular filters, but filters can also be more dynamic. Filters block mail, but also self resolve in some set period of time. People don’t need to babysit the filters because if sender behaviour improves, then the filters automatically notice and fall off.
Then we have authentication and the protocols now being layered on top of that. This is a technology that is benefiting everyone, but has been strongly influenced by the ISPs and employees of the ISPs. This permits ISPs to filter on more than just IP reputation, but to include specific domain reputations as well.
Another factor in the removal of the human is that there are a lot of dishonest people out there. Some of those dishonest people send mail. Some of them even found contacts inside the ISPs. Yes, there are some bad people who lied and cheated their way into filtering exceptions. These people were bad enough and caused enough problems for the ISPs and the ISP employees who were lied to that systems started to have fewer and fewer places a human could override the automatic decisions.
All of this contributes to the fact that the Human Representative is becoming a more and more elusive target. In a way that’s good, though; it levels the playing field and doesn’t give con artists and scammers better access to the inbox than honest people. It means that smaller senders have a chance to get mail to the inbox, and it means that fewer people have to make judgement calls about the filters and what mail is worthy or not. All mail is subject to the same conditions.
The Human Representative is endangered. And I think this is a good thing for email.

Motivating people

I’ve been thinking a lot about motivating people recently. What really motivates people to do things? Why do we make the choices we make? How do you convince people to do things when they’re unsure they want to do those things?
Let me give you an example. Friends of mine are fostering dogs for local rescues. A neighbor of theirs is trying to start a rescue herself. The neighbor is trying to motivate people by posting pictures of dead dogs in garbage bags. On one level, I get the neighbor’s point: that image is what motivates her to take action. But all that’s doing for other people, my friends included, is driving them away from working with her.
What she needs is a better grasp of how to motivate people. She needs to learn how to speak to people in a way that will motivate them to help her. Unfortunately, she thinks that what motivates her will motivate everyone, except it doesn’t. In fact, it’s doing the exact opposite for some people who are actually sympathetic to her cause.
What does this have to do with email?
I’m often surprised at how many marketing professionals can’t or won’t tailor their argument to their audience. Look at filters, many marketers have told me over the years about how mean ISPs are to them, how the ISPs make poor filtering decisions and how what should really happen is marketers should tell the ISPs to fix their filters.
In very few cases, though, have I seen a marketer actually try and talk to an ISP rep on their terms. It seems so simple to me: marketers are people who motivate people for a living so they should be able to market their own wants to ISPs. They just need to find the right message, but they don’t seem to be able to think about things from the ISP perspective.
I’m not sure I actually have an answer. But how do we motivate people to do things has been a major topic in my head recently. I think the best motivation is often to convince the other party that a given course is in their best interest. The tricky bit is selling that message.
How have you sold a message the other party didn’t want to hear?

Content, trigger words and subject lines

There’s been quite a bit of traffic on twitter this afternoon about a recent blog post by Hubspot identifying trigger words senders should avoid in an email subject line. A number of email experts are assuring the world that content doesn’t matter and are arguing on twitter and in the post comments that no one will block an email because those words are in the subject line.
As usually, I think everyone else is a little bit right and a little bit wrong.
The words and phrases posted by Hubspot are pulled out of the Spamassassin rule set. Using those words or exact phrases will cause a spam score to go up, sometimes by a little (0.5 points) and sometimes by a lot (3+ points). Most spamassassin installations consider anything with more than 5 points to be spam so a 3 point score for a subject line may cause mail to be filtered.
The folks who are outraged at the blog post, though, don’t seem to have read the article very closely. Hubspot doesn’t actually say that using trigger words will get mail blocked. What they say is a lot more reasonable than that.

Listen to me talk about filtering, blocklists and delivery

I did an interview with Practical eCommerce a few weeks ago. The podcast and transcript are now available.
I want to thank Kerry and the rest of the staff there for the opportunity to talk email and filtering with their readers.
Happy Thanksgiving everyone in the US.

The sledgehammer of confirmed opt-in

We focused Monday on Trend/MAPS blocking fully confirmed opt-in (COI) mail, because that is the Gold Standard for opt-in. It is also Trend/MAPS stated policy that all mail should be COI. There are some problems with this approach. The biggest is that Trend/MAPS is confirming some of the email they receive and then listing COI senders.
The other problem is that typos happen by real people signing up for mail they want. Because MAPS is using typo domains to drive listings, they’re going to see a lot of mail from companies that are doing single opt-in. I realize that there are problems with single opt-in mail, but the problems depends on a lot of factors. Not all single opt-in lists are full of traps and spam and bad data.
In fact, one ESP has a customer with a list of more than 50 million single opt-in email addresses. This sender mails extremely heavily, and yet sees little to no blocking by public or private blocklists.
Trend/MAPS policy is singling out senders that are sending mail people signed up to receive. We know for sure that hard core spammers spend a lot of time and money to identify spamtraps. The typo traps that Trend/MAPS use are pretty easy to find and I have no doubt that the real, problematic spammers are pulling traps out of their lists. Legitimate senders, particularly the ESPs, aren’t going to do that. As one ESP rep commented on yesterday’s post:

A Disturbing Trend

Over the last year or so we’ve been hearing some concerns about some of the blacklisting policies and decisions at Trend Micro / MAPS.
One common thread is that the ESP customers being listed aren’t the sort of sender who you’d expect to be a significant source of abuse. Real companies, gathering addresses from signup forms on their website. Not spammers who buy lists, or who harvest addresses, or who are generating high levels of complaints – rather legitimate senders who are, at worst, being a bit sloppy with their data management. When Trend blacklist an IP address due to a spamtrap hit from one of these customers the actions they are demanding before delisting seem out of proportion to the actual level of abuse seen – often requiring that the ESP terminate the customer or have the customer reconfirm the entire list.
“Reconfirming” means sending an opt-in challenge to every existing subscriber, and dropping any subscriber who doesn’t click on the confirmation link. It’s a very blunt tool. It will annoy the existing recipients and will usually lead to a lot of otherwise happy, engaged subscribers being removed from the mailing list. While reconfirmation can be a useful tool in cleaning up senders who have serious data integrity problems, it’s an overreaction in the case of a sender who doesn’t have any serious problems. “Proportionate punishment” issues aside, it often won’t do anything to improve the state of the email ecosystem. Rather than staying with their current ESP and doing some data hygiene work to fix their real problems, if any, they’re more likely to just move elsewhere. The ESP loses a customer, the sender keeps sending the same email.
If this were all that was going on, it would just mean that the MAPS blacklists are likely to block mail from senders who are sending mostly wanted email.
It’s worse than that, though.
The other thread is that we’re being told that Trend/MAPS are blocking IP addresses that only send confirmed, closed-loop opt-in email, due to spamtrap hits – and they’re not doing so accidentally, as they’re not removing those listings when told that those addresses only emit COI email. That’s something it’s hard to believe a serious blacklist would do, so we decided to dig down and look at what’s going on.
Trend/MAPS have registered upwards of 5,000 domains for use as spamtraps. Some of them are the sort of “fake” domain that people enter into a web form when they want a fake email address (“fakeaddressforyourlist.com”, “nonofyourbussiness.com”, “noneatall.com”). Some of them are the sort of domains that people will accidentally typo when entering an email address (“netvigattor.com”, “lettterbox.com”, “ahoo.es”). Some of them look like they were created automatically by flaky software or were taken from people obfuscating their email addresses to avoid spam (“notmenetvigator.com”, “nofuckinspamhotmail.com”, “nospamsprintnet.com”). And some are real domains that were used for real websites and email in the past, then acquired by Trend/MAPS (“networkembroidery.com”, “omeganetworking.com”, “sheratonforms.com”). And some are just inscrutable (“5b727e6575b89c827e8c9756076e9163.com” – it’s probably an MD5 hash of something, and is exactly the sort of domain you’d use when you wanted to be able to prove ownership after the fact, by knowing what it’s an MD5 hash of).
Some of these are good traps for detecting mail sent to old lists, but many of them (typos, fake addresses) are good traps for detecting mail sent to email addresses entered into web forms – in other words, for the sort of mail typically sent by opt-in mailers.
How are they listing sources of pure COI email, though? That’s simple – Trend/MAPS are taking email sent to the trap domains they own, then they’re clicking on the confirmation links in the email.
Yes. Really.
So if someone typos their email address in your signup form (“steve@netvigattor.com” instead of “steve@netvigator.com”) you’ll send a confirmation email to that address. Trend/MAPS will get that misdirected email, and may click on the confirmation link, and then you’ll “know” that it’s a legitimate, confirmed signup – because Trend/MAPS did confirm they wanted the email. Then at some later date, you’ll end up being blacklisted for sending that 100% COI email to a “MAPS spamtrap”. Then Trend/MAPS require you to reconfirm your entire list to get removed from their blacklist – despite the fact that it’s already COI email, and risking that Trend/MAPS may click on the confirmation links in that reconfirmation run, and blacklist you again based on the same “spamtrap hit” in the future.

Changes at Gmail

As I’ve said before, I can usually tell when some ISP changes their filtering algorithm because I start getting tons and tons of calls about delivery problems at that ISP. This past month it’s been Gmail.
There have been two symptoms I’ve been hearing about. One is an increase in bulk folder delivery for mail that previously was reliably hitting the inbox. The other is a bit more interesting. I’ve heard of 3 different mailers, with good reputations and very clean lists, that are seeing 4xx delays on some of their mail. The only consistency I, and my colleagues at some ESPs, have identified is that the mail is “bursty.”
The senders affected by this do send out mail daily, but the daily mail is primarily order confirmations or receipts or other transactional mails. They send bi-weekly newsletters, though, exploding their volume from a few tens of thousands up to hundreds of thousands. This seems to trigger Gmail to defer mail. It does get delivered eventually. It’s frustrating to try and deal with because neither side is really doing anything wrong, but good senders are seeing delivery delays.
For the bulk foldering, Bronto has a good blog post talking about the changes and offering some solid suggestions for how to deal with them. I’m also hearing from some folks who are reliable that Gmail may be rolling back some of the bulk foldering changes based on feedback from their users.
So if you’re seeing changes at Gmail, it’s not just you.

Feedback loops

There are a lot of different perspectives on Feedback Loops (FBLs) and “this is spam” buttons across the email industry.
Some people think FBLs are the best thing since sliced bread and can’t figure out why more ISPs don’t offer them. These people use use the data to clean addresses off their lists, lower complaints and send better mail. They use the complaints as a data source to help them send mail their recipients want. Too many recipients opted out on a particular offer? Clearly there is a problem with the offer or the segmentation or something.
Other people, though, think the existence of “this is spam” buttons and FBLs is horrible. They call people who click “this is spam” terrorists or anti-commerce-net-nazis. They want to be able to dispute every click of the button. They think that too many ISPs offer this is spam buttons and too many ESPs and network providers pay way to much attention to complaints. The argue ISPs should remove these buttons and stop paying attention to what recipients think.
Sadly, I’m not actually making up the terminology in the last paragraph. There really are who think that the problem isn’t with the mail that they’re sending but that the recipients can actually express an opinion about it and the ISPs listen to those opinions. “Terrorists” and “Nazis” are the least of the things they have called people who complain about their mail.
One of the senior engineers at Cloudmark recently posted an article talking about FBLs and “this is spam” buttons. I think it’s a useful article to read as it explains what value FBLs play in helping spam filters become more accurate.

Change is required

I get a lot of calls from senders who tell me that they have not changed what they were doing, but all of a sudden their mail isn’t performing the way it used to. Sometimes it’s simply less effective marketing, but more often than not the issue is mail being blocked or filtered to the bulk folder.
What worked today won’t work tomorrow. Spammers are forever evolving new techniques to get past spam filters. ISPs are forever evolving new techniques to stop them.
One of the current driving forces for spam filter development is focused on the individual recipients. Recipient wants and needs are king in the world of ISP mail filtering. Much of that is driven by the underlying business models of the free ISPs. They are selling eyeballs to their advertisers and that relies on keeping as many eyeballs around for as long as possible.
An early version of the recipient driven filtering was “add to your address book” where individual users could over ride ISP delivery decisions by actively adding a From: address to their address book. The ISPs have been refining this over time. For instance, if you reply to an email in some clients, you are prompted to add that address to your address books. If you take an email out of your bulk folder and move it to your inbox then that address is automatically added to your address book.
But the refinements haven’t stopped there. ISPs are now making smart decisions about what emails a particular recipient will want to receive. This raises a number of challenges to senders. How do you send email to ten thousand or a hundred thousand or a million people and make it relevant to all of them?
Smart senders will take the individual delivery challenge in stride. They will change along with the ISPs, to send mail that their recipients want to receive. Change is inevitable and required.

Is your mail being bulk foldered?

Daisy at Signup.to posted a list of 11 things to do when mail is going to the bulk folder. Her suggestions are a good start to troubleshooting and fixing persistent bulk foldering of mail.
One thing she doesn’t mention is that while bulk foldering can sometimes be the result of poor content, more often it’s the result of unengaged recipients. Think of bulk foldering this way: the ISP has some subscribers they’re pretty sure want your mail, so they’re not going to block your mail. But they’re pretty sure a lot of subscribers don’t want your mail so they’re not going to deliver it to the inbox.
The trick to getting mail moved out of the bulk folder is to get more people engaged with your email marketing. This is tough to do if they’re not actively checking their bulk folder for mail but there are some ways I’ve helped clients get mail into the inbox.

Content based filters

Content based filters are incredibly complex and entire books could be written about how they work and what they look at. Of course, by the time the book was written it would be entirely obsolete. Because of their complexity, though, I am always looking for new ways to explain them to folks.
Content based filters look at a whole range of things, from the actual text in the message, to the domains, to the IP addresses those domains and URLs point to. They look at the hidden structure of an email. They look at what’s in the body of the message and what’s in the headers. There isn’t a single bit of a message that content filters ignore.
Clients usually ask me what words they should change to avoid the filters. But this isn’t the right question to ask. Usually it’s not a word that causes the problem. Let me give you a few examples of what I mean.
James H. has an example over on the Cloudmark blog of how a single missing space in an email caused delivery problems for a large company. That missing space changed a domain name in the message sufficiently to be caught by a number of filters. This is one type of content filter, that focuses on what the message is advertising or who the beneficiary of the message is. Some of my better clients get caught by these types of filters occasionally. A website they’re linking to or a domain name they’re using in the text of the message has a bad reputation. The mail gets bulked or blocked because of that domain in the message.
One of my clients went from 100% inbox every day to random failures at different domains. Their overall inbox was still in the 96 – 98% range, but there was a definite change. The actual content of their mail hadn’t changed, but we kept looking for underlying causes. At one point we were on the phone and they mentioned their new content management system. Sure enough, the content management company had a poor reputation and the delivery problems started exactly when they started using the content management. The tricky part of this was that the actual domains and URLs in the messages never changed, they were still clickthrough.clientdomain.example.com. But those URLs now pointed to an IP address that a lot of spammers were abusing. So there were delivery problems. We made some changes to their setup and the delivery problems went away.
The third example is one from quite a long time ago, but illustrates a key point. A client was testing email sends through a new ESP. They were sending one-line mail through the ESPs platform to their own email account. Their corporate spamfilter was blocking the mail. After much investigation and a bit of string pulling, I finally got to talk to an engineer at the spamfiltering company. He told me that they were blocking the mail because it “looked like spam.” When pressed, he told me they blocked anything that had a single line of text and an unsubscribe link. Once the client added a second line of text, the filtering issue went away.
These are just some of the examples of how complex content based filters are. Content is almost a misnomer for them, as they look at so many other things including layout, URLs, domains and links.

Spamfilters: a marketer's best friend

I was cleaning out my spam folder this afternoon. I try and do it at least once a day, otherwise the volume gets so bad I don’t actually look at the mail I just mark it all as read. I realized, though, that spamfilters are actually a marketer’s best friend.
If there were no spam filters keeping all the crap people get out of their inbox (in my case over 1000 messages a day) then spam would overwhelm even the most dedicated email junkie. I couldn’t do my job without my spam filters, and in fact the recent rash of virus spew is ending up in my inbox and making finding real mail a problem. I do a lot of sorting before mail ever hits my inbox, and I’m still struggling to deal with the couple hundred “your order has shipped!” and “please her tonight!” emails that my local bayesian filters haven’t caught up to, yet.
Today’s stats:
Work inbox: 17 messages
Work spam: 419
95.9% spam
Personal inbox: 40
Personal spam: 975
95.9% spam
Without filters, I couldn’t accurately find that 4.1% of real mail that I get. Without filters, I couldn’t do my job. Without filters, I couldn’t find the real receipts from purchases I actually made. Without filters, I couldn’t read and respond to mail I wanted.
A mailbox overflowing with spam is unuseable, and email marketers should be thankful that providers work so hard to keep spam out. Otherwise, email wouldn’t be useful for anything.

I'm on a blocklist! HELP!

Recently, an abuse desk rep asked what to do when customers were complaining about being assigned an IP address located on a blocklist. Because not every blocklist actually affects mail delivery it’s helpful to identify if the listing is causing a problem before diving in and trying to resolve the issue.

Why do ISPs do that?

One of the most common things I hear is “but why does the ISP do it that way?” The generic answer for that question is: because it works for them and meets their needs. Anyone designing a mail system has to implement some sort of spam filtering and will have to accept the potential for lost mail. Even the those recipients who runs no software filtering may lose mail. Their spamfilter is the delete key and sometimes they’ll delete a real mail.
Every mailserver admin, whether managing a MTA for a corporation, an ISP or themselves inevitably looks at the question of false positives and false negatives. Some are more sensitive to false negatives and would rather block real mail than have to wade through a mailbox full of spam. Others are more sensitive to false positives and would rather deal with unfiltered spam than risk losing mail.
At the ISPs, many of these decisions aren’t made by one person, but the decisions are driven by the business philosophy, requirements and technology. The different consumer ISPs have different philosophies and these show in their spamfiltering.
Gmail, for instance, has a lot of faith in their ability to sort, classify and rank text. This is, after all, what Google does. Therefore, they accept most of the email delivered to Gmail users and then sort after the fact. This fits their technology, their available resources and their business philosophy. They leave as much filtering at the enduser level as they can.
Yahoo, on the other hand, chooses to filter mail at the MTA. While their spamfoldering algorithms are good, they don’t want to waste CPU and filtering effort on mail that they think may be spam. So, they choose to block heavily at the edge, going so far as to rate limit senders that they don’t know about the mail. Endusers are protected from malicious mail and senders have the ability to retry mail until it is accepted.
The same types of entries could be written about Hotmail or AOL. They could even be written about the various spam filter vendors and blocklists. Every company has their own way of doing things and their way reflects their underlying business philosophy.

Define "spam"

A comment came through recently from Trent asking me to define spam. It’s been a while since I’ve talked about how I define spam, so let’s look at it.
Personally, I describe spam as unsolicited bulk email. If I didn’t ask for it and it looks like bulk mail then I consider it spam. In many cases the spammers have multiple email addresses of mine so I can demonstrate the mail was sent in bulk.
In my consulting and working with clients, though, I rarely use the word spam. There are so many different definitions of spam, I have no way to know if my clients understand what I am saying, so I avoid the term as much as humanly possible. An example of some of the few definitions of spam I’ve seen used over the years.

New Delivery tools

A couple nifty new delivery tools were published over the weekend.
Mickey published Bounce P.I. where senders can paste in an error message or bounce and it will tell you what filter generated it. If the rejection is unrecognized, it will flag the message internally and it will be researched to see if the filter can be identified.
Steve has a new tool at the DKIMCore site. The key generating tool and the record checking tool have been up for a while. This weekend, though, he published a tool to check the validity the DKIM record published in DNS. Tool output shows if the record is valid, the version and the public key.

TWSD: Dumb and dumber

I recently received a spam offering to get one of my personal websites listed in foreign search engines. Harvesting addresses off websites is dumb. Even dumber is sending a followup a week later with a notice at the top.

How Spamfilters Work

AllSpammedUp has a post describing the primary techniques anti-spam filters use to identify mail as spam or not spam. While is this not sender or delivery focused knowledge, it is important for people sending mail to have a basic understanding of filtering mechanisms. Without that base knowledge, it’s difficult to troubleshoot problems and resolve issues.

Delivery lore

(Image from Bad Astronomy)
Almost every delivery consultant, delivery expert or deliverability blog offers their secrets to understanding spam filters. As a reader, though, how do you know if the author knows what they’re talking about? For instance, on one of the major delivery blogs had an article today saying that emails with a specific subject line will not get past spam filters.
This type of statement is nothing new. The lore around spam filters and what they do and do not do permeates our industry. Most of the has achieved the status of urban legend, and yet is still repeated as gospel. Proof? I sent an email with the subject line quoted in the above blog post to my aol, yahoo, gmail and hotmail accounts. Within 3 minutes of sending the email it was in the inbox of all 4 accounts
I can come up with any number of reasons why the email ended up in my inbox, rather than being caught by spam filters as the delivery expert originally claimed. But none of those reasons really matter. The expert in question is spreading delivery lore that is demonstrably false. Emails with that subject line will get through spam filters. I even added an extra 4 exclamation points in the subject line.
Not all delivery lore is true. In fact, most lore involving “always” “all” “never” or “none” is not going to be true. Just because you read it on the internet, and because it came from someone claiming to know what they’re talking about does not absolve individual senders from critically thinking about the information.

Who is Julia and why won't she leave me alone?

There seems to be some new spam software in use. Julia <random last name> keeps telling me about her new webcam, how much she wants to date me and wants to know when I want to visit. These spams started February 1. I’ve had 179 caught by my MUA filters, and 152 caught by spamassassin (SA score >7 are filtered to a special account).
This is exactly the type of pattern that causes people to write filters that years later people look at and ask why someone thought this was a reasonable marker for spam.
The good folks over at MailChimp have examined some of the scoring rules that their clients trigger. They found some “Julia” type markers. Some oddities they reported on: