Horses, not zebras

I was first introduced to the maxim “When you hear hoofbeats, think horses not zebras” when I worked in my first molecular biology lab 20-some-odd years ago. I’m no longer a gene jockey, but I still find myself applying this to troubleshooting delivery problems for clients.
It’s not that I think all delivery problems are caused by “horses”, or that “zebras” never cause problems for email delivery. It’s more that there are some very common causes of delivery problems and it’s a more effective use of time to address those common problems before getting into the less common cases.
This was actually something that one of the mailbox provider reps said at M3AAWG in SF last month. They have no problem with personal escalations when there’s something unusual going on. But, the majority of issues can be handled through the standard channels.
What are the horses I look for with delivery problems.

  1. Technical issues. These are actually getting rarer as companies move to the designed-for-bulk MTAs like MessageSystems and Port25. But they’re still worth checking and can contribute to delivery problems. Luckily, technical issues are often the easiest to solve, and once they’re solved usually stay solved.
  2. Content issues. These are getting much more common as ISPs start looking at all the URLs and links in emails, including the landing pages.  Gmail, for instance, does almost all their filtering based on content rather than originating IP. This is not that difficult to solve, but can be harder to solve than technical issues.
  3. Address collection issues. Most delivery problems start at the point of address collection, and these can be challenging problems to solve. Not only do you have to solve the problem moving forward, you also have to decide what to do with the addresses you’ve previously collected. And the problems are only solved as long as someone knows why things are done the way they are.

 

Related Posts

Setting expectations at the point of sale

In my consulting, I emphasize that senders must set recipient expectations correctly. Receiver sites spend a lot of time listening to their users and design filters to let wanted and expected mail through. Senders that treat recipients as partners in their success usually have much better email delivery than those senders that treat recipients as targets or marks.
Over the years I’ve heard just about every excuse as to why a particular client can’t set expectations well. One of the most common is that no one does it. My experience this weekend at a PetSmart indicates otherwise.
As I was checking out I showed my loyalty card to the cashier. He ran it through the machine and then started talking about the program.
Cashier: Did you give us your email address when you signed up for the program?
Me: I’m not sure, probably not. I get a lot of email already.
Cashier: Well, if you do give us an email address associated with the card every purchase will trigger coupons sent to your email address. These aren’t random, they’re based on your purchase. So if you purchase cat stuff we won’t send you coupons for horse supplies.
I have to admit, I was impressed. PetSmart has email address processes that I recommend to clients on a regular basis. No, they’re not a client so I can’t directly take credit. But whoever runs their email program knows recipients are an important part of email delivery. They’re investing time and training into making sure their floor staff communicate what the email address will be used for, what the emails will offer and how often they’ll arrive.
It’s certainly possible PetSmart has the occasional email delivery problem despite this, but I expect they’re as close to 100% inbox delivery as anyone else out there.

Read More

Ad-hoc analysis

I often pull emails into a database to analyze them, but sometimes I want something simpler. Emails are typically stored in one of two ways: mbox format, where an entire mailbox is stored in a single file, and maildir format, where a mailbox is a directory with one file in it for each email.
My desktop mail application is Mail.app on OS X, and it stores messages in a maildir-ish format, so I’m going to work with that here. If you’re using mbox format mailboxes it’s a little trickier (but you can use a tool called formmail to split an mbox style format into a maildir directory and go from there).
I want to gather some statistics on mail I’ve sent to abuse desks, so the first thing I do is open up a terminal window and change directory to where my “Sent Messages” mailbox is:
cd Library/Mail/V2/IMAP-steve@misc.wordtothewise.com/Sent Messages.mbox
(Tab completion is really useful for navigating through the mailbox hierarchy.)
Then I need to go through every email (file) in that directory, for each file find the “To:” header and check to see if it was sent to an abuse desk. If it was sent to an abuse desk I want to find the email address for each one, count how many times I see that email address and find the top twenty or so abuse desks I send reports to. I can do all that with a single command line:
find . -type f -exec egrep -m1 '^To:' {} ; | egrep -o 'abuse@[a-zA-Z0-9._-]+' | sort | uniq -c | sort -nr | head -20
(Enter that all as a single line, even though it’s wrapped into two here).
That’s a bit much to understand all at once, so lets redo that in several stages, with an intermediate file so we can see what’s going on.
find . -type f -exec egrep -m1 '^To:' {} ; >tolines.txt
The find command finds all the files in a directory and does something with them. In this case we start looking in the current directory (“.”), look just for files (“-type f”) and for each file we find we run that file through another command (“-exec egrep -m1 ‘^To:’ {} ;”) and write the result of that command to a file (“>tolines.txt”). The egrep command we run for each file goes through the file and prints out the first (“-m1”) line it finds that begins with “To:” (“‘^To:'”). If you run that and take a look at the file it creates you can see one line for each message, containing the “To:” header (or at least the first line of it).
The next thing to do is to go through that and pull out just the email addresses – and just the ones that are sent to abuse desks:
egrep -o 'abuse@[a-zA-Z0-9._-]+' tolines.txt
This uses egrep a second time, this time to look for lines that look like an email address (“‘abuse@[a-zA-Z0-9._-]+'”) and when it finds one print out just the part of the line that matched the pattern (“-o”).
Running that gives us one line of output for each email we’re interested in, containing the address it was sent to. Next we want to count how many times we see each one. There’s a command line idiom for that:
egrep -o 'abuse@[a-zA-Z0-9._-]+' tolines.txt | sort | uniq -c
This takes all the lines and sorts (“sort”, reasonably enough) them – so that identical lines will be next to each other – then counts runs of identical lines (“uniq -c”). We’re nearly there – the result of this is a count and an email address on each line. We just need to find the top 20:
egrep -o 'abuse@[a-zA-Z0-9._-]+' tolines.txt | sort | uniq -c | sort -nr | head -20
Each line begins with the count, so we can use sort again, this time telling it to sort by number, high to low (“sort -nr”). Finally, “head -20” will print just the first 20 lines of the result.
The final result is this:

Read More

This month in email: February 2014

After a few months of hiatus, I’m resurrecting the this month in email feature. So what did we talk about in February?
Industry News
There was quite a bit of industry news. M3AAWG was in mid-February and there were actually a few sessions we were allowed to blog about. Gmail announced their new pilot FBL program. Ladar Levinson gave the keynote talking about the Lavabit shutdown and his new darkmail program. Brian Krebs won the Mary Litynski award for his work in investigating online security issues. The 4 major mailbox providers talked about their spam filters and spam filtering philosophy.
February was also the month where different companies evaluated their success or failure of products. LinkedIn announced the shutdown of their Intro product and Facebook announced the shutdown of their Facebook.com email service.
Security Issues
Cloudmark published their 2013 report on the Global Spam Threat and we discovered that the massive Target breach started through phishing. I also noticed a serious uptick in the amount of phishing mails in my own mailbox. There is  new round of denial of service attacks using NTP amplification. We provided information on how to secure your NTP servers.
Address Collection
The Hip Hop group De La Soul released their entire catalog for free, online, using a confirmed opt-in email process. On the flip side, the M3AAWG hotel required anyone logging into the wifi network to give an email address and agree to receive marketing mail. We also discovered that some political mailing lists were being used in ways the politicians and recipients didn’t expect.
Email Practices
I talked about how to go about contacting an ISP that doesn’t have a postmaster page or a published method of contact. Much of that information is actually relevant for contacting ISPs that do have a contact method, too. Finally, I talked about how ISPs measure engagement and how that’s significantly different from how ESPs think it is.
 

Read More