Trawling through the junk folder

As a break from writing unit tests this morning I took a few minutes to go through my Mail.app junk folder, looking for false positives for mail delivered over the past six weeks.
trashcans
We don’t do any connection level rejection here, so any mail sent to me gets delivered somewhere. Anything that looks like malware gets dumped in one folder and never read, anything that scores a ridiculously high spamassassin score gets dumped in another folder and never read, mailing lists get handled specially and everything else gets delivered to Mail.app to deal with. That means that Mail.app sees less of the ridiculously obvious spam and is mostly left to do bayesian filtering, and whatever other magic Apple implemented.
There were about thirty false positives, and they were all B2C bulk advertising mail. I receive a lot of 1:1 mail, transactional mail and B2B marketing mail and there were no false positives at all for any of those.
All the false positives were authenticated with both SPF and DKIM. All of them were for marketing lists I’d signed up for while making a purchase. All of them were “greymail” – mail that I’d agreed to receive, and that was inoffensive but not compelling. While I easily spotted all of them as false positives via the from address and subject, none of them were content I’d particularly missed.
Almost all of the false positives were sent through ESPs I recognized the name of, and about 80% of them were sent through just two ESPs (though that wasn’t immediately obvious, as one of them not only uses random four character domain names, it uses several different ones – stop doing that).
If you’d asked me to name two large, legitimate ESPs from whom I recalled receiving blatant, blatant spam recently, it would be those same two ESPs. Is Mail.app is picking up on my opinions of the mail those ESPs are sending? It’s possible – details specific to a particular ESPs mail composition and delivery pipelines are details that a bayesian learning filter may well recognize as efficient tokens.

Related Posts

URL reputation and shorteners

A bit of  a throwback post from Steve a few years ago. The problem has gotten a little better as some shortening companies are actually disabling spammed URLs, and blocking URLs with problematic content. I still don’t recommend using a public URL shortener in email messages, though.
Any time you put a URL in mail you send out, you’re sharing the reputation of everyone who uses URLs with that hostname. So if other people send unwanted email that has the same URL in it that can cause your mail to be blocked or sent to the bulk folder.
That has a bunch of implications. If you run an affiliate programme where your affiliates use your URLs then spam sent by your affiliates can cause your (clean, opt-in, transactional) email to be treated as spam. If you send a newsletter with advertisers URLs in it then bad behaviour by other senders with the same advertisers can cause your email to be spam foldered. And, as we discussed yesterday, if spammers use the same URL shortener you do, that can cause your mail to be marked as spam.
Even if the hostname you use for your URLs is unique to you, if it resolves to the same IP address as a URL that’s being used in spam, that can cause delivery problems for you.
What does this mean when it comes to using URL shorteners (such as bit.ly, tinyurl.com, etc.) in email you send out? That depends on why you’re using those URL shorteners.
The URLs in the text/html parts of my message are big and ugly
Unless the URL you’re using is, itself, part of your brand identity then you really don’t need to make the URL in the HTML part of the message visible at all. Instead of using ‘<a href=”long_ugly_url”> long_ugly_url </a>’ or ‘<a href=”shortened_url”> shortened_url </a>’ use ‘<a href=”long_ugly_url”> friendly phrase </a>’.
(Whatever you do, don’t use ‘<a href=”long_ugly_url”> different_url </a>’, though – that leads to you falling foul of phishing filters).
The URLs in the text/plain parts of my message are big and ugly
The best solution is to fix your web application so that the URLs are smaller and prettier. That will make you seem less dated and clunky both when you send email, and when your users copy and paste links to your site via email or IM or twitter or whatever. “Cool” or “friendly” URLs are great for a lot of reasons, and this is just one. Tim Berners-Lee has some good thoughts on this, and AListApart has two good articles on how to implement them.
If you can’t do that, then using your own, branded URL shortener is the next best thing. Your domain is part of your brand – you don’t want to hide it.
I want to use a catchy URL shortener to enhance my brand
That’s quite a good reason. But if you’re doing that, you’re probably planning to use your own domain for your URL shortener (Google uses goo.gl, Word to the Wise use wttw.me, etc). That will avoid many of the problems with using a generic URL shortener, whether you implement it yourself or use a third party service to run it.
I want to hide the destination URL from recipients and spam filters
Then you’re probably spamming. Stop doing that.
I want to be able to track clicks on the link, using bit.ly’s neat click track reporting
Bit.ly does have pretty slick reporting. But it’s very weak compared to even the most basic clickthrough reporting an ESP offers. An ESP can tell you not just how many clicks you got on a link, but also which recipients clicked and how many clicks there were for all the links in a particular email or email campaign, and how that correlates with “opens” (however you define that).
So bit.ly’s tracking is great if you’re doing ad-hoc posts to twitter, but if you’re sending bulk email you (or your ESP) can do so much better.
I want people to have a short URL to share on twitter
Almost all twitter clients will abbreviate a URL using some URL shortener automatically if it’s long. Unless you’re planning on using your own branded URL shortener, using someone else’s will just hide your brand. It’s all probably going to get rewritten as t.co/UgLy in the tweet itself anyway.
If your ESP offers their own URL shortener, integrating into their reporting system for URLs in email or on twitter that’s great – they’ll be policing users of that just the same as users of their email service, so you’re unlikely to be sharing it with bad spammers for long enough to matter.
All the cool kids are using bit.ly, so I need to to look cool
This one I can’t help with. You’ll need to decide whether bit.ly links really look cool to your recipient demographic (Spoiler: probably not) and, if so, whether it’s worth the delivery problems they risk causing.
And, remember, your domain is part of your brand. If you’re hiding your domain, you’re hiding your branding.
So… I really do need a URL shortener. Now what?
It’s cheap and easy to register a domain for just your own use as a URL shortener. Simply by having your own domain, you avoid most of the problems. You can run a URL shortener yourself – there are a bunch of freely available packages to do it, or it’s only a few hours work for a developer to create from scratch.
Or you can use a third-party provider to run it for you. (Using a third-party provider does mean that you’re sharing the same IP address as other URL shorteners – but everyone you’re sharing with are probably people like you, running a private URL shortener, so the risk is much, much smaller than using a freely available public URL shortener service.)
These are fairly simple fixes for a problem that’s here today, and is going to get worse in the future.

Read More

January 2015 – The Month in Email

It’s February already! January went fast, right? At WttW, we are gearing up for MAAWG SF later this month — will we see you there?
We started the year with a set of predictions about email. Mostly we think email will continue to be great at some things and not-so-great at other things, and we’ll keep fighting the good fight to make it better.
As always, I’m interested in filters and how spammers continue to work around them to reach the inbox. I also wrote about how the language of an email impacts delivery, and wrote an expanded response to a comment suggesting email filters should be illegal. You can guess where I stand on that (and if you can’t, perhaps you might read more about how email is an inherently malicious traffic stream…)
I also took a moment to point out a trend I’m really enjoying, which is the rise of content marketing (a.k.a. giving customers useful and interesting information they can’t find elsewhere). As I said in the post, I’ll be curious to see how ROI plays out with this strategy.
We also talked about some of the less exciting content we see in email, notably the infamous Murkowski Statement, by which a spammer declares “Nope! Nothing to see over here!”
Steve also pointed out some content shenanigans in the form of hidden preview text, with some additional clarification from the original marketer in the comments.
In industry news, the big story was that Microsoft has partially implemented DMARC for Office365, and was the first to make a public statement about the specific ways they’ve chosen to implement. In my post, I did a walkthrough of a message to illustrate a bit about how this works, which might be useful if you’re trying to wrap your head around DMARC implementations.
We also talked about consolidation in the ESP space, and got a number of comments from readers about who they think might be next. Shortly thereafter, Listcast was acquired by MailerMailer.
Josh noted a few major shutdowns: Yahoo China email services and the AHBL list. The latter explores the challenges inherent in decommissioning a blacklist, and there’s a good discussion in the comments, so you might check it out if you missed that earlier this month.
Josh also pointed to the Salesforce State of Marketing report, which is always a useful set of metrics about how marketers are using email and other channels. It’s definitely worth a read.

Read More

June 2015: the Month in Email

Happy July! We are back from another wonderful M3AAWG conference and enjoyed seeing many of you in Dublin. It’s always so great for us to connect with our friends, colleagues, and readers in person. I took a few notes on Michel van Eeten’s keynote on botnets, and congratulated our friend Rodney Joffe on winning the prestigious Mary Litynski Award.
In anti-spam news, June brought announcements of three ISP-initiated CAN-SPAM cases, as well as a significant fine leveled by the Canadian Radio-television and Telecommunications Commission (CRTC) against Porter Airlines. In other legal news, a UK case against Spamhaus has been settled, which continues the precedent we’ve observed that documenting a company’s practice of sending unsolicited email does not constitute libel.
In industry news, AOL started using Sender Score Certification, and Yahoo announced (and then implemented) a change to how they handle their Complaint Feedback Loop (CFL). Anyone have anything to report on how that’s working? We also noted that Google has discontinued the Google Apps for ISPs program, so we expect we might see some migration challenges along the way. I wrote a bit about some trends I’m seeing in how email programs are starting to use filtering technologies for email organization as well as fighting spam.
Steve, Josh and I all contributed some “best practices” posts this month on both technical issues and program management issues. Steve reminded us that what might seem like a universal celebration might not be a happy time for everyone, and marketers should consider more thoughtful strategies to respect that. I wrote a bit about privacy protection (and pointed to Al Iverson’s post on the topic), and Josh wrote about when senders should include a physical address, what PTR (or Reverse DNS) records are and how to use them, testing your opt-out process (do it regularly!), and advice on how to use images when many recipients view email with images blocked.

Read More