Recent Posts

Are botnets really the spam problem?

Over the last few years I’ve been hearing some people claim that botnets are the real spam problem and that if you can find a sender then they’re not a problem. Much of this is said in the context of hating on Canada for passing a law that requires senders actually get permission before sending email.
Botnets are a problem online. They’re a problem in a lot of ways. They can be used for denial of service attacks. They can be used to mine bitcoins. They can be used to host viruses. They can be used to send spam. They are a problem and a lot of people spend a lot of time and money trying to take down botnets.
For the typical end user, though, botnets are a minor contributor to spam in the inbox. Major ISPs, throughout the world, have worked together to address botnets and minimize the spam traffic from them. Those actions have been effective and many users never see botnet spam in their inbox, either because it’s blocked during send or blocked during receipt.
Most of the spam end users have to deal with is coming from people who nominally follow CAN SPAM. They have a real address at the bottom of the email. They’re using real ISPs or ESPs. They have unsubscribe links. Probably some of the mail is going to opt-in recipients. This mail is tricky, and expensive, to block, so a lot more of it gets through.
Much of this mail is sent by companies using real ISP connections. Brian Krebs, who I’ve mentioned before, wrote an article about one hosting company who previously supported a number of legal spammers. This hosting company was making $150,000 a month by letting customers send CAN SPAM legal mail. But the mail was unwanted enough that AOL blocked all of the network IP space – not just the spammer space, but all the IP space.
It’s an easy decision to block botnet sources. The amount of real mail coming from botnet space is zero. It’s a much bigger and more difficult decision to block legitimate sources of emails because there’s so much garbage coming from nearby IPs. What AOL did is a last resort when it’s clear the ISP isn’t going to stop spam coming out from their space.
Botnets are a problem. But quasi legitimate spammers are a bigger problem for filter admins and end users. Quasi legitimate spammers tend to hide behind ISPs and innocent customers. Some send off shared pools at ESPs and hide their traffic in the midst of wanted mail. They’re a bigger problem because the mail is harder to filter. They are bigger problems because a small portion of their recipients actually do want their mail. They’re bigger problems because some ISPs take their money and look the other way.
Botnets are easy to block, which makes them a solved problem. Spam from fixed IPs is harder to deal with and a bigger problem for endusers and filters.

Give Recipients Options

A few years ago I subscribed to a financial website that emails out articles about investing as well as a recap of your investments. For the first few months I enjoyed reading these emails but as time went on, I found them less valuable and receiving them every other day they turned into a burden to clean up and deal with.
My options were to either unsubscribe or I could create a rule in Outlook to file away the emails to possibly read them later.
options What I would really like is the option to define how often I would receive the updates. If I’m actively looking to change my investments, I would want to receive the emails daily. I would also like to have the option for either a weekly or monthly email.
The frequency of mailings should be tailored to the subscriber. Buying a new car? I may want to see emails and reviews daily. Just bought a new blender? I want to receive emails for the first few days learning about the different features and recipes. The idea is to present options to each subscriber on what they prefer. It’s better to treat subscribers as individuals rather than sending the same message to your entire list.
The newsletter I was receiving does not provide me with any type of control over how many times I receive the updates. The newsletter is also lacking a working unsubscribe link leaving me no alternative to clicking “this is junk”.
Senders should consider providing recipients with options:

Google Postmaster Tools

Earlier this month Google announced a new set of tools for senders at their Postmaster Tools site. To get into the site you need to login to Google, but they also have a handy support page that doesn’t require a login for folks who want to see what the page is about.
We did register, but don’t send enough mail to get any data back from Google. However, the nice folks at SendGrid were kind enough to share their experiences with me and show me what the site looked like with real data, when I spoke at their recent customer meeting.
Who can register?
Anyone can register for Google Postmaster tools. All you need is the domain authenticated by DKIM (the d= value) or by SPF (the Return Path value).
Who can see data?
Google is only sharing data with trusted domains and only if a minimum volume is sent from those domains. They don’t describe what a trusted domain is, but I expect the criteria include a domain with some history (no brand new domains) and a reasonable track record (some or all of the mail is good).
For ESPs who want to monitor all the mail they send, every mail needs to be signed with a common d= domain. Individual customers that want their own d= can do so. These customers can register for their own access to just their mail.
ESPs that want to do this need to sign with the common key first, and then with the customer’s more selective key.
How does it work?
Google collects data from DKIM and/or SPF authenticated mail, aggregates it and presents it to a Google user that has authenticated the domain.
How do I authenticate?

IPv6 Email is a little different

On Monday I talked about how big IPv6 address space is, and how many IPv6 addresses will be available to end users. We’re mostly an email blog, though, so what’s the relevance to sending email?
If the recipient you’re sending to has an IPv6 mailserver you can send mail to them over IPv6, if you choose to. If they only have an IPv6 mailserver, with no IPv4 mailserver at all then you have to send over IPv6 to reach them.
For a long time I was pretty sure that IPv6-only mailservers were unlikely to be an issue any time soon – as IPv6 rolls out end users will get IPv6 addresses, and that will free up a huge number of IPv4 addresses that can then be used where they’re more valuable, for webservers and mailservers. As I’ve watched IPv4 addresses run out and the rise of a secondary market I’m begining to think that hoarding may make IPv4 addresses effectively unavailable or prohibitively expensive for small companies and individuals in some regions. If so, then a few IPv6-only mailservers will encourage others to support sending and receiving email over IPv6, which will in turn make IPv6-only servers more viable.
And you might want to use IPv6 even if the recipient has a dual-stack IPv4+IPv6 mailserver. As one example, Gmail accepts mail on IPv6 – and scuttlebutt is that right now their IPv6 servers are somewhat more forgiving for properly authenticated email, which is interesting. And if you’re running short of IPv4 addresses yourself, routing all your gmail recipients over IPv6 instead might free up some capacity and save you from having to go IPv4 address shopping.
But there are a few things to know before starting to send over IPv6.
IPv6 to IPv4 fallback
If you turn on IPv6 support on a mailserver it is likely to prefer IPv6 when sending mail to dual-stack recipients. That’s great if everything is set up perfectly, but if your IPv6 network configuration is flakey, or your authentication is not good enough for IPv6 mail, or the recipient has an IPv6-specific configuration problem then your delivery over IPv6 might fail. How to choose an MX for delivery – and how to fall back to an IPv4 MX – isn’t terribly well defined so there’s some risk of delivery of a message failing repeatedly. You should check how your smarthosts handle this sort of delivery failure.
There’s no legacy IPv6 to support
There are twenty year old servers sending email over IPv4, so attempts to enforce better authentication both of mailservers and messages have moved very slowly so as not to disrupt mail from those old servers.
IPv6 is a whole new world, though. Any mailserver set up to send via IPv6 has been set up relatively recently and it’s much more reasonable to expect it’s operators to follow best practices (PDF). If you want to send mail to Google over IPv6 you have to have “good” reverse DNS, and you have to authenticate the mail you send with at least one of SPF or DKIM. Google are much less tolerant of violations of those requirements for mail sent over IPv6, more likely to mark messages as spam or reject them altogether compared to delivery attempts over IPv4.
What does “good” reverse DNS mean? The IP address you’re sending from must have reverse DNS that resolves to a hostname, and that hostname must resolve back to the sending IP address. (You’ll sometimes see that described as “FCrDNS”.)
One customer, one /64
As I mentioned on Monday a consumer end user should be allocated no less than a /64 of IPv6 space. If you’re in the IPv4 mindset of addresses being scarce and valuable you might decide that you don’t need to do that with your customers, maybe assigning each of them a/124 to send their mail through. 16 IP addresses is plenty, right?
A large hosting company did that recently, assigning each of their customers a small range of IPv6 addresses out of a single /64 – and they discovered why it’s a terrible idea. They had no more than the usual level of email delivery problems on IPv4, but all of their IPv6 mail was blocked at a lot of destinations. Because a /64 is the smallest recommended range to assign to a user it’s also the smallest quantum that reputation services and blacklists will block by. Bad behaviour by one of their customers got the /64 that customer was sending from blocked – along with all the other customers sending from other parts of that /64.
So don’t do that.

LinkedIn addresses frequency issues

Yesterday LinkedIn announced they’re decreasing the amount of mail they’re sending to users.

IPv6 is big. Really big. You just won’t believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it’s a long way down the road to the chemist, but that’s just peanuts to IPv6.
Read More

Microsoft Send

Microsoft Send is a new mail client by Microsoft for iPhones and soon Windows Phone and Android phones. Send is designed to send quick, short messages to contacts. Instead of building a chat application build on a proprietary protocol, Send sends and receives its messages over email and uses your existing mailbox to handle the messages. What makes Send neat is that I can start a conversation within the app and when I get back to my computer, I can log into Outlook Web Access and continue the conversation.
MicrosoftSend

Messages to and from the Send app do not utilize subjects lines.

Sending a message from my personal account with Office365 to my Word to the Wise account and the email looks like any other email I received except with the #Send on the subject line.
Inbox

The message goes through the same outbound mail servers as if I sent it from Outlook or OWA, so emails pass SPF.

SPF

If you are signing with DKIM, the emails will be signed and authenticated.

(Office365 will sign emails with DKIM soon, it’s on the Office RoadMap.)
For an email to show up within the Send app, the subject contains #Send.
Microsoft has taken a unique approach to building a messaging app that utilizes existing SMTP infrastructure. If you’re sending to a tech savvy list, take a look at your logs to see how many recipients are using Microsoft Send and consider reaching out to them specifically using #Send.

Gmail having issues

As of 7/22/15, 1:17 PM, Google reports the issue is resolved.

Over on the mailop list multiple people are reporting delivery problems to Gmail.
The Google status page confirms this:

Yes, Virginia, there is list churn

Yesterday I talked about how data collection, management, and maintenance play a crucial role in deliverability. I mentioned, briefly, the idea that bad data can accumulate on a list that isn’t well managed. Today I’d like to dig into that a little more and talk about the non-permanence of email addresses.
A common statistic used to describe list churn is that 30% of addresses become invalid in a year. This was research done by Return Path back in the early 2000’s. The actual research report is hard to find, but I found a couple articles and press releases discussing the info.

Data is the key to deliverability

Last week I had the pleasure of speaking to the Sendgrid Customer Advisory Board about email and deliverability. As usually happens when I give talks, I learned a bunch of new things that I’m now integrating into my mental model of email.
One thing that bubbled up to take over a lot of my thought processes is how important data collection and data maintenance is to deliverability. In fact, I’m reaching the conclusion that the vast majority of deliverability problems stem from data issues. How data is collected, how data is managed, how data is maintained all impact how well email is delivered.
Collecting Data
There are many pathways used to collect data for email: online purchases, in-store purchases, signups on websites, registration cards, trade shows, fishbowl drops, purchases, co-reg… the list goes on and on. In today’s world there is a big push to make data collection as frictionless as possible. Making collection processes frictionless (or low friction) often means limiting data checking and correction. In email this can result in mail going to people who never signed up. Filters are actually really good at identifying mail streams going to the wrong people.
The end result of poor data collection processes is poor delivery.
There are lots of way to collect data that incorporates some level of data checking and verifying the customer’s identity. There are ways to do this without adding any friction, even. About 8 years ago I was working with a major retailer that was dealing with a SBL listing due to bad addresses in their store signup program. What they ended up implementing was tagged coupons emailed to the user. When the user went to the store to redeem the coupons, the email address was confirmed as associated with the account. We took what the customers were doing anyway, and turned it into a way to do closed loop confirmation of their email address.
Managing Data
Data management is a major challenge for lots of senders. Data gets pulled out of the database of record and then put into silos for different marketing efforts. If the data flow isn’t managed well, the different streams can have different bounce or activity data. In a worst case scenario, bad addressees like spamtraps, can be reactivated and lead to blocking.
This isn’t theoretical. Last year I worked with a major political group that was dealing with a SBL issue directly related to poor data management. Multiple databases were used to store data and there was no central database. Because of this, unsubscribed and inactivated addresses were reactivated. This included a set of data that was inactivated to deal with a previous SBL listing. Eventually, spamtraps were mailed again and they were blocked. Working with the client data team, we clarified and improved the data flow so that inactive addresses could not get accidentally or unknowingly reactivated.
Maintaining Data
A dozen years ago few companies needed to think about any data maintenance processes other than “it bounces and we remove it.” Most mailbox accounts were tied into dialup or broadband accounts. Accounts lasted until the user stopped paying and then mail started bouncing. Additionally, mailbox accounts often had small limits on how much data they could hold. My first ISP account was limited to 10MB, and that included anything I published on my website. I would archive mail monthly to keep mail from bouncing due to a full mailbox.
But that’s not how email works today. Many people have migrated to free webmail providers for email. This means they can create (and abandon) addresses at any time. Free webmail providers have their own rules for bouncing mail, but generally accounts last for months or even years after the user has stopped logging into them. With the advent of multi gigabyte storage limits, accounts almost never fill up.
These days, companies need to address what they’re going to do with data if there’s no interaction with the recipient in a certain time period. Otherwise, bad data just keeps accumulating and lowering deliverability.
Deliverability is all about the data. Good data collection and good data management and good data maintenance results in good email delivery. Doing the wrong thing with data leads to delivery problems.