Thoughts on bounce handling

This week’s Wednesday question comes from D.

What are your thoughts on bounce handling

There’s no specific standard for bounce handling as we often talk about it. While a lot of people will mention the RFCs, the RFCs only say what to do with a single email that gets a 4xy or a 5xy during the SMTP transaction. If the message gets a 5xy during the SMTP transaction, then the email MUST NOT be retried. If the message gets a 4xy during the SMTP transaction then it can be retried.
What the RFCs don’t mention is what to do with future emails to that email address. In context, this makes sense. When the RFCs were originally written there weren’t lists of tens or hundreds of thousands of addresses. At that time, every email was sent by a person and the bounces came directly back to that person. The person could decide, based on the bounce, what to do about future emails.
Volume, however, means we have to automate bounce handling and parse hundreds of different rejection messages.
I’m somewhat agnostic on exactly how someone should bounce handle. As long as a sender is making a good faith effort to remove undeliverable addresses within a reasonable time frame, I don’t think the details are critical. Removing after 4 bounces or after 5 bounces isn’t going to make a huge difference in delivery.
I don’t advocate bounce handling for 4xx messages. If you have a compliant MTA, then it will retry mail rejected with a 4xx. If that mail cannot be delivered over your retry period (usually over 3 – 5 days) then it should “hard bounce” with a 5xx error and that will be measured as a hard bounce.
The handshake standard that those ISPs and ESPs in the room agreed to back in 2002(ish) was 3 consecutive bounces in more than 14 days meant an address should be removed. It basically gives the senders mailing daily get 14 chances to mail to a non existent addresses. Mailers that send weekly get 3 attempts.
Sometimes ISPs do have issues. Very rarely they’ll lose some sync between their MTA and their backend database and indicate addresses are invalid when they really aren’t. Giving a minimum 2 weeks and a minimum number of bounces keeps these (again, VERY RARE) happenings from removing all the addresses off a list.
I don’t think spamblocks should be necessarily used to purge addresses, only unknown user 5xy responses (usually 500 or 550).
For some of my clients with more configurable systems, I will often have them purge addresses on the first bounce *if* that is also the first time they’ve mailed that address. It’s a statistics thing: the chance of a valid address getting on a list the same day as an ISP falls over is fairly low. Therefore, being aggressive in bounce handling on the first mailing is a good idea. Also, if a company is trying to re-engage users or start mailing a segment that hasn’t been mailed recently, then they should consider removing on a single bounce.
Don’t forget there are times when mail will fail even before the start of a SMTP session. This means that you WILL NOT get a 5xy or 4xy message on many MTAs (some of the commercial ones manage this for you, most of the open source ones don’t). These failures are typically DNS failures; things like no MX, a MX of “.”, or NXDomain are all signs that address is undeliverable and should be handled as an unknown user response. MX . or NXdomain should be purged immediately.

Opting customers in to new programs
Censoring email

Related Posts

20% of email doesn't make it to the inbox

Return Path released their global delivery report for the second half of 2009. To put together the report, they look at mail delivery to the Mailbox Monitor accounts at 131 different ISPs for 600,000+ sends. In the US, 20% of the email sent by Mailbox Monitor customers to Return Path seed accounts doesn’t make it to the inbox. In fact, 16% of the email just disappears.
I’ve blogged in the past about previous Return Path deliverability studies. The recommendations and comments in those previous posts still apply. Senders must pay attention to engagement, permission, complaints and other policy issues. But none of those things really explain why email is missing.
Why is so much mail disappearing? It doesn’t match with the philosophy of the ISPs. Most ISPs do their best to deliver email that they accept and I don’t really expect that ISPs are starting to hard block so many Return Path customers in the middle of a send. The real clue came looking at the Yahoo numbers. Yahoo is one of those ISPs that does not delete mail they have accepted, but does slow down senders. Other ISPs are following Yahoo’s lead and using temporary failures as a way to regulate and limit email sent by senders with poor to inadequate reputations. They aren’t blocking the senders outright, but they are issuing lots of 4xx “come back later” messages.
What is supposed to happen when an ISP issues a 4xx message during the SMTP transaction is that email should be queued and retried. Modern bulk MTAs (MessageSystems, Port25, Strongmail) allow senders to fine tune bounce handling, and designate how many times an email is retried, even allowing no retries on a temporary failure.
What if the missing mail is a result of senders aggressively handling 4xx messages? Some of the companies I’ve consulted for delete email addresses from mailing lists after 2 or 3 4xx responses. Other companies only retry for 12 – 24 hours and then the email is treated as hard bounced.
Return Path is reporting this as a delivery failure, and the tone of discussion I’m seeing seems to be blaming ISPs for overly aggressive spamfiltering. I don’t really think it’s entirely an ISP problem, though. I think it is indicative of poor practices on the part of senders. Not just the obvious permission and engagement issues that many senders deal with, but also poor policy on handling bounces. Perhaps the policy is fine, but the implementation doesn’t reflect the stated policy. Maybe they’re relying on defaults from their MTA vendor.
In any case, this is yet another example of how senders are in control of their delivery problems. Better bounce handling for temporary failures would lower the amount of email that never makes it to the ISP. This isn’t sufficient for 100% inbox placement, but if the email is never handed off to the ISP it is impossible for that email to make it to the inbox.

Read More

AOL transmitting 4xx error for user unknown

AOL is currently returning “451 4.3.0 <invaliduser@aol.com>: Temporary lookup failure” in some cases when they really mean “550 user unknown.” This message from AOL should be treated as 5xx failure and the message should not be retried (if at all possible) and the failure should be counted as a hard bounce for list management purposes.
This is something broken at AOL’s end, and the guys with the magic fingers that keep the system running are working to fix it. Right now there doesn’t seem to be an ETA on a fix, though.
Even if you are a sender who is able to stop the retries, you may see some congestion and delays when sending to AOL for the time being. Senders who don’t get the message, or who are unable to stop their MTAs from retrying 4xx mail will continue to attempt delivery of these messages until their servers time out. This may cause congestion for everyone and a noticeable  slowdown on the AOL MTAs.
AOL blog post on the issue
HT: Annalivia

Read More

Bounces, complaints and metrics

In the email delivery space there are a lot of numbers we talk about including bounce rates, complaint rates, acceptance rates and inbox delivery rates. These are all good numbers to tell us about a particular campaign or mailing list. Usually these metrics all track together. Low bounce rates and low complaint rates correlate with high delivery rates and high inbox placement.

Read More