Recent Posts

Measuring open rate

In this part of my series on Campaign Stats and Measurements I will be examining open rates, how they are used, where they fail and how the can be effectively used.
There has been an lot written about open rates recently, but there are two posts that stand out to me. One was the EEC’s post on renaming open rate to render rate and Mark Brownlow’s excellent post on what open rate does and does not measure. I’ve also weighed in on the subject.
Overall, I find open rates to be a very frustrating metric. Some senders, particularly those relatively new to email marketing, are so sure they know what open rate is and what it means, that they don’t take any time to actually understand the number. While the name “open rate” seems self explanatory, it’s actually not. Open rate is actually not a measure of how many recipients open an email. However, there are times where open rate is a useful metric for measuring a marketing program over time.
What is an open?
If asked, most people will tell you that open rate is the number of emails that were opened by the recipients. The problem is that this isn’t actually true. An open is counted when a tagged image in an email is rendered by the recipient’s email client. Not all mail clients render images by default, but the emails are still available for the recipient to read. If a user clicks on a link in an email that has not had an image rendered, some ESPs count that as an open as well as a click. In other cases, visiting a link in an email with no image rendered is just a click, no open is recorded.
What is the open rate?
Open rate is generally the percentage of email opens divided by some number representing the number of emails sent. Many senders use the number of emails sent minus the number of bounced emails, others use just the number of emails sent without factoring in the number of emails bounced.
Open rate is a secondary metric. While it does not measure the success, or failure, of a campaign directly, it can be used as a indicator for campaigns. Many people use open rate as a metric because it’s easy to measure. Direct metrics, such as clicks or average purchase or total purchase, may take days or even weeks to collect and analyze. Open rates can be calculated quickly and easily.
What the open rate isn’t
Open rate is not a measure of how many people opened a mail. It is not a measure of how many people read a mail. It really only records that an image in a particular email is loaded and, sometimes, that a link was clicked on. Open rates can be wildly different depending on how the sender measures opens and how the sender measures sends.
What senders use open rates for
To compare their open rates with industry averages
As I talked about above, this use of open rates is problematic at best. You cannot compare numbers, even when they have the same name, if the numbers were arrived at using different calculations. Open rate is not open rate and unless you know the underlying algorithm used you cannot compare two open rates. This is a poor use of open rate.
As a metric for advertising rates
Since a sender can manipulate the open rate by using different calculation methods, this is a good metric for the advertiser to use. It is not so great for the purchaser though, who is at the mercy of the sender’s metrics. There are contractual ways a purchaser can protect herself from an unscrupulous marketer, but only if she understands how open rate can be manipulated and takes steps to define what open rate is in use.
To judge the success of campaigns over time
A single open data point doesn’t mean very much, however, using consistently measured open rates a sender can measure trends. Open trends over time are one area that open rates can help senders judge the success, or failure, of a marketing campaign.
As one metric in A/B testing
Comparing open rates in A/B testing gives some indication of which campaigns recipients may be more interested in. As with trends over time, the lone measurement isn’t useful, but as a comparative metric, it may provide senders with insight into a particular mailing.
To judge the engagement of recipients
Over the long term, recipients who do not interact with a mailing become dead weight on the list. Too many non responders can hurt a sender’s reputation at an ISP. List hygiene, in the form of removing people who never open or click on an email, is an important part of reputation management.
As metrics for email campaigns go, open rate is limited in what it measures about an email campaign. However, as a quick way to measure trending or do head to head comparisons it is a useful metric.

Verizon does not have a FBL

When I posted my initial cut of the ISP information page earlier this year, there was a comment asking about a Verizon FBL. At that time, I talked to some of the people-who-would-know over at Verizon and asked if they do have a FBL. The answer was a definite no.
For some reason, though, I continue to receive questions about the Verizon FBL. Based on the questions, the best I can extrapolate is that there is an ESP out there, somewhere, that states they have a Verizon FBL. It is possible, albeit unlikely, that they have a special agreement with Verizon. However, there is no generally available Verizon FBL.
If Verizon does make a FBL widely available, I will mention it here and update the ISP information page with the data. Until then, be very cautious with claims that there is a Verizon FBL.

Campaign stats and measurements

Do you know what your campaign stats mean? Do you know what it is that you’re measuring? I think there are a lot of emailers out there who have no idea what they are measuring and what those measurements mean.
The most common measurement used is “open rate.” There’s been quite a bit of discussion recently about open rates, how they’re calculated, and is there a better way. In my own opinion, open rate can be useful, but only in some circumstances. More often it is a distraction from real measurements
Not only has there been the recent discussions about “open rate” versus “render rate” and a lot of confusion among people about what the underlying issues are, but I’ve also been working through some campaign stats questions with other people that indicate maybe they don’t actually understand the numbers they’re using.
For instance, what do the delivery statistics reported by the various mailbox monitoring companies mean? If you have 100% inbox delivery as measured by the program, does that mean all your mail has reached the recipient’s inbox?
What about bounce rates? Everyone says “keep them low” but what does low mean? How do you measure them?
Over the next few posts, I’ll talk about the different stats and measurements in common use and what they do and don’t mean.

Deliverability versus delivery

Deliverability is a term so many people use every day, but what do we really mean when we use it? Is there an accepted definition of deliverability? Is the concept different than delivery?
At a recent conference I was running a session talking about email delivery, senders and the roles senders play in the email industry and at that particular organization. The discussion went on for a while, and the subject of deliverability versus delivery came up. J.D. Falk had a comment about the difference that resonated with me. Paraphrased, he said:

Open rates climbing, click rates dropping

Ken Magill reported on a study published by Epsilon (pdf link) on Tuesday. This report shows open rates are climbing but click-through rates are falling.

Three ways spammers get your address

Paul explains 3 ways spammers get your email address.

Privacy policies in court

Venkat has an analysis of a case where an individual provided a unique address to a vendor and that vendor released the address in violation of the posted privacy policy. The federal court rejected the suit due to the failure of the plaintiff to provide evidence of harm.
I posted last week about privacy policies and how often they are intentionally or unintentionally violated and when email addresses leak. Courts have consistently ruled against plaintiffs. It seems that the courts believe merely revealing information, even in contradiction to a posted privacy policy, is not actionable by the plaintiff.
As a consumer, I really don’t like the ruling. If a company is going to post a privacy policy, then they should follow it and if they don’t, I should be able to hold them responsible for their lies. Back in the land of reality, I am not surprised at the rulings. Individuals have never owned their personal information, it is the property of the people who compile and sell data
It does mean, however, that privacy polices are not worth the paper they’re written on.

I have an email delivery problem. Can you help?

I see a lot of requests for help with some sort of delivery problem, sent to me as an individual, sent to Laura as part of a consulting relationship, sent to ISPs, sent to organizations running blacklists or sent to industry mailing lists, both public and private.
Some of them could be done better. OK, most of them could be done better, some of them could be done a lot better. Here’s some things to do to get the best response from your delivery consultant, your fellow mailing list members or your ISP contact.
Be specific about what delivery problem you’re seeing
Are you seeing SMTP hard rejections? Are you seeing slow delivery (due to soft rejections, or connection level timeouts)? Or is the mail being delivered, but ending up in the bulk folder?
Bad question and answer: “Is anyone seeing delivery problems with Yahoo?” leads to “There’s a ‘y’ in the month, of course we’re seeing delivery problems at Yahoo.”
Better question for a useful answer: “I started to see a lot more soft bounces from Yahoo last Thursday, and our probe accounts there are seeing mail ending up in the bulk folder – is anyone else seeing this?”
Describe the symptoms
One of the most important things is to describe the symptoms you’re seeing, rather than just your guess at what the underlying problem is.
Bad question: “Our client, the Breast Cancer Foundation, is seeing delivery problems because of the word ‘breast’ – how do we get them whitelisted?”
Better question: “Our client, the Breast Cancer Foundation, is getting filed in the junk folder at AOL – what should we look at to work out why?”
Mention where you’re getting your information
Are you seeing detailed rejections in your mail logs, or just seeing your outbound mail queues growing? Are you seeing bulk folder delivery on your own email accounts at the receiver, or are you relying on data from a commercial mailbox monitoring company?
Provide the basic information about delivery that’s usually going to be needed

What IP address are you sending email from? What domain are you sending to? Or, if you’re contacting the ISP, what email address? What sort of mail are you sending? (If you’re contacting a blacklist or ISP, include the IP address in the subject line).
Are you the only person sending email from that IP address, or is it shared by other users?
If you’re getting rejections, bounces or deferrals, include the rejection message or bounce message.
How much mail are you sending to the domain where you’re seeing a problem? How much is being rejected or delayed?
What, if anything, have you changed recently? If you’re mailing for yourself, did you just start mailing a new bunch of contacts you’ve acquired (or discovered, in an old database)? If you’re an ESP, is this a new customer, or have they been mailing successfully for a while?
What are your typical complaint / feedback loop rates? What are your typical user unknown or rejection rates? If you’re monitoring inbox delivery, what are your inbox rates at this receiving domain?
Do you have a feedback loop or whitelist set up with this receiving domain? Are you using any sort of authentication (SPF, DKIM)?
Bad question: “I’m seeing a lot of bounces from AOL. Any idea why?”
Better question: “I’ve been sending mail without any problems for a while, but most of my AOL recipients have been rejected today, with the message 554 HVU:B1 http://postmaster.info.aol.com/errors/554hvub1.html. We don’t have a feedback loop or whitelist with AOL. What happened?”
Say what you’ve already done
If there’s a URL in the rejection message, did you click on it? Did the page it led to help?
Did you contact the receiving ISP already? What did they say?
Timeliness – there’s no such thing as a delivery emergency
Email delivery problems are never a life-or-death issue. Don’t demand immediate responses to email. Don’t follow up with a phone call five minutes after sending the email.
Conversely, if you saw a delivery problem three weeks ago which has since fixed itself then not only does nobody care, it’s likely that some of the information that would have helped diagnose the problem may no longer exist.
Appropriate communication channel

It’s very difficult to resolve a delivery issue by ‘phone. Partly that’s because you need to communicate some detailed information (like IP addresses) where even a single typo can make any analysis worthless, so cutting and pasting is the only way to avoid problems. But it’s also partly because it’s often something that can require significant work to analyse (checking databases, delivery logs, reputation sources, email structure, checking with other people working in the field), all of which is difficult to do while also dealing with someone on the ‘phone.
Avoid using IM for this too, unless you’ve been asked to do so. And if you’re using IM, offer to send the bulkier data (logs, IP addresses, sample messages) by email.
If you’re communicating with someone where answering your questions is part of their job description it’s even more important to use email, so that all the information is stored in their ticketing system – both for future reference by them or their colleagues, and also to make sure that their work is visible to their employer.
Respond appropriately to questions
It’s likely that whoever you’re asking may need additional information, such as a sample message. If they ask you for more information, respond with it as soon as possible, while they still have the issue in their mind.
Don’t argue that the information is irrelevant, or they don’t need to know it. Really, they wouldn’t ask if they didn’t think it might be relevant. If you can’t provide it (because you don’t have it, or for privacy or contract reasons) explain that, and see if there’s something else that might provide the same useful information.
Respond appropriately to answers
You asked the question because you didn’t know the answer, and you thought the person or group you asked might. So rejecting an answer you get just because it seems wrong to you or you don’t fully understand it is probably a bad idea.
I regularly see clients who are paying good money for extremely competent deliverabilty advice refuse to accept the answer they’re given – insisting that the problems can’t possibly be due to the content of the email, or the reputation of the sending IP address, or the types of links included in the email. Then, after wasting several (billable) hours arguing with their consultant they actually try the change suggested, and it helps with the problem.
If you’re asking a mailing list, rather than an individual, then it’s possible you will get some completely useless suggestions. It’s still better to politely explain why you think they don’t apply (and listen to any replies you get to that) than to reject or ignore them out of hand.
Courtesy
You’re asking for help. Even if you’re paying the person you’re asking for advice you’re still more likely to get service above and beyond if you’re pleasant to deal with. Wrap a little social lubricant around your question – a “Hi!”, “When you get a moment, could you take a look at…”, “Thanks!” goes a long way.
Be respectful of their time. If it’s not an urgent issue, let them know that. Try not to ramble.
When the problem is resolved, write and thank them for their effort. If you identified the problem and it wasn’t what they thought, explain briefly what the issue was. You never stop learning, and it’s possible that knowing that will help them elsewhere.
Be nice. Be brief. If you’re easy to deal with, you’ll get more helpful responses.

Open rates

Right now, there is no way to compare open rates as everyone calculates them differently. Mark Brownlow covers this today.

The not April Fools post

I thought for a while about putting up an April Fools post, but decided against it. However, today being April first, anything I post is going to be treated suspiciously.
So I decided to ask my readers: what would you like to have me blog about over the next few months? What topics have I touched on that you’d like me to explore deeper? Is there something I’ve not discussed that you would like me to post on?
Also, what information would make your job easier? I know my ISP info chart was tweeted and heavily linked to, is there other information I could put together that would be similarly useful?