What does open rate tell you

There has been an lot written about open rates in the past, but there are two posts that stand out to me. One was the EEC’s post on renaming open rate to render rate and Mark Brownlow’s excellent post on what open rate does and does not measure. I’ve also weighed in on the subject. The issue is still very confused.
If asked, most people will tell you that open rate is the number of emails that were opened by the recipient. The problem is that this isn’t actually true. Open rate is measured by the number of people that display an image in an email. Traditionally this has been a uniquely tagged 1×1 pixel, until some filters and mail clients stopped displaying 1×1 pixels. More recently, every image in an email is tagged, so opening one image would record as an open.
So open rate doesn’t actually tell a sender how many people opened and read an email. It really only records that an image in a particular email is loaded. It does not record when an email is opened. Some people don’t load images by default. Some people don’t load images at all, even when they open and actively read the text portion of the email.
Clearly, there are some uses for open rates. It can give a useful metric when comparing different forms of the same email (A/B testing) and when looking at user engagement over time. However, we have also recently seen that open rate is not predictive for click through rate.

Related Posts

Delivery Metrics

Last week ReturnPath published a study that shows 20% of permission based email fails to be delivered to the inbox. For this study, ReturnPath looked at the mail sent by their mailbox monitor customers and counted the number of deliveries to the inbox, the number of deliveries to the bulk folder and the number of emails that were not delivered.
At US ISPs 21% of the permission based emails sent to the ReturnPath probe network did not make it to the inbox. 3% of the emails sent went to the bulk folder and 17% did not make it to the mailbox at all.  MSN/Hotmail and Gmail were the worst ISPs to get mail to. They each failed to deliver more than 20% of the mail that was sent to them. At Canadian ISPs, even less of the mail made it to the inbox, primarily because primus.ca is such a large portion of the Canadian market and they use Postini as a filter. Postini is a quite aggressive filter and takes no feedback from senders.
ReturnPath’s take home message on the survey is that one set of metrics is not enough to effectively evaluate a marketing program. Senders need to know more about their mailings than they can discover from just the bounce rate or the revenue rate or response rate or open rate.
There are a lot of reasons an email doesn’t get to the recipient’s inbox or bulk folder. Mail can be hard blocked at the MTA, and rejected by the ISP outright. Mail can be soft blocked at the MTA and the ISP can slow down sending. Sometimes this is enough to cause the sending MTA to stop attempting to deliver the mail, thus causing mail to not show up. Both of these types of blocks are usually visible when looking at the bounce rate.
Some ISPs accept mail but then fail to deliver it to the recipient. Everything on the sender end says the ISP accepted it for delivery but the ISP just drops it on the floor. This is the type of block that a mailbox monitoring program is best able to identify.
Despite all the discussions of numbers, many marketers are still not measuring the variables in their email campaigns. Ken Magill wrote today about a study released by eROI that indicates more than a third of marketers are not doing any testing on their mailings.
Now, both of these studies are done in an attempt to sell products, however, the numbers discussed should be making smart senders think about what they are measuring in regards to their email campaign, how they are measuring those factors and what the measurements mean.

Read More

When an open is not a sign of interest

A lot of people, including myself, are using opens as one of the measures of engagement. This, as a general rule, is not a bad measure. However, there are people who will open email not because they’re interested in it, but because they know it is spam.
Take, for instance, the email address I acquired in 1993. Yes, I still have this address. I stopped using it to sign up for lists in 1999 and stopped using it for most of the rest of my mail around 2001. This address, though, is on any number of spam mailing lists. The spam that gets through is usually sent by hard-core spammers. The ISP that hosts that mailbox uses Communigate Pro to filter mail, so much of the casual spam is filtered.
Generally, if I open an email (and load images or click through) on that account it is only in order to track down a spammer. For instance, I’m getting a lot of spam there from affiliates offering me the opportunity to purchase printing services for a very low price. I have actually been opening the mail, and clicking through. But I’m not clicking through because I’m interested in purchasing. I’m clicking through to see if my reports to abuse@ printer are resulting in any action against the spammers. (They’re not).
The thing is, though, I know that by clicking through on ads, I’ve now been promoted by the spammer to the “clicks on emails! it’s a live address!” list. Which only means I’m going to get more spam from them. Lucky me.
Using clicks and opens as a measure of engagement isn’t necessarily bad. But when using them you have to understand the limitations of the measurement and that what you may think it’s telling you isn’t actually what it’s telling you.

Read More

Asking the right question

My job as a consultant does involve answering questions and solving problems. Often the most important, and most overlooked, thing that I do is change the question that clients are asking. It is not that this changes the problem or even, really, changes the solution. It does change how people think of the problem, and changing how they think of the problem drives better solutions.
This can be applied to the current Email Experience Council (EEC) discussion on metrics and defining what a render rate was. Loren has a post up today detailing a number of common email situations and explaining in which cases an email is counted as open and in which cases an email is counted as unopened.
Right now an open in email terms is actually quite simple: a tagged image on a remote webserver was loaded. That’s all an open is. It used to be that no one was blocking images by default, so this was actually quite an accurate way to measure how many people were opening and presumably reading an email (at least for people using mail clients that display HTML and images).
But, as spammers started including more and more explicit images in email, recipients started asking for images to be blocked. In response to recipient requests, ISPs started blocking images by default. No longer was open rate a measure of which recipients opened and read an email, it became a measure of something completely different.
The EEC has recognized this is a problem and have decided that standardization would be a solution. As the first step to standardization they have identified two problems: open rate isn’t calculated in any standard way and the resulting ratio doesn’t describe what most people think it describes. Their recent publication The Email Render Rate defines standard calculations for render rates. This way render rates as reported by different ESPs can be directly compared. Changing the name from open rate to render rate changes what most people expect that the term means. No longer is this a measure of how many recipients opened the mail, but rather it is a measure of how many email clients rendered the images in the mail.
Maybe a better solution could be arrived at by changing the question? Instead of “how can we standardize render rate?” perhaps they should ask the question: “What do people think they’re measuring when they talk about open rates?”
Once the “what?” question is answered, perhaps a good solution to the “how?” question will become more obvious.

Read More