Recent Posts

The data are what they are

I’ve had a lot less opportunity to blog at the recent M3AAWG conference than I expected. Some of it because of the great content and conversations. Another piece has to do with lack of time and focus to edit and refine a longer post prompted by the conference. The final issue is the confidential nature of what we talk about.
With that being said, I can talk about a discussion I had with different folks over the looking at A/B testing blog post from Mailchimp. The whole post is worth a quick read, but the short version is when you’re doing A/B testing, design the test so you’re testing the relevant outcomes. If you are looking for the best whatever to get engagement, then your outcome should be engagement. If you’re looking for the best thing to improve revenue, then test for revenue.
Of course, this makes perfect sense. If you do a test, the test should measure the outcome you want. Using a test that looks at engagement and hoping that translates to revenue is no better than just picking one option at random.
That particular blog post garnered a round of discussion in another forum where folks disagreed with the data. To listen to the posters, the data had to be wrong because it doesn’t conform to “common wisdom.” The fact that data doesn’t conform to common wisdom doesn’t make that data wrong. The data is the data. It may not answer the question the researcher thought they were asking. It may not conform to common wisdom. But barring fraud or massive collection error, the data are always that. I give Mailchimp the benefit of the doubt when it comes to how they collect data as I know they have a number of data scientists on staff. I’ve also talked with various employees about digging into their data.
At the same time the online discussion of the Mailchimp data was happening, there was a similar discussion happening at the conference. A group of researchers got together to ask a question. They did their literature review, they stated their hypothesis, they designed the tests, they ran the tests. Unfortunately, despite this all being done well, the data showed that their test condition had no effect. The data were negative. They asked the question a different way, still negative. They asked a third way and still saw no difference between the controls and the test.
They presented this data at the conference. Well, this data went against common wisdom, too, and many of the session participants challenged the data. Not because it was collected badly, it wasn’t, but because they wanted it to say something else. It was the conference session equivalent of data dredging or p-hacking.

Overall, the data collected in any test from a simple marketing A/B testing through to a phase III clinical trial, is the answer to the question you asked. But just having the data doesn’t always make the next step clear. Sometimes the question you asked isn’t what you tested. This doesn’t mean you can retroactively find signal in the noise.
Mailchimp’s research shows that A/B testing for open rates doesn’t have any affect on revenue. If your final goal is to know which copy or subject line makes more revenue, then you need to test for revenue. No amount of arguing is going to change that data.

UPDATE: Spike in Yahoo unknown users

I still don’t have any solid information on the cause of the Yahoo bounces. I do know that folks inside Yahoo are looking into the issue.
However, multiple people (including my clients) are reporting that the addresses that are bouncing have very recent click and open activity. Other reports say these addresses deliver on a resend.
It looks like my advice yesterday was incorrect. I’m currently telling clients to continue mailing addresses for the time being.

Possible spike in Yahoo unknown users

Multiple folks are mentioning seeing an increase in “user unknown” responses from Yahoo. Some people are discussing this with Yahoo.
~~Right now, best advice is to believe these are accurate user unknowns.~~ UPDATE: There is increasing evidence these are not valid user unknowns. See next post.

Speaking in June

ActiveCampaign is hosting their very first user conference in Chicago in June. I am honored to be a part of their speaker lineup.
Early bird registration only $450 through April 30.

Happy International Women's Day

It’s International Women’s Day, and I thought I’d take a moment to mention some of the many, many women who have inspired me and helped me along the way. Some of them work in deliverability and compliance. Others are business colleagues. Still others are cheerleaders and inspiration. All of them make the world a better place.

What does good IP Reputation get you?

Today I was discussing some mailing list posts with an ESP colleague. He was telling me some interesting numbers he’d collected from different IP pools they maintain. He was testing routing mail through IPs based on subscription process and routing based on engagement metrics. The data showed that inboxing rates were similar across the test groups. As he put it, “IP reputation didn’t have much impact on inbox delivery.”

I’m not surprised. I’ve been talking for a while about how IP reputation is less important in reaching the inbox. In fact, it was almost 5 years ago now that I wrote The Death of IP Based Reputation. I updated it in 2015 with Deliverability and IP Reputation. Overall, IP reputation is a much smaller piece of reaching the inbox now than it has been in the past. I’ve talked about the reasons for this in the above posts. The short version is:

And… we're back

There was an unexpected break in blogging over the last 2 weeks. Between M3AAWG, a week of house guests and some upcoming big changes I didn’t get much writing finished. I started, and am still working on, about half a dozen different posts.
Thanks for your patience, we’ll get back to our regularly scheduled writing soon.

2017 Deliverability Benchmark report

Return Path has released their 2017 Deliverability Benchmark Report. I haven’t had a chance to look at it, but did download it earlier today.
EContent has a summary of the article up, with the headline Research Finds Email Senders with Strong Subscriber Engagement Are Likely to See Less Email Delivered to Spam. Useful data points they pulled out include:

Lack of blogging, in 3 images

Metric Monetization

As a digital channel, email provides a lot of different metrics for marketers to use. Not only can marketers measure things like open and click rates, but they can tie these numbers back to a particular recipient. This treasure trove of information leads to obsessing over making the numbers look good. For good deliverability senders want low bounce rates, low spamtrap rates, and high engagement rates.
These metrics are important because they’re some of the things that filters look at when making delivery decisions. We care about this data because the receiver ISPs care about the data. The ISPs care about this data because they are characteristics of wanted and/or opt in email.

Over the past few years a number of companies sell services selling good metrics.