How accurate are reports?

One of the big topics of discussion in various deliverability circles is the problems many places are seeing with delivery to Microsoft properties. One of the challenges is that Microsoft seems to be happy with how their filters are working, while senders are seeing vastly different data. I started thinking about reporting, how we generate reports and how do we know the reports are correct.

Everyone I know has bodged a SQL query at some point or another. I shared one of my scripts with Steve just recently and he pointed out that I left out a % so that one line wasn’t going to match. When I’m creating scripts, I check and compare them with manual queries and making sure the the right number of records are updated. But, apparently I missed this one query. What it does mean is all my reporting will be wrong. Now, in this case, it’s not a huge deal. The domain in question belonged to a free email provider acquired back in 2016 and who may or may not actually provide email services any longer.

Microsoft has its own history of problematic reporting in their SNDS product. They provide clear “red, yellow, green” coding of mail. According to their documentation red is definitely spam, green is definitely not spam and yellow is the 80% in the middle. Makes total sense and sounds awesome. The problem is that the colors seem to have no correlation to how mail is delivered. I’ve had clients with solid red and great inbox delivery and solid green and all their mail goes to spam. The reporting doesn’t match the behaviour. In fact, my go to answer for SNDS color questions is “the colors are a lie.”

Thing is, the colors have been a lie for as long as I’ve been using SNDS. I’ve told MS folks the colors are a lie. I’ve filed reports about the colors not accurately reflecting delivery. Most of the time the MS employees simply agree with me.

All of which led me to the place that maybe some of the problem with Microsoft is that some of their internal reporting is wrong. That what they’re seeing isn’t accurately reflecting what’s happening with delivery. Maybe someone bodged a SQL query but there’s no incentive to go back and check all the queries. When you’re monitoring delivery and filtering, there has to be reporting, there’s just too much information to go through it by hand.

The part about Microsoft is all rampant speculation on my part. But there is a lesson here for anyone working in “big data” – and these days email marketing and deliverability is big data. Regularly run your reports against a known data set, make sure it’s reporting what you think it is and that it’s giving you accurate information. Inaccurate reports are unactionable but unless you check you’d never know if your reporting was broken.

 

Related Posts

Targets and measures

Read More

OATH and Microsoft updates

I’ve seen multiple people asking questions about what’s going to happen with the Yahoo and AOL FBLs after the transition to the new Oath infrastructure. The most current information we have says that the AOL FBL (IP based) is going away. This FBL is handled by the AOL infrastructure. As AOL users are moved to the new infrastructure any complaints based on their actions will come through the Yahoo complaint feedback loop (CFL). The Yahoo CFL is domain based. Anyone who has not signed up for the Yahoo CFL should do so.
When registering you will need each domain and the selectors you’re planning on using. Yahoo will send an email with a confirmation link that needs to be clicked on within a short period of time in order to activate the FBL.
Microsoft’s SNDS program had an outage at the end of last week. That’s been fixed, but the missing data will not be back populated into the system. This has happened a couple times in the past. It seems the system gets a live feed of data. If, for some reason, the data is interrupted, then it’s gone and doesn’t get populated.

Read More

What kind of mail do filters target?

All to often we think of filters as a linear scale. There’s blocking on one end, and there’s an inbox on the other. Every email falls somewhere on that line.
Makes sense, right? Bad mail is blocked, good mail goes to the inbox. The bulk folder exists for mail that’s not bad enough to block, but isn’t good enough to go to the inbox.
Once we get to that model, we can think of filters as just different tolerances for what is bad and good. Using the same model, we can see aggressive filters block more mail and send more mail to bulk, while letting less into the inbox. There are also permissive filters that block very little mail and send most mail to the inbox.
That’s a somewhat useful model, but it doesn’t really capture the full complexity of filters. There isn’t just good mail and bad mail. Mail isn’t simply solicited or unsolicited. Filters take into account any number of factors before deciding what to do with mail.

Read More