How accurate are reports?

One of the big topics of discussion in various deliverability circles is the problems many places are seeing with delivery to Microsoft properties. One of the challenges is that Microsoft seems to be happy with how their filters are working, while senders are seeing vastly different data. I started thinking about reporting, how we generate reports and how do we know the reports are correct.

Everyone I know has bodged a SQL query at some point or another. I shared one of my scripts with Steve just recently and he pointed out that I left out a % so that one line wasn’t going to match. When I’m creating scripts, I check and compare them with manual queries and making sure the the right number of records are updated. But, apparently I missed this one query. What it does mean is all my reporting will be wrong. Now, in this case, it’s not a huge deal. The domain in question belonged to a free email provider acquired back in 2016 and who may or may not actually provide email services any longer.

Microsoft has its own history of problematic reporting in their SNDS product. They provide clear “red, yellow, green” coding of mail. According to their documentation red is definitely spam, green is definitely not spam and yellow is the 80% in the middle. Makes total sense and sounds awesome. The problem is that the colors seem to have no correlation to how mail is delivered. I’ve had clients with solid red and great inbox delivery and solid green and all their mail goes to spam. The reporting doesn’t match the behaviour. In fact, my go to answer for SNDS color questions is “the colors are a lie.”

Thing is, the colors have been a lie for as long as I’ve been using SNDS. I’ve told MS folks the colors are a lie. I’ve filed reports about the colors not accurately reflecting delivery. Most of the time the MS employees simply agree with me.

All of which led me to the place that maybe some of the problem with Microsoft is that some of their internal reporting is wrong. That what they’re seeing isn’t accurately reflecting what’s happening with delivery. Maybe someone bodged a SQL query but there’s no incentive to go back and check all the queries. When you’re monitoring delivery and filtering, there has to be reporting, there’s just too much information to go through it by hand.

The part about Microsoft is all rampant speculation on my part. But there is a lesson here for anyone working in “big data” – and these days email marketing and deliverability is big data. Regularly run your reports against a known data set, make sure it’s reporting what you think it is and that it’s giving you accurate information. Inaccurate reports are unactionable but unless you check you’d never know if your reporting was broken.

 

Related Posts

SNDS News

A number of people have mentioned over the last week or so that they’re seeing a lot of outages, failures and general ickiness with SNDS. I contacted Microsoft and asked about it. SNDS has been undergoing some upgrades and improvements and the outages were not intended to be end user visible. They’re going to keep a closer eye on things, while they finish the upgrades.
The good news in all of this is that SNDS is being upgraded and maintained. SNDS is still a functioning part of the Microsoft infrastructure, and this is good news for anyone who uses it as a data source.

Read More

Microsoft using Spamhaus Lists

An on the ball reader sent me a note today showing a bounce message indicating microsoft was rejecting mail due to a Spamhaus Blocklist Listing.
5.7.1 Client host [10.10.10.10] blocked using Spamhaus. To request removal from this list see http://www.spamhaus.org/lookup.lasso (S3130). [VE1EUR03FT043.eop-EUR03.prod.protection.outlook.com]
The IP in question is listed on the CSS, which means at a minimum Microsoft is using the SBL. I expect they’re actually using the ZEN list. ZEN provides a single lookup for 3 different lists: the SBL, XBL and PBL. The XBL is a list of virus infected machines and the PBL is a list of IPs that the IP owners state shouldn’t be sending email. Both of these lists are generally safe to use. If MS is using the SBL, it’s very likely they’re using the other two as well.
 

Read More

What's up with microsoft?

A c/p from an email I sent to a mailing list.
I think we’re seeing a new normal, or are still on the pathway to a new normal. Here’s my theory.
1) Hotmail made a lot of underlying code changes, learning from 2 decades of spam filtering. They had a chance to write a new codebase and they took it.
2) The changes had some interesting effects that they couldn’t test for and didn’t expect. They spent a month or two shaking out the effects and learning how to really use the new code.
3) They spent a month or two monitoring. Just watching. How are their users reacting? How are senders reacting? How are the systems handling everything?
3a) They also snagged test data along the way and started learning how their new code base worked and what it can do.
4) As they learned more about the code base they realized they can do different and much more sophisticated filtering.
5) The differences mean that some mail that was previously OK and making it to the inbox isn’t any longer.
5a) From Microsoft’s perspective, this is a feature not a bug. Some mail that was making it to the inbox previously isn’t mail MS thinks users want in their inbox. So they’re filtering it to bulk. I’ll also step out on a limb and say that most of the recipients aren’t noticing or caring about the missing mail, so MS sees no reason to make changes to the filters.
6) Expect at least another few rounds of tweak and monitor before things settle into something that changes more gradually.
Overall, I think delivery at Microsoft really is more difficult and given some of the statements coming out of MS (and some of the pointed silence) I don’t think they’re unhappy with this.

Read More