How accurate are reports?

One of the big topics of discussion in various deliverability circles is the problems many places are seeing with delivery to Microsoft properties. One of the challenges is that Microsoft seems to be happy with how their filters are working, while senders are seeing vastly different data. I started thinking about reporting, how we generate reports and how do we know the reports are correct.

Everyone I know has bodged a SQL query at some point or another. I shared one of my scripts with Steve just recently and he pointed out that I left out a % so that one line wasn’t going to match. When I’m creating scripts, I check and compare them with manual queries and making sure the the right number of records are updated. But, apparently I missed this one query. What it does mean is all my reporting will be wrong. Now, in this case, it’s not a huge deal. The domain in question belonged to a free email provider acquired back in 2016 and who may or may not actually provide email services any longer.

Microsoft has its own history of problematic reporting in their SNDS product. They provide clear “red, yellow, green” coding of mail. According to their documentation red is definitely spam, green is definitely not spam and yellow is the 80% in the middle. Makes total sense and sounds awesome. The problem is that the colors seem to have no correlation to how mail is delivered. I’ve had clients with solid red and great inbox delivery and solid green and all their mail goes to spam. The reporting doesn’t match the behaviour. In fact, my go to answer for SNDS color questions is “the colors are a lie.”

Thing is, the colors have been a lie for as long as I’ve been using SNDS. I’ve told MS folks the colors are a lie. I’ve filed reports about the colors not accurately reflecting delivery. Most of the time the MS employees simply agree with me.

All of which led me to the place that maybe some of the problem with Microsoft is that some of their internal reporting is wrong. That what they’re seeing isn’t accurately reflecting what’s happening with delivery. Maybe someone bodged a SQL query but there’s no incentive to go back and check all the queries. When you’re monitoring delivery and filtering, there has to be reporting, there’s just too much information to go through it by hand.

The part about Microsoft is all rampant speculation on my part. But there is a lesson here for anyone working in “big data” – and these days email marketing and deliverability is big data. Regularly run your reports against a known data set, make sure it’s reporting what you think it is and that it’s giving you accurate information. Inaccurate reports are unactionable but unless you check you’d never know if your reporting was broken.

 

Mailbox providers
What’s a bounce?

Related Posts

Microsoft deprecating SmartScreen filters

At the beginning of the month Microsoft announced that they were deprecating the SmartScreen filters used by the desktop Microsoft mail clients. These are the filters used in Exchange and various version of Outlook mail. This is yet further consolidation of spam filtering between the Microsoft free webmail domains, Office365 hosted domains and self hosted Exchange servers.  The online services (hotmail.com, outlook.com, Office365, live.com, etc) have been  using these filters for a while. The big change now is that they’re being pushed down to Exchange and Outlook users not hosted on the Microsoft site.
EOP was developed for Outlook.com (and friends) as well as Office365 users. From Microsoft’s description, it sounds like the type of machine learning engine that many providers are moving to.
Microsoft has published quite a bit of information about these filters and how they work on their website. One of the best places to start is the Anti-spam Protection FAQ. Something senders should pay attention to is the final question on that page: “What are a set of best outbound mailing practices that will ensure that my mail is delivered?” Those are all things  deliverability folks recommend for good inbox delivery.
Poking around looking at the links and descriptions, there is a host of great information about spam filtering at Microsoft and how it works.
A page of note is their Exchange Online Protection Overview. This describes the EOP process and how the filters work.
MS_filterProcess

Read More

What kind of mail do filters target?

All to often we think of filters as a linear scale. There’s blocking on one end, and there’s an inbox on the other. Every email falls somewhere on that line.
Makes sense, right? Bad mail is blocked, good mail goes to the inbox. The bulk folder exists for mail that’s not bad enough to block, but isn’t good enough to go to the inbox.
Once we get to that model, we can think of filters as just different tolerances for what is bad and good. Using the same model, we can see aggressive filters block more mail and send more mail to bulk, while letting less into the inbox. There are also permissive filters that block very little mail and send most mail to the inbox.
That’s a somewhat useful model, but it doesn’t really capture the full complexity of filters. There isn’t just good mail and bad mail. Mail isn’t simply solicited or unsolicited. Filters take into account any number of factors before deciding what to do with mail.

Read More

What's up with microsoft?

A c/p from an email I sent to a mailing list.
I think we’re seeing a new normal, or are still on the pathway to a new normal. Here’s my theory.
1) Hotmail made a lot of underlying code changes, learning from 2 decades of spam filtering. They had a chance to write a new codebase and they took it.
2) The changes had some interesting effects that they couldn’t test for and didn’t expect. They spent a month or two shaking out the effects and learning how to really use the new code.
3) They spent a month or two monitoring. Just watching. How are their users reacting? How are senders reacting? How are the systems handling everything?
3a) They also snagged test data along the way and started learning how their new code base worked and what it can do.
4) As they learned more about the code base they realized they can do different and much more sophisticated filtering.
5) The differences mean that some mail that was previously OK and making it to the inbox isn’t any longer.
5a) From Microsoft’s perspective, this is a feature not a bug. Some mail that was making it to the inbox previously isn’t mail MS thinks users want in their inbox. So they’re filtering it to bulk. I’ll also step out on a limb and say that most of the recipients aren’t noticing or caring about the missing mail, so MS sees no reason to make changes to the filters.
6) Expect at least another few rounds of tweak and monitor before things settle into something that changes more gradually.
Overall, I think delivery at Microsoft really is more difficult and given some of the statements coming out of MS (and some of the pointed silence) I don’t think they’re unhappy with this.

Read More