Superstition, correlation and reality

I’m not a huge baseball fan, probably a side effect of growing up in a city with no MLB team. SF_giants_ImageBut I do enjoy the social aspects of rooting for local teams when they’re winning big games. Last night I was following the World Series score online and switched over to watch the last inning. I posted something about the game on FB just about 30 seconds before the Giant’s outfield bobbled what should have been a single (at best). I immediately posted an apology, “Sorry about that, shouldn’t have said anything!”
Do I really think that my post somehow cursed two outfielders and caused them to bobble a simple play? No, of course not. But it is a very human response. In fact, there’s an entire advertising campaign centered around the the weird things people do while watching sports.
There is a lot of superstition in email delivery, too. I think that’s a combination of filtering necessarily being a black box, human’s built in tendency to see patterns in random data, and a need to be able to control and affect outcomes.
Figuring out cause and effect in the real world is not trivial. In my research days we set out to control as many confounding factors as possible so we could demonstrate the cause and the effect. That’s really hard to do when you’re not at a lab bench. In the real world, we can’t always control things directly. Instead, we have to rely on statistics and representative (or non-representative) samples.
Delivery isn’t even close to a science and one of the major issues is that filters are always changing. I’ve certainly seen occasions where multiple clients, or colleagues, were having problems delivering to one ISP or another. One of my clients made a change and saw their delivery improve. They patted themselves on the back for figuring out the problem. At the same time, though, other folks saw their delivery improve without making any changes. I can’t always convince people that whatever they did had nothing to do with their delivery improving.
The flip side is I can’t always convince people to stop doing somethings that they don’t need to do. I see a lot of mail with both DomainKeys and DKIM signatures. In most cases both signatures have the same selectors. DomainKeys is deprecated. No one, and I mean no one with a modern email system, is checking DomainKeys without checking DKIM. Senders can safely stop signing with DomainKeys and have nothing happen. It doesn’t matter, lots of ESPs and sender sign with both. They’re not going to change it. I’ve had multiple groups tell me they’re afraid to stop signing because it might hurt their delivery.
The reality is I didn’t make the Giant’s outfield bobble the ball because I posted to FB that I was watching the bottom of the 9th inning. The reality is that DomainKeys is deprecated and there’s no benefit to signing with both DomainKeys and DKIM. The reality is we are humans and we are inherently superstitious. Most of the times our superstitions are harmless. But sometimes they cause us more work than we need to do and provide no tangible benefits.

Related Posts

Spam, Phish or Malware?

Some mornings I check mail from my phone. This showed up this morning.
PizzaHutMail
My first thought was “oh, no, Pizza Hut is spamming, wonder who sold them my address.”
Then I remembered that iOS is horrible and won’t show you anything other than the Friendly From and maybe it was some weird phishing scheme.
When I got to my real mail client I checked headers, and sure enough, it wasn’t really from Pizza Hut. I’m guessing actually malware, but I don’t have a forensics machine to click the link and I’m not doing it on anything I can’t wipe (and have isolated from the rest of my network).
The frustrating thing for me is that this is an authenticated email. It not from Pizza Hut, the address belongs to some company in France. Apparently, that company has had their systems cracked and malware sent through them. Fully authenticated malware, pretending to be Pizza Hut, and passing authentication on various devices.
Pizza Hut isn’t currently publishing a DMARC record, but in this case, a DMARC record for Pizza Hut wouldn’t matter. None of the email addresses in the headers point to Pizza Hut.
I spent last week listening to a lot of people discussing DMARC and authentication and protecting people from scams and headers. But those all the protocols in the world won’t protect against this kind of thing. Phishing and malware can’t be fixed by technology alone. Even if every domain on the planet published a p=reject policy, mail like this would still get through.
 
 
 

Read More

DKIM and DomainKeys, Spam and Ham

I’ve been preaching “DKIM is great! DomainKeys is obsolete, get rid of it!” for several years now. I thought I’d take a look at my mailbox and see who was using authentication.
I’ve divided this into “Ham” and “Spam”. Spam is, well, all the spam I’ve received over the past couple of years. Ham is the non-spam mail in my inbox, whether personal, business, bulk or transactional. I’ve excluded most of the discussion mailing lists I’m on (not least because many of them consist of people in the email industry or are email standards development mailing lists, so have email authentication levels that are way outside the norm).

Read More

Content based filtering

Content filtering is often hard to explain to people, and I’m not sure I’ve yet come up with a good way to explain it.
A lot of people think content reputation is about specific words in the message. The traditional content explanation is that words like “Free” or too many exclamation points in the subject line are bad and will be filtered. But it’s not the words that are the issue it’s that the words are often found in spam. These days filters are a lot smarter than to just look at individual words, they look at the overall context of the message.
ISP_tolerances
Even when we’re talking content filters, the content is just a way to identify mail that might cause problems. Those problems are evaluated the same way IP reputation is measured: complaints, engagement, bad addresses. But there’s a lot more to content filtering than just the engagement piece. What else is part of content evaluation?

Read More