Following the SMTP rules

An old blog post from 2013, that’s still relevant today.
“Blocked for Bot-like Behavior”
An ESP asked about this error message from Hotmail and what to do about it.
“Bot-like” behaviour usually means the sending server is doing something that bots also do. It’s not always that they’re spamming, often it’s a technical issue. But the technical problems make the sending server look like a bot, so the ISP is not taking any chances and they’re going to stop accepting mail from that server.
If you’re an ESP what should you look for when tracking down what the problem is?
First make sure your server isn’t infected with anything and that you’re not running an open relay or proxy. Second, make sure your customers aren’t compromised or have had their accounts hijacked.
Then start looking at your configuration.
HELO/EHLO values

  • are you using a consistent one for each IP or are the values changing?
  • are you using a FQDN (fully qualified domain name, i.e., a.example.com) for the HELO?
  • are you using a bare IP for the HELO/EHLO?
  • does that FQDN match the rDNS of the IP?

IP issues

  • does your IP have rDNS?
  • does that rDNS in any way look like it might be dynamic?
  • how many IPs are you using to send the same message?

Header issues

  • are you using a consistent hostname in the envelope From?
  • is that hostname similar to the hostname in the IP address?
  • are you sending a MessageID?
  • are you authenticating?
  • is that authentication correct?

Content

  • are you sending small amounts of the same content over different IP addresses?
  • is the content you’re sending being sent by other entities?
  • are any of the URLs you’re linking to infected with anything?
  • are any of the URLs you’re linking to serving ads that might be spreading viruses?

Sending issues

  • are you closing connections promptly or are you holding them open?
  • are you opening connections from different IPs and sending the same content at the same time?

The major bot-like behaviours are sending small numbers of messages from many different IP addresses, and using bad HELO/EHLO values. Even small senders using shared pools can trigger this filter at Hotmail. Try not to split small volumes of mail over multiple IPs whenever possible and particularly when you are getting this error message.
 

Related Posts

Are botnets really the spam problem?

Over the last few years I’ve been hearing some people claim that botnets are the real spam problem and that if you can find a sender then they’re not a problem. Much of this is said in the context of hating on Canada for passing a law that requires senders actually get permission before sending email.
Botnets are a problem online. They’re a problem in a lot of ways. They can be used for denial of service attacks. They can be used to mine bitcoins. They can be used to host viruses. They can be used to send spam. They are a problem and a lot of people spend a lot of time and money trying to take down botnets.
For the typical end user, though, botnets are a minor contributor to spam in the inbox. Major ISPs, throughout the world, have worked together to address botnets and minimize the spam traffic from them. Those actions have been effective and many users never see botnet spam in their inbox, either because it’s blocked during send or blocked during receipt.
Most of the spam end users have to deal with is coming from people who nominally follow CAN SPAM. They have a real address at the bottom of the email. They’re using real ISPs or ESPs. They have unsubscribe links. Probably some of the mail is going to opt-in recipients. This mail is tricky, and expensive, to block, so a lot more of it gets through.
Much of this mail is sent by companies using real ISP connections. Brian Krebs, who I’ve mentioned before, wrote an article about one hosting company who previously supported a number of legal spammers. This hosting company was making $150,000 a month by letting customers send CAN SPAM legal mail. But the mail was unwanted enough that AOL blocked all of the network IP space – not just the spammer space, but all the IP space.
It’s an easy decision to block botnet sources. The amount of real mail coming from botnet space is zero. It’s a much bigger and more difficult decision to block legitimate sources of emails because there’s so much garbage coming from nearby IPs. What AOL did is a last resort when it’s clear the ISP isn’t going to stop spam coming out from their space.
Botnets are a problem. But quasi legitimate spammers are a bigger problem for filter admins and end users. Quasi legitimate spammers tend to hide behind ISPs and innocent customers. Some send off shared pools at ESPs and hide their traffic in the midst of wanted mail. They’re a bigger problem because the mail is harder to filter. They are bigger problems because a small portion of their recipients actually do want their mail. They’re bigger problems because some ISPs take their money and look the other way.
Botnets are easy to block, which makes them a solved problem. Spam from fixed IPs is harder to deal with and a bigger problem for endusers and filters.

Read More

Whirlwind that is M3AAWG

It’s been a great conference, and it’s only about half done. As is common at these conferences, I write down lots of things we should do and need to publish. The difference is now that we are growing I may have the time to put the polish on them and get them published.
Today’s keynote discussed the economics of botnet mitigation. Michel van Eeten from Delft University of Technology presented information compiled from some different datasets about botnets.
Good news
Botnet infection rates are relatively stable. They’ve not spiraled out of control like some people were predicting.
Interesting news
More than 50% of bot infections are contained on 50 ISPs in the entire world.
Bad news
Centers set up specifically to fix botnet infections don’t really have a big impact on infection cure rate.
Good news
ISP actions and walled gardens do have an impact on infection cure rates.
The biggest take away from the session is that ISPs are critical in both protecting from infection and helping users cure infection once it happens.

Read More

Confusing the engineers

We went camping last weekend with a bunch of friends. Had a great time relaxing on the banks of the Tuolumne River, eating way too much and visiting.
On Saturday I was wearing a somewhat geeky t-shirt. It said 554: abort mission. (Thank you MessageSystems). At some point on Saturday every engineer came up to me, read my shirt and then looked at me and said “That’s not HTTP.”
That lead to various discussions about how their junior engineers don’t actually know SMTP at all. Why? Because the SMTP libraries just work. Apparently the HTTP libraries aren’t that great, so folks have to learn more about HTTP to troubleshoot and use them.
I’m sure there’s a joke in there somewhere: A Kindle engineer, an Android engineer and a robot engineer walk into a campsite…
EmailFilters_boxes_forblogIt did leave me thinking, though, about how it’s not that easy to run your own mail server these days. Gone are the days when running your own server was cost effective and easy. These days, there is just too much spam coming in. Crafting filters is a skilled job. It’s not that hard to run good filters. But to run good filters takes time to do well.
There are also a lot of challenges to sending mail. One of the discussions I had at the campsite was how hard it was to configure outbound mail. The engineer was helping a friend set up a website and trying to get the website to send notifications to the friend. But without setting up authentication the mail kept silently failing.
Of course, we do run our own mail server. But it’s our job and, in many ways, it keeps us honest. We don’t run many filters meaning we see what spammers are doing and can use our own experiences to better understand what commercial filters are dealing with.
For most people, though, I really think using a service is the right solution. Find one with filters that meet your needs and just pay them to deal with the headache.
 

Read More