Is your website up? Are you sure?

“What would you do for 25% more sales?”
It’s panicked gift-buying season, and I got mail this morning from Boutique Academia, part of their final push before Christmas.
Inbox__18_975_messages__26_unread_
They’re hoping for some Christmas sales in the next three days. They do make some lovely jewelry – ask Laura about her necklace some time – so I clicked on their mail.
Failed_to_open_page
That’s not good. I like Boutique Academia, and fixing email and dns problems is What We Do, so I took a look.
Safari isn’t quite as bad with not-exactly-truthful error messages as Internet Explorer, but I still don’t really trust it. Perhaps the problem is with the click-tracking domain in the email, rather than with boutiqueacademia.com? So I open the base page at http://boutiqueacademia.com, get redirected immediately to https://www.boutiqueacademia.com – which fails to load.
15542402_1501169219896451_6901276936993410491_n
OK, start with the basics. DNS.

~ ∙ dig www.boutiqueacaedmia.com
; <<>> DiG 9.8.3-P1 <<>> www.boutiqueacademia.com
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 19459
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;www.boutiqueacademia.com. IN A

That’s pretty authoritative (DNS joke … never mind). The response says “NXDOMAIN”, which is DNS-speak for “no such domain”. It means that there isn’t any DNS record for www.boutiqueacademia.com – no A record, nor any DNS record of any other type for that hostname. And it isn’t something that can be caused by a temporary network glitch – it means that the DNS server for the domain says authoritatively “there’s no DNS record”.
This is beginning to look familiar. Lets see what the authoritative DNS servers for boutiqueacademia.com have to say.

~$ dig +short boutiqueacademia.com ns
ns2.hostdns4u.com.
ns1.hostdns4u.com.
~$ dig @ns1.hostdns4u.com www.boutiqueacademia.com
; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> @ns1.hostdns4u.com www.boutiqueacademia.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4587
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;www.boutiqueacademia.com. IN A
;; ANSWER SECTION:
www.boutiqueacademia.com. 30 IN CNAME boutique-academia.myshopify.com.

There’s our problem. Boutique Academia are using hostdns4u.com to host the DNS for their domain – and hostdns4u.com are sending broken DNS responses. They’re responding with a CNAME to myshopify, the ecommerce site, but they’re also responding with the status “NXDOMAIN” – there’s no DNS record for this hostname.
That response violates the way DNS works. If it’s returning any matching DNS record – such as the CNAME – it should return a “NOERROR” status, not “NXDOMAIN”.
Where this gets interesting is when you ask what a DNS resolver will do with this invalid response. That’s not really defined, and different recursive resolvers will treat this response in different ways. Some will ignore the NXDOMAIN and return the CNAME record. Some will ignore the CNAME record and return NXDOMAIN.
That means that depending on which sort of DNS server your recipient is using everything may work perfectly, or any attempt to visit the website may give an error. If the DNS server you’re using is sloppy about NXDOMAINs you may never see the problem in your own web browser.
As far as I can tell this bug is specific to CNAME records, and exists at several DNS hosting companies.
The end result is that if you’re using an ecommerce or blog-hosting company that uses CNAMEs in your DNS zone to point to their servers (such as shopify or wordpress) and you’re  hosting your DNS with a company that has this bug then a significant fraction of the Internet will not be able to reach your website.
You’ll see their opens and clicks on your email. If you’re using pay-per-click advertising you’re paying for each customer. But they’ll never see your site.
I’m not sure what fraction of the Internet is using recursive resolvers that are strict about invalid CNAME responses, but I’d guess it’s at least 20%. If your website suffers from this issue then you could get 25% more sales by moving your DNS hosting to a company that doesn’t suffer from this issue.
(I have told Boutique Academia about this issue. Hostdns4u.com have a cPanel error page for their website, so I didn’t try to contact them.)

Related Posts

HE.net DNS problems

Hurricane Electric had a significant outage of their authoritative DNS servers this morning, causing them to return valid responses with no results for all(?) queries. This will have caused delivery problems for any mail going to domains using HE.net DNS – which will include some of their colocation customers, as well as users of their free services – but also will have caused reverse DNS to fail for most servers hosted by Hurricane Electric worldwide, so if any of your mail is being sent from HE hosted machines you may have seen problems.
(We’re HE customers so we noticed. Still happy with them as a vendor.)

Read More

DNS, SERVFAIL, firewalls and Microsoft

When you look up a host name, a mailserver or anything else there are three types of reply you can get. The way they’re described varies from tool to tool, but they’re most commonly referred to using the messages dig returns – NXDOMAIN, NOERROR and SERVFAIL.
NXDOMAIN is the simplest – it means that there’s no DNS record that matches your query (or any other query for the same host name).
NOERROR is usually what you’re hoping for – it means that there is a DNS record with the host name you asked about. There might be an exact match for your query, or there might not, you’ll need to look at the answer section of the response to see. For example, if you do “dig www.google.com MX” you’ll get a NOERROR response – because there is an A record for that hostname, but no answers because there’s no MX record for it.
SERVFAIL is the all purpose “something went wrong” response. By far the most common cause for it is that there’s something broken or misconfigured with the authoritative DNS for the domain you’re querying so that your local DNS server sends out questions and never gets any answers back. After a few seconds of no responses it’ll give up and return this error.
Microsoft
Over the past few weeks we’ve heard from a few people about significant amounts of delivery failures to domains hosted by Microsoft’s live.com / outlook.com, due to SERVFAIL DNS errors. But other people saw no issues – and even the senders whose mail was bouncing could resolve the domains when they queried Microsofts nameservers directly rather than via their local DNS resolvers. What’s going on?
A common cause for DNS failures is inconsistent data in the DNS resolution tree for the target domain. There are tools that can mechanically check for that, though, and they showed no issues with the problematic domains. So it’s not that.
Source ports and destination ports
If you’re even slightly familiar with the Internet you’ve heard of ports – they’re the numbered slots that servers listen on to provide services. Webservers listen on port 80, mailservers on port 25, DNS servers on port 53 and so on. But those are just the destination ports – each connection comes from a source port too (it’s the combination of source port and destination port that lets two communicating computers keep track of what data should go where).
Source ports are usually assigned to each connection pretty much randomly, and you don’t need to worry about them. But DNS has a history of the source port being relevant (it used to always use source port 53, but most servers have switched to using random source ports for security reasons). And there’s been an increasing amount of publicity about using DNS servers as packet amplifiers recently, with people being encouraged to lock them down. Did somebody tweak a firewall and break something?
Both source and destination ports range between 1 and 65535. There’s no technical distinction between them, just a common understanding that certain ports are expected to be used for particular services. Historically they’ve been divided into three ranges – 1 to 1023 are the “low ports” or “well known ports”, 1024-49151 are “registered ports” and 49152 and up are “ephemeral ports”. On some operating systems normal users are prevented from using ports less than 1024, so they’re sometimes treated differently by firewall configurations.
While source ports are usually generated randomly, some tools let you assign them by hand, including dig. Adding the flag -b "0.0.0.0#1337" to dig will make it send queries from  source port 1337. For ports below 1024 you need to run dig as root, but that’s easy enough to do.
A (slightly) broken firewall
“sudo dig -b "0.0.0.0#1024" live.com @ns2.msft.net” queries one of Microsofts nameservers for their live.com domain, and returns a good answer.
“sudo dig -b "0.0.0.0#1023" live.com @ns2.msft.net” times out. Trying other ports above and below 1024 at random gives similar results. So there’s a firewall or other packet filter somewhere that’s discarding either the queries coming from low ports or the replies going back to those low ports.
Older DNS servers always use port 53 as their source port – blocking that would have caused a lot of complaints.
But “sudo dig -b "0.0.0.0#53" live.com @ns2.msft.net” works perfectly. So the firewall, wherever it is, seems to block DNS queries from all low ports, except port 53. It’s definitely a DNS aware configuration.
DNS packets go through a lot of servers and routers and firewalls between me and Microsoft, though, so it’s possible it could be some sort of problem with my packet filters or firewall. Better to check.
“sudo dig -b "0.0.0.0#1000" google.com @ns1.google.com” works perfectly.
So does “sudo dig -b "0.0.0.0#1000" amazon.com @pdns1.ultradns.net“.
And “sudo dig -b "0.0.0.0#1000" yahoo.com @ns1.yahoo.com“.
The problem isn’t at my end of the connection, it’s near Microsoft.
Is this a firewall misconfiguration at Microsoft? Or should DNS queries not be coming from low ports (other than 53)? My take on it is that it’s the former – DNS servers are well within spec to use randomly assigned source ports, including ports below 1024, and discarding those queries is broken behaviour.
But using low source ports (other than 53) isn’t something most DNS servers will tend to do, as they’re hosted on unix and using those low ports on unix requires jumping through many more programming hoops and involves more security concerns than just limiting yourself to ports above 1023. There’s no real standard for DNS source port randomization, which is something that was added to many servers in a bit of a hurry in response to a vulnerability that was heavily publicized in 2008. Bind running on Windows seems to use low ports in some configurations. And even unix hosted nameservers behind a NAT might have their queries rewritten to use low source ports. So discarding DNS queries from low ports is one of the more annoying sorts of network bugs – one that won’t affect most people at all, but those it does affect will see it much of the time.
If you’re seeing DNS issues resolving Microsoft hosted domains, or you’re seeing patterns of unexpected SERVFAILs from other nameservers, check to see if they’re blocking queries from low ports. If they are, take a look and see what ranges of source ports your recursive DNS resolvers are configured to use.
(There’s been some discussion of this recently on the [mailop] mailing list.)

Read More

4 things spammers do legitimate marketers don't

I’ve never met a spammer that claims to be a spammer. Most that I’ve met claim to be legitimate marketers (or high volume email deployers). But there are things spammers do that I never expect to see a legitimate marketer doing.
I’ve written about these things throughout the blog (tag: TWSD), but it’s probably time to actually pull them together into a single post.

Read More