I can't click through if you don't exist

Recipients can’t click through if you don’t exist
A tale of misconfigured DNS wrecking someone’s campaign.
I got mail this morning from A Large Computer Supplier, asking me to fill in a survey about them. I had some feedback for them, mostly along the lines of “It’s been two decades since I bought anything other than rackmount servers from you, maybe I’m not a good advertising target for $200 consumer laptops?” so I clicked the link.
 
Failed_to_open_page
 
(I’ve replaced the real domain with survey.example.com in this post, to protect the innocent, but everything else is authentic).
That’s not good. The friendly error messages web browsers give sometimes hide the underlying problem, but that looks like a DNS problem. Did they do something stupid, like putting the wrong URL in the mail they sent?
 

~ ∙ host survey.example.com
Host survey.example.com not found: 3(NXDOMAIN)

 
“NXDOMAIN”. That means that there are no records in DNS for the hostname I looked up. From my part of the Internet, at least, that hostname doesn’t exist. I used to build DNS software, so I find the variety of ways in which people break their DNS interesting. Time to dig a little deeper.
 

~ ∙ host -t ns example.com
example.com name server ns2.dreamhost.com.
example.com name server ns1.dreamhost.com.
example.com name server ns3.dreamhost.com.
~ ∙ host survey.example.com ns1.dreamhost.com
Using domain server:
Name: ns1.dreamhost.com
Address: 66.33.206.206#53
Aliases:
Host survey.example.com not found: 3(NXDOMAIN)

 
Here I look up the authoritative servers for the domain, and find it’s hosted by dreamhost. Then I check the records at one of the authoritative servers, ns1.dreamhost.com, and it’s returning NXDOMAIN too. survey.example.com doesn’t exist. Oops.
I told a little fib
Except … I’ve not been entirely truthful about how I investigate DNS issues. “host” is a user-friendly tool, and it provides nice, brief output for normal queries, so it’s the tool I use when I’m showing queries to clients or putting them on the blog. But I’m a DNS geek, so the tool I actually use is “dig“. Dig is anything but user-friendly. The results it gives you aren’t really interpreted at all, just a human-readable representation of the raw DNS packets – verbose, with lots of output that doesn’t necessarily make sense unless you’re familiar with how DNS works under the covers ([rfc 1034] and [rfc 1035] if you really want to know).
This is what that last query looks like using dig:

~ ∙ dig @ns1.dreamhost.com survey.example.com
; <<>> DiG 9.8.3-P1 <<>> @ns1.dreamhost.com survey.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 54756
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;survey.example.com. IN A
;; ANSWER SECTION:
survey.example.com. 14400 IN CNAME example-com.surveygizmo.com.
;; AUTHORITY SECTION:
surveygizmo.com. 14400 IN SOA ns1.dreamhost.com. hostmaster.dreamhost.com. 2011072000 14618 1800 1814400 14400
;; Query time: 54 msec
;; SERVER: 66.33.206.206#53(66.33.206.206)
;; WHEN: Fri Oct 10 10:25:13 2014
;; MSG SIZE rcvd: 147

Well … now I’m interested. With dig we can see exactly what the response from the authoritative server is – and it’s very broken. It’s returning an NXDOMAIN response, saying definitively that there are no records of any type for survey.example.com of any type. But it’s also returning an answer record for survey.example.com – a CNAME that redirects to the survey vendor. That’s really not allowed.
I contacted the firm running the survey and gave them a heads-up that their DNS was broken – and they replied telling me that it was working fine for them. I wonder how that could be.
I have three different DNS resolvers on my network: PowerDNS, a very solid and standards-compliant resolver. BIND, the oldest resolver, often installed by default and full of both features and bugs. And also whatever embedded resolver Mikrotik appliances use, likely similar to the embedded resolvers used in consumer routers.
Lets see what that record looks like through different resolvers:
First, PowerDNS

~ ∙ dig @192.168.80.100 survey.example.com
; <<>> DiG 9.8.3-P1 <<>> @192.168.80.100 survey.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21305
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;survey.example.com. IN A
;; ANSWER SECTION:
survey.example.com. 14400 IN CNAME example-com.surveygizmo.com.
;; Query time: 194 msec
;; SERVER: 192.168.80.100#53(192.168.80.100)
;; WHEN: Fri Oct 10 14:18:41 2014
;; MSG SIZE rcvd: 93

 
PowerDNS returns what it received from the authoritative server – an NXDOMAIN and an answer. Most applications are going to see the NXDOMAIN and stop there, unable to resolve the hostname.
Secondly, Mikrotik

~ ∙ dig @192.168.80.1 survey.example.com
; <<>> DiG 9.8.3-P1 <<>> @192.168.80.1 survey.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 37002
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;survey.example.com. IN A
;; Query time: 114 msec
;; SERVER: 192.168.80.1#53(192.168.80.1)
;; WHEN: Fri Oct 10 14:18:01 2014
;; MSG SIZE rcvd: 36

 
The embedded resolver Mikrotik uses sees the NXDOMAIN response and provides just that, without the answer record.
 
And finally, BIND

steve@scratch:~$ dig @127.0.0.1 survey.example.com
; <<>> DiG 9.9.5-3-Ubuntu <<>> @127.0.0.1 survey.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1911
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 5
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;survey.arborvistallc.com. IN A
;; ANSWER SECTION:
survey.example.com. 14400 IN CNAME example-com.surveygizmo.com.
example-com.surveygizmo.com. 300 IN A 216.152.128.142
;; AUTHORITY SECTION:
surveygizmo.com. 172800 IN NS jack.ns.cloudflare.com.
surveygizmo.com. 172800 IN NS dina.ns.cloudflare.com.
;; ADDITIONAL SECTION:
dina.ns.cloudflare.com. 172800 IN A 173.245.58.107
dina.ns.cloudflare.com. 172800 IN AAAA 2400:cb00:2049:1::adf5:3a6b
jack.ns.cloudflare.com. 172800 IN A 173.245.59.121
jack.ns.cloudflare.com. 172800 IN AAAA 2400:cb00:2049:1::adf5:3b79
;; Query time: 868 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Oct 10 07:58:13 PDT 2014
;; MSG SIZE rcvd: 253

 
BIND handles it differently (and, I think, wrongly). It sees that there’s an answer, so it returns an answer, along with a lot of other related records. And it returns a NOERROR response, instead of the NXDOMAIN it received. Any client application, such as a web browser, will see that as a perfectly reasonable response, and clicking on the link will work, ending up at surveygizmo.
And the moral of this story is…
Steve gets overly excited by obscure DNS bugs, mostly.
But also, it’s possible to mess up your DNS records such that it will work perfectly for you, and some fraction of your recipients, while being broken for the rest of your recipients (anyone at an ISP not using BIND in this example). So if you get reports that your links aren’t working (or your SPF records or DKIM records are bad) don’t assume that it can’t be a DNS problem because it works correctly when you check them.

Related Posts

Internet Storm Center on the Spamhaus DOS

The Internet Storm Center (ISC) has a blog post up discussing the DOS attack against Spamhaus. They do confirm they saw traffic approaching 300Gbps against Spamhaus. They also point out that most people probably never knew.

Read More

Maybe the sky is only falling a little bit

There was quite a bit of breathless reporting last week about the DoS against Spamhaus and how it was large enough to break the Internet. As the postmortem has gone on, a few things are becoming clear.

Read More

More on the attack against Spamhaus and how you can help

While much of the attack against Spamhaus has been mitigated and their services and websites are currently up, the attack is still ongoing.  This is the biggest denial of service attack in history, with as much as 300 gigabits per second hitting Spamhaus servers and their upstream links.
This traffic is so massive, that it’s actually affecting the Internet and web surfers in some parts of the world are seeing network slowdown because of this.
While I know that some of you may be cheering at the idea that Spamhaus is “paying” for their actions, this does not put you on the side of the good. Spamhaus’ actions are legal. The actions of the attackers are clearly illegal. Not only is the attack itself illegal, but many of the sites hosted by the purported source of the attacks provide criminal services.
By cheering for and supporting the attackers, you are supporting criminals.
Anyone who thinks that an appropriate response to a Spamhaus listing is an attack on the very structure of the Internet is one of the bad guys.
You can help, though. This attack is due to open DNS resolvers which are reflecting and amplifying traffic from the attackers. Talk to your IT group. Make sure your resolvers aren’t open and if they are, get them closed. The Open Resolver Project published its list of open resolvers in an effort to shut them down.
Here are some resources for the technical folks.
Open Resolver Project
Closing your resolver by Team Cymru
BCP 38 from the IETF
Ratelimiting DNS
News Articles (some linked above, some coming out after I posted this)
NY Times
BBC News
Cloudflare update
Spamhaus dDOS grows to Internet Threatening Size
Cyber-attack on Spamhaus slows down the internet
Cyberattack on anti-spam group Spamhaus has ripple effects
Biggest DDoS Attack Ever Hits Internet
Spamhaus accuses Cyberbunker of massive cyberattack

Read More