Pdns: qname-minimization + rfc8020 interactions with bad auths

[crami]

EDIT by @habbie: I'm reopening this ticket, not because there's anything that needs to be done in our code, but so we can track offending auths.

Short description

When qname-minimization is enabled in pdns-recursor 4.3.0-alpha3 the name global.aliexpress.com.gds.alibabadns.com does not resolve.

Environment

• Operating system: debian 10/buster
• Software version: recursor 4.3.0 alpha3
• Software source: PowerDNS repository

Steps to reproduce

• Enable qname-minimization in recursor.conf (qname-minimization=yes)
• dig global.aliexpress.com.gds.alibabadns.com.

Expected behaviour

; <<>> DiG 9.11.5-P4-5.1+b1-Debian <<>> global.aliexpress.com.gds.alibabadns.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57957
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;global.aliexpress.com.gds.alibabadns.com. IN A

;; ANSWER SECTION:
global.aliexpress.com.gds.alibabadns.com. 92 IN CNAME us1111.alicdn.com.edgekey.net.
us1111.alicdn.com.edgekey.net. 92 IN    CNAME   e1429.x.akamaiedge.net.
e1429.x.akamaiedge.net. 20  IN  A   2.20.217.209

;; Query time: 12 msec
;; SERVER: 193.5.68.1#53(193.5.68.1)
;; WHEN: Wed Oct 30 19:37:41 CET 2019
;; MSG SIZE  rcvd: 161

Actual behaviour

Result:

 <<>> DiG 9.11.5-P4-5.1+b1-Debian <<>> global.aliexpress.com.gds.alibabadns.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 1322
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;global.aliexpress.com.gds.alibabadns.com. IN A

;; AUTHORITY SECTION:
gds.alibabadns.com. 600 IN  SOA gdsns1.alibabadns.com. none. 2018122017 1800 600 3600 360

;; Query time: 486 msec
;; SERVER: 193.5.68.1#53(193.5.68.1)
;; WHEN: Wed Oct 30 19:37:52 CET 2019
;; MSG SIZE  rcvd: 116

[omoerbeek]

Can you try with nothing-below-nxdomain=no ?

[pieterlexis]

This domain is broken in the way that RFC 8020 prevents:

global.aliexpress.com.gds.alibabadns.com/CNAME: A query for global.aliexpress.com.gds.alibabadns.com results in a NOERROR response, while a query for its ancestor, com.gds.alibabadns.com, returns a name error (NXDOMAIN), which indicates that subdomains of com.gds.alibabadns.com, including global.aliexpress.com.gds.alibabadns.com, don't exist. (106.11.35.18, 140.205.67.254, 140.205.122.66, 198.11.138.254, UDP_-_EDNS0_4096_D_K)

see https://dnsviz.net/d/global.aliexpress.com.gds.alibabadns.com/dnssec/

[crami]

With nothing-below-nxdomain=no configured the lookup works.

[omoerbeek]

The summarize: domain is broken, the new RFC8020 functionality in alpha3 makes the recursor more strict. Even with no qname minimization, the recursor will report NXDOMAIN if one of the ancestors NXDOMAIN respones is cached.

[Habbie]

Another incident of this kind, with logs:

pdns_recursor[14351]: amp-api-search-edge.apps-lb.itunes-apple.com.akadns.net: Name ‘apps-lb.itunes-apple.com.akadns.net’ and below, is negatively cached via ‘akadns.net’ for another 180 seconds

[rgacogne]

Yet another one for buy.garmin.com. CNAMEd to buy.cdn.garmingtm.com. with an invalid NXDOMAIN returned for cdn.garmingtm.com. :

Oct 31 11:55:11 [1]    buy.cdn.garmingtm.com: Name 'cdn.garmingtm.com' and below, is negatively cached via 'garmingtm.com' for another 900 seconds

[Habbie]

Oct 31 11:55:11 [1] buy.cdn.garmingtm.com: Name 'cdn.garmingtm.com' and below, is negatively cached via 'garmingtm.com' for another 900 seconds

This is AWS.

[phonedph1]

There is some large? email provider with name server setups like:

e.starbucks.com. 32467 IN NS d.ns.e.starbucks.com. ; (Indeterminate) auth=1
d.ns.e.starbucks.com. 82132 IN A 216.15.189.58 ; (Indeterminate) auth=0

but ns.e.starbucks.com. nxdomain's. So far I've found 300 like this. I doubt these would cause much issues directly but I'm still trolling a cache dump to find any others.

[Habbie]

see https://dnsviz.net/d/global.aliexpress.com.gds.alibabadns.com/dnssec/

Or, as we log it:

global.aliexpress.com.gds.alibabadns.com: Name 'com.gds.alibabadns.com' and below, is negatively cached via 'gds.alibabadns.com' for another 600 seconds

[phonedph1]

I need to write better tooling for this, but these are some quick ones I found so far.

FB cdn: instagram.fada1-2.fna.fbcdn.net. (good) fada1-2.fna.fbcdn.net. (nx)
Netflix: probe-dradis-anycast.prod.ftl.netflix.com. (good) prod.ftl.netflix.com. (nx)
Yahoo: global-jpop.mail.gm0.yahoodns.net. (good) mail.gm0.yahoodns.net. (nx)
Apple: challenge.gc.fe.apple-dns.net. (good) gc.fe.apple-dns.net. (nx) <- r53 hosted though
Microsoft: fe2.update.microsoft.com.nsatc.net. (good) microsoft.com.nsatc.net. (nx)
WhatsApp/FB: media.fcxh1-1.fna.whatsapp.net. (good) fcxh1-1.fna.whatsapp.net. (nx)
3GPP (which are delegated to providers I believe): epdg.epc.mnc610.mcc302.pub.3gppnetwork.org. (good) epc.mnc610.mcc302.pub.3gppnetwork.org. (nx)

[Habbie]

I've emailed Davey Song at Alibaba about the alibabadns issues, as requested by him at OARC.

[Habbie]

Akamai has indicated they hope to fix their last ENT edge cases by January.

[Habbie]

3GPP (which are delegated to providers I believe)

Yes, indeed. Your example is Bell Canada at ns-tor.wireless.bell.ca. and ns-mtl.wireless.bell.ca.

[mnordhoff]

3gppnetwork.org is the magic of hundreds of independently operated sketchy GSLBs in one domain.

I think 1.1.1.1 might have disabled QNAME minimization on the whole domain after the first dozen reports of problems.

[omoerbeek]

I just confirmed that with

nothing-below-nxdomain=no
qname-minimization=yes

all mentioned names resolve with alpha3.

[phonedph1]

Some more common entries (one taken from each domain)

Original: nccih.sites.infr.search.usa.gov. found NX at: sites.infr.search.usa.gov.
Original: cds.n3g7v6y3.hwcdn.net. found NX at: n3g7v6y3.hwcdn.net.
Original: 1392045061.rsc.cdn77.org. found NX at: rsc.cdn77.org.
Original: ic-464dc900-04c477-2ustreamuhs.s.loris.llnwd.net. found NX at: s.loris.llnwd.net.
Original: cdn.static.17k.com.w.kunlungr.com. found NX at: static.17k.com.w.kunlungr.com.
Original: sjc03-public-node09.chat.services.video.ibm.com. found NX at: chat.services.video.ibm.com.
Original: static-nayax-com.nayax.netdna-cdn.com. found NX at: nayax.netdna-cdn.com.
Original: app.coocent.net.w.cdngslb.com. found NX at: coocent.net.w.cdngslb.com.
Original: lol.aci.game.qq.com. found NX at: aci.game.qq.com.
Original: img1.tbcdn.cn.danuoyi.tbcache.com. found NX at: tbcdn.cn.danuoyi.tbcache.com.
Original: img.cnu.cc.w.alikunlun.com. found NX at: cnu.cc.w.alikunlun.com.
Original: occ-0-33-3996.1.nflxso.net. found NX at: 1.nflxso.net.

Also things on: dnswl.org, spamhaus.org, spamcorp.net, uribl.com, mcafee.com. (DNS RBL type thing, real dumpster fire there)

Everything is bad.

[Habbie]

I've copied all examples except for @phonedph1's second list just above into https://github.com/dns-violations/dns-violations/pull/70/files where I'd like to track it instead.

[vttale]

I've let Akamai know.

[Habbie]

I've let Akamai know.

Thanks David, I've talked to them now!

[omoerbeek]

master contains a feature to only do 8020 on DNSSEC validated NXDs: #8511
So I'm closing this; it remains important to report 8020 violations to the responsible parties publishing zones and to mention them in the dns-violations repo mentioned above.