Pdns: Recursor returns ServFail for most domains, but not all

[kdex]

• Program: Recursor
• Issue type: Bug report

Short description

No matter which server is used in forward-zones, most recursive DNS queries end up with status ServFail, although they should have resolved just fine. I've tested against both the Google DNS as well as OpenDNS in the forward-zones option to make sure that the issue is with PowerDNS.

Environment

• Operating system: Arch Linux, 4.9.8-1-ARCH
• Software version: 4.0.4-2
• Software source: Official Arch Linux package (community/powerdns-recursor)

Steps to reproduce

pdns.conf

allow-recursion=127.0.0.1, ::1, 0.0.0.0/0
chroot=/var/empty
config-dir=/etc/powerdns
daemon=no
distributor-threads=3
guardian=no
local-port=53
loglevel=10
module-dir=/usr/lib/powerdns
setgid=nobody
setuid=nobody
socket-dir=/var/run
webserver=no

launch=gpgsql
gpgsql-host=127.0.0.1
gpgsql-dbname=pdns
gpgsql-user=pdns
gpgsql-password=secret
gpgsql-dnssec=yes

log-dns-queries=yes
recursor=127.0.0.1:5300

recursor.conf

forward-zones=.=208.67.222.222;208.67.220.220
local-port=5300
trace=on

1. Observe how digging github.com works fine:

$ dig @localhost github.com       

; <<>> DiG 9.11.0-P3 <<>> @localhost github.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65433
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;github.com.                    IN      A

;; ANSWER SECTION:
github.com.             148     IN      A       192.30.253.113
github.com.             148     IN      A       192.30.253.112

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Tue Feb 21 01:06:17 CET 2017
;; MSG SIZE  rcvd: 71

1. Digging google.com returns ServFail however:

$ dig @localhost google.com 

; <<>> DiG 9.11.0-P3 <<>> @localhost google.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47590
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.                    IN      A

;; Query time: 12 msec
;; SERVER: ::1#53(::1)
;; WHEN: Tue Feb 21 01:07:27 CET 2017
;; MSG SIZE  rcvd: 39

Expected behaviour

Digging google.com should have yielded an A-record IP with a non-ServFail status.

Actual behaviour

Digging google.com doesn't yield the correct answer, but instead a ServFail. This is true for many domains; this is very likely caused by the same reason that also makes the security polls fail (see log below).

• Examples for domains that can be resolved fine are amazon.com, fox.com and github.com.
• Examples for domains that can not be resolved are google.com, powerdns.com and xmpp.org.

Other information

Here's the output of $ sudo journalctl -xfeu pdns-recursor:

Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 1 [6/1] question for 'github.com|A' from 127.0.0.1
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Wants DNSSEC processing in query for A
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Looking for CNAME cache hit of 'github.com|CNAME'
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: No CNAME cache hit of 'github.com|CNAME' found
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: No cache hit for 'github.com|A', trying to find an appropriate NS record
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Cache consultations done, have 1 NS to contact
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com.: Nameservers: <empty>(0.00ms)
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Domain has hardcoded nameservers
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Resolved '.' NS (empty) to: 208.67.222.222, 208.67.220.220
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Trying IP 208.67.222.222:53, asking 'github.com|A'
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: Got 3 answers from (empty) (208.67.222.222), rcode=0 (No Error), aa=0, in 5ms
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: accept answer 'github.com|A|192.30.253.113' from '.' nameservers? 1 YES! - This answer was received from a server we forward to.
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: accept answer 'github.com|A|192.30.253.112' from '.' nameservers? 1 YES! - This answer was received from a server we forward to.
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: OPT answer '.' from '.' nameservers
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: determining status after receiving this packet
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: answer is in: resolved to '192.30.253.113|A'
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: answer is in: resolved to '192.30.253.112|A'
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 [6] github.com: status=got results, this level of recursion done
Feb 21 01:05:58 web pdns_recursor[18162]: Feb 21 01:05:58 1 [6/1] answer to question 'github.com|A': 2 answers, 1 additional, took 1 packets, 5.489 ms, 0 throttled, 0 timeouts, 0 tcp connections, rcode=0
Feb 21 01:06:17 web pdns_recursor[18162]: Feb 21 01:06:17 1 [7/1] question for 'github.com|A' from 127.0.0.1
Feb 21 01:06:17 web pdns_recursor[18162]: Feb 21 01:06:17 [7] github.com: Wants DNSSEC processing in query for A
Feb 21 01:06:17 web pdns_recursor[18162]: Feb 21 01:06:17 [7] github.com: Looking for CNAME cache hit of 'github.com|CNAME'
Feb 21 01:06:17 web pdns_recursor[18162]: Feb 21 01:06:17 [7] github.com: No CNAME cache hit of 'github.com|CNAME' found
Feb 21 01:06:17 web pdns_recursor[18162]: Feb 21 01:06:17 [7] github.com: Found cache hit for A: 192.30.253.113[ttl=148] 192.30.253.112[ttl=148]
Feb 21 01:06:17 web pdns_recursor[18162]: Feb 21 01:06:17 1 [7/1] answer to question 'github.com|A': 2 answers, 1 additional, took 0 packets, 0 ms, 0 throttled, 0 timeouts, 0 tcp connections, rcode=0
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 stats: 4 questions, 75 cache entries, 0 negative entries, 25% cache hits
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 stats: throttle map: 2, ns speeds: 3
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 stats: outpacket/query ratio 163%, 0% throttled, 0 no-delegation drops
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 stats: 0 outgoing tcp connections, 1 queries running, 0 outgoing timeouts
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 stats: 4 packet cache entries, 0% packet cache hits
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Wants DNSSEC processing in query for TXT
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Looking for CNAME cache hit of 'recursor-4.0.4.security-status.secpoll.powerdns.com|CNAME'
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: No CNAME cache hit of 'recursor-4.0.4.security-status.secpoll.powerdns.com|CNAME' found
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: No cache hit for 'recursor-4.0.4.security-status.secpoll.powerdns.com|TXT', trying to find an appropriate NS record
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Cache consultations done, have 1 NS to contact
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com.: Nameservers: <empty>(0.00ms)
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Domain has hardcoded nameservers
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Resolved '.' NS (empty) to: 208.67.222.222, 208.67.220.220
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Trying IP 208.67.222.222:53, asking 'recursor-4.0.4.security-status.secpoll.powerdns.com|TXT'
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: (empty) (208.67.222.222) returned a ServFail, trying sibling IP or NS
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Trying IP 208.67.220.220:53, asking 'recursor-4.0.4.security-status.secpoll.powerdns.com|TXT'
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: (empty) (208.67.220.220) returned a ServFail, trying sibling IP or NS
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: Failed to resolve via any of the 1 offered NS at level '.'
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 recursor-4.0.4.security-status.secpoll.powerdns.com: failed (res=-1)
Feb 21 01:06:59 web pdns_recursor[18162]: Feb 21 01:06:59 Could not retrieve security status update for '4.0.4' on 'recursor-4.0.4.security-status.secpoll.powerdns.com', RCODE = Server Failure
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 1 [8/1] question for 'google.com|A' from 127.0.0.1
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Wants DNSSEC processing in query for A
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Looking for CNAME cache hit of 'google.com|CNAME'
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: No CNAME cache hit of 'google.com|CNAME' found
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: No cache hit for 'google.com|A', trying to find an appropriate NS record
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Cache consultations done, have 1 NS to contact
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com.: Nameservers: <empty>(1.25ms)
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Domain has hardcoded nameservers
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Resolved '.' NS (empty) to: 208.67.222.222, 208.67.220.220
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Trying IP 208.67.222.222:53, asking 'google.com|A'
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: (empty) (208.67.222.222) returned a ServFail, trying sibling IP or NS
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Trying IP 208.67.220.220:53, asking 'google.com|A'
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: (empty) (208.67.220.220) returned a ServFail, trying sibling IP or NS
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: Failed to resolve via any of the 1 offered NS at level '.'
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 [8] google.com: failed (res=-1)
Feb 21 01:07:27 web pdns_recursor[18162]: Feb 21 01:07:27 1 [8/1] answer to question 'google.com|A': 0 answers, 1 additional, took 2 packets, 11.471 ms, 0 throttled, 0 timeouts, 0 tcp connections, rcode=2
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Wants DNSSEC processing in query for TXT
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Looking for CNAME cache hit of 'recursor-4.0.4.security-status.secpoll.powerdns.com|CNAME'
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: No CNAME cache hit of 'recursor-4.0.4.security-status.secpoll.powerdns.com|CNAME' found
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: No cache hit for 'recursor-4.0.4.security-status.secpoll.powerdns.com|TXT', trying to find an appropriate NS record
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Cache consultations done, have 1 NS to contact
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com.: Nameservers: <empty>(0.00ms)
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Domain has hardcoded nameservers
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Resolved '.' NS (empty) to: 208.67.222.222, 208.67.220.220
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Trying IP 208.67.222.222:53, asking 'recursor-4.0.4.security-status.secpoll.powerdns.com|TXT'
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: (empty) (208.67.222.222) returned a ServFail, trying sibling IP or NS
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Trying IP 208.67.220.220:53, asking 'recursor-4.0.4.security-status.secpoll.powerdns.com|TXT'
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: (empty) (208.67.220.220) returned a ServFail, trying sibling IP or NS
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: Failed to resolve via any of the 1 offered NS at level '.'
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 recursor-4.0.4.security-status.secpoll.powerdns.com: failed (res=-1)
Feb 21 01:08:18 web pdns_recursor[18162]: Feb 21 01:08:18 Could not retrieve security status update for '4.0.4' on 'recursor-4.0.4.security-status.secpoll.powerdns.com', RCODE = Server Failure

[phonedph1]

Your config is incorrect. forward-zones will send without the RD bit set, which will give you some very hit and miss results when querying upstream. Try adding the + symbol or using forward-zones-recurse instead and see if the problem persists.

[kdex]

forward-zones-recurse seems to work fine — thank you.

[RocketLR]

Your config is incorrect. forward-zones will send without the RD bit set, which will give you some very hit and miss results when querying upstream. Try adding the + symbol or using forward-zones-recurse instead and see if the problem persists.

YES!!!! When i read the setup guides, they said that it didnt matter where you placed the forward zones.. What a big fat LIE!!!
Thank you!!!!