Dnscrypt-proxy: [org tld] Stops resolving after ~20 mins

[rprimus]

Tue Mar 26 10:41:57 GMT 2019

Setup:

hw:  Ubiquity ER-X EdgeOS v1.10.8
dnsmasq:
Dnsmasq version 2.78-23-g9e09429  Copyright (c) 2000-2017 Simon Kelley
Compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
config:
server=127.0.0.1#153

dnscrypt-proxy: v2.0.21
config:
listen_addresses = ['127.0.0.1:153', '[::1]:153']

Tests:

root@ubnt:/config/linux-mipsle# /config/linux-mipsle/dnscrypt-proxy -resolve www.cpan.org
Resolving [www.cpan.org]

Domain exists:  probably not, or blocked by the proxy
Canonical name: -
IP addresses:   -
TXT records:    -
root@ubnt:/config/linux-mipsle# ./dnscrypt-proxy -resolve bitbucket.org
Resolving [bitbucket.org]

Domain exists:  probably not, or blocked by the proxy
Canonical name: -
IP addresses:   -
TXT records:    -

root@ubnt:/config/linux-mipsle# ./dnscrypt-proxy -resolve www.google.com
Resolving [www.google.com]

Domain exists:  probably not, or blocked by the proxy
Canonical name: www.google.com.
IP addresses:   2a00:1450:4001:81a::2004, 172.217.168.228
TXT records:    -

root@ubnt:/config/linux-mipsle# ./dnscrypt-proxy -resolve inch.com
Resolving [inch.com]

Domain exists:  yes, 3 name servers found
Canonical name: inch.com.
IP addresses:   67.207.86.2
TXT records:    v=spf1 mx a ~all

---

root@ubnt:/config/linux-mipsle# dig -t ns -p 153 bitbucket.org

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> -t ns -p 153 bitbucket.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28325
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1252
;; QUESTION SECTION:
;bitbucket.org.         IN  NS

;; ANSWER SECTION:
bitbucket.org.      4307    IN  NS  ns-1305.awsdns-35.org.
bitbucket.org.      4307    IN  NS  ns-1746.awsdns-26.co.uk.
bitbucket.org.      4307    IN  NS  ns-445.awsdns-55.com.
bitbucket.org.      4307    IN  NS  ns-584.awsdns-09.net.

;; Query time: 50 msec
;; SERVER: 127.0.0.1#153(127.0.0.1)
;; WHEN: Tue Mar 26 11:45:35 2019
;; MSG SIZE  rcvd: 234

root@ubnt:/config/linux-mipsle# dig -t ns  bitbucket.org
;; Truncated, retrying in TCP mode.

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> -t ns bitbucket.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 28877
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;bitbucket.org.         IN  NS

;; Query time: 16 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Mar 26 11:45:54 2019
;; MSG SIZE  rcvd: 31

root@ubnt:/config/linux-mipsle# dig -p 153 www.cpan.org

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> -p 153 www.cpan.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39757
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.cpan.org.          IN  A

;; ANSWER SECTION:
www.cpan.org.       599 IN  CNAME   dualstack.osff.map.fastly.net.
dualstack.osff.map.fastly.net. 599 IN   A   151.101.30.217

;; Query time: 449 msec
;; SERVER: 127.0.0.1#153(127.0.0.1)
;; WHEN: Tue Mar 26 10:37:31 2019
;; MSG SIZE  rcvd: 141

---

root@ubnt:/config/linux-mipsle# dig www.cpan.org
;; Truncated, retrying in TCP mode.

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> www.cpan.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62976
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.cpan.org.          IN  A

;; Query time: 24 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Mar 26 10:37:58 2019
;; MSG SIZE  rcvd: 30

Logs:

dnscrypt-proxy.log:

[2019-03-26 11:18:14] [NOTICE] Source [public-resolvers.md] loaded
[2019-03-26 11:18:14] [NOTICE] dnscrypt-proxy 2.0.21
[2019-03-26 11:18:14] [NOTICE] Now listening to 127.0.0.1:153 [UDP]
[2019-03-26 11:18:14] [NOTICE] Now listening to 127.0.0.1:153 [TCP]
[2019-03-26 11:18:14] [NOTICE] Now listening to [::1]:153 [UDP]
[2019-03-26 11:18:14] [NOTICE] Now listening to [::1]:153 [TCP]
[2019-03-26 11:18:15] [NOTICE] [arvind-io] OK (crypto v2) - rtt: 175ms
[2019-03-26 11:18:15] [NOTICE] [bottlepost-dns-nl] OK (crypto v2) - rtt: 37ms
[2019-03-26 11:18:18] [INFO] [cloudflare] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
[2019-03-26 11:18:18] [NOTICE] [cloudflare] OK (DoH) - rtt: 34ms
[2019-03-26 11:18:18] [INFO] [cloudflare-ipv6] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
[2019-03-26 11:18:18] [NOTICE] [cloudflare-ipv6] OK (DoH) - rtt: 28ms
[2019-03-26 11:18:18] [NOTICE] [d0wn-is-ns2] OK (crypto v1) - rtt: 68ms
[2019-03-26 11:18:19] [NOTICE] [d0wn-tz-ns1] OK (crypto v1) - rtt: 166ms
[2019-03-26 11:18:19] [NOTICE] [d0wn-tz-ns1-ipv6] OK (crypto v1) - rtt: 195ms
[2019-03-26 11:18:19] [NOTICE] [de.dnsmaschine.net] OK (crypto v2) - rtt: 37ms
[2019-03-26 11:18:19] [NOTICE] [denise] OK (crypto v2) - rtt: 47ms
[2019-03-26 11:18:19] [NOTICE] [dnscrypt.ca-1] OK (crypto v1) - rtt: 90ms
[2019-03-26 11:18:19] [NOTICE] [dnscrypt.ca-1-ipv6] OK (crypto v1) - rtt: 185ms
[2019-03-26 11:18:19] [NOTICE] [dnscrypt.ca-2] OK (crypto v1) - rtt: 89ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.ca-2-ipv6] OK (crypto v1) - rtt: 184ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.eu-dk] OK (crypto v2) - rtt: 38ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.eu-dk-ipv6] OK (crypto v2) - rtt: 52ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.eu-nl] OK (crypto v2) - rtt: 30ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.me] OK (crypto v2) - rtt: 36ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.nl-ns0] OK (crypto v2) - rtt: 24ms
[2019-03-26 11:18:20] [NOTICE] [dnscrypt.nl-ns0-ipv6] TIMEOUT
[2019-03-26 11:18:21] [NOTICE] [dnscrypt.uk-ipv4] OK (crypto v2) - rtt: 60ms
[2019-03-26 11:18:21] [NOTICE] [dnscrypt.uk-ipv6] OK (crypto v2) - rtt: 17ms
[2019-03-26 11:18:24] [INFO] [doh-crypto-sx] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
[2019-03-26 11:18:24] [NOTICE] [doh-crypto-sx] OK (DoH) - rtt: 49ms
[2019-03-26 11:18:26] [INFO] [doh-crypto-sx-ipv6] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
[2019-03-26 11:18:26] [NOTICE] [doh-crypto-sx-ipv6] OK (DoH) - rtt: 42ms
[2019-03-26 11:18:31] [NOTICE] [edociccio] OK (crypto v2) - rtt: 59ms
[2019-03-26 11:18:31] [NOTICE] [ev-va] OK (crypto v2) - rtt: 167ms
[2019-03-26 11:18:31] [NOTICE] [ev-to] OK (crypto v2) - rtt: 101ms
[2019-03-26 11:18:31] [NOTICE] [freetsa.org] OK (crypto v1) - rtt: 187ms
[2019-03-26 11:18:33] [INFO] [gridns-jp] TLS version: 303 - Protocol: h2 - Cipher suite: 52392
[2019-03-26 11:18:33] [NOTICE] [gridns-jp] OK (DoH) - rtt: 260ms
[2019-03-26 11:18:33] [INFO] [gridns-jp-ipv6] TLS version: 303 - Protocol: h2 - Cipher suite: 52392
[2019-03-26 11:18:33] [NOTICE] [gridns-jp-ipv6] OK (DoH) - rtt: 264ms
[2019-03-26 11:18:35] [INFO] [gridns-sg] TLS version: 303 - Protocol: h2 - Cipher suite: 52392
[2019-03-26 11:18:35] [NOTICE] [gridns-sg] OK (DoH) - rtt: 265ms
[2019-03-26 11:18:35] [INFO] [gridns-sg-ipv6] TLS version: 303 - Protocol: h2 - Cipher suite: 52392
[2019-03-26 11:18:35] [NOTICE] [gridns-sg-ipv6] OK (DoH) - rtt: 266ms
[2019-03-26 11:18:36] [NOTICE] [ibksturm] OK (crypto v2) - rtt: 900ms
[2019-03-26 11:18:36] [INFO] [2.dnscrypt-cert.ipredator.se.] the key validity period for this server is excessively long (1095 days), significantly reducing reliability and forward security.
[2019-03-26 11:18:36] [NOTICE] [ipredator] OK (crypto v1) - rtt: 34ms
[2019-03-26 11:18:36] [NOTICE] [kenshiro] OK (crypto v2) - rtt: 31ms
[2019-03-26 11:18:36] [NOTICE] [lucenera] OK (crypto v2) - rtt: 50ms
[2019-03-26 11:18:36] [INFO] [2.dnscrypt-cert.opennic2.eth-services.de.] the key validity period for this server is excessively long (3650 days), significantly reducing reliability and forward security.
[2019-03-26 11:18:36] [NOTICE] [opennic-ethservices] OK (crypto v1) - rtt: 55ms
[2019-03-26 11:18:37] [INFO] [powerdns-doh] TLS version: 304 - Protocol: h2 - Cipher suite: 4867
[2019-03-26 11:18:37] [NOTICE] [powerdns-doh] OK (DoH) - rtt: 31ms
[2019-03-26 11:18:37] [NOTICE] [publicarray-au] OK (crypto v2) - rtt: 395ms
[2019-03-26 11:18:42] [NOTICE] [qag.me] TIMEOUT
[2019-03-26 11:18:42] [NOTICE] [qualityology.com] OK (crypto v2) - rtt: 177ms
[2019-03-26 11:18:42] [NOTICE] [scaleway-fr] OK (crypto v2) - rtt: 26ms
[2019-03-26 11:18:42] [NOTICE] [securedns] OK (crypto v1) - rtt: 30ms
[2019-03-26 11:18:42] [NOTICE] [securedns-ipv6] OK (crypto v1) - rtt: 28ms
[2019-03-26 11:18:42] [INFO] [securedns-doh] TLS version: 304 - Protocol:  - Cipher suite: 4867
[2019-03-26 11:18:42] [NOTICE] [securedns-doh] OK (DoH) - rtt: 101ms
[2019-03-26 11:18:43] [NOTICE] [soltysiak] OK (crypto v1) - rtt: 40ms
[2019-03-26 11:18:43] [NOTICE] [suami] OK (crypto v2) - rtt: 52ms
[2019-03-26 11:18:43] [NOTICE] [trashvpn.de] OK (crypto v2) - rtt: 32ms
[2019-03-26 11:18:43] [NOTICE] [ventricle.us] OK (crypto v2) - rtt: 117ms
[2019-03-26 11:18:43] [NOTICE] [zeroaim-ipv6] OK (crypto v2) - rtt: 33ms
[2019-03-26 11:18:43] [NOTICE] Server with the lowest initial latency: dnscrypt.uk-ipv6 (rtt: 17ms)
[2019-03-26 11:18:43] [NOTICE] dnscrypt-proxy is ready - live servers: 47

---

query.log: [dig www.cpan.org]

[2019-03-26 11:28:43]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:28:43]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:28:43]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:28:43]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:28:43]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:28:43]   127.0.0.1   org DNSKEY  PASS

---

query.log: [dig -p 153 www.cpan.org]

[2019-03-26 11:30:46]   127.0.0.1   www.cpan.org    A   PASS

---

query.log: [./dnscrypt-proxy -resolve www.cpan.org]

[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    NS  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    NS  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    NS  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    NS  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    A   PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    TXT PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    TXT PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    TXT PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   www.cpan.org    TXT PASS
[2019-03-26 11:33:08]   127.0.0.1   cpan.org    DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   org DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  AAAA    PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  A   PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  A   PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:08]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:08]   127.0.0.1   resolver.dnscrypt.info  A   PASS
[2019-03-26 11:33:08]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:09]   127.0.0.1   info    DNSKEY  PASS
[2019-03-26 11:33:09]   127.0.0.1   resolver.dnscrypt.info  A   PASS
[2019-03-26 11:33:09]   127.0.0.1   dnscrypt.info   DS  PASS
[2019-03-26 11:33:09]   127.0.0.1   info    DNSKEY  PASS

This problem started happening yesterday. At that time, I was using v2.0.19 and upgraded to v2.0.21.

For the above tests, I disabled all IPv4 and IPv6 blacklists setup on the router (ipset flush ...).

Tue Mar 26 13:12:18 GMT 2019

Before sending this, I rebooted router and all was well for 20 mins before org stopped resolving. I changed dns settings on laptop (macOS) to use local dnscrypt [unbound:53, dnscrupt-proxy:50]. Same symptoms. restarting all dns services made no difference -tested with dnscrypt-proxy -resolve openbsd.org.

openbsd.org

: ; dig openbsd.org

; <<>> DiG 9.10.6 <<>> openbsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 3616
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;openbsd.org.           IN  A

;; Query time: 149 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Mar 26 12:46:35 GMT 2019
;; MSG SIZE  rcvd: 40

➜  log
: ; dnscrypt-proxy -resolve openbsd.org
Resolving [openbsd.org]

Domain exists:  probably not, or blocked by the proxy
Canonical name: -
IP addresses:   -
TXT records:    -

I went through the config and turned caching off cache = false, restarted all dns services and org tld was once again resolving.

working openbsd.org

: ; dnscrypt-proxy -resolve openbsd.org
Resolving [openbsd.org]

Domain exists:  yes, 8 name servers found
Canonical name: openbsd.org.
IP addresses:   129.128.5.194
TXT records:    _globalsign-domain-verification=mVYWxIl-2ab_B1yPPFxEmDCLrBcl6ucouXJOU_P0_C v=spf1 mx a:lists.openbsd.org a:mail.openbsd.org a:cvs.openbsd.org a:shear.ucar.edu ~all
Resolver IP:    146.185.167.43 (securedns.eu.)

Questions:

1. Is the cache stored on filesystem? (This is the only reason I can think of for (mis)behaviour to survive a process restart.
2. Can the cache be queried?
3. What would cause this behaviour only for domains in the org tld?
4. How would you suggest I debug this further - should it happen again (with cache = true)?

Thanks.

[jedisct1]

Try refused_code_in_responses = true.

[rprimus]

Wed Mar 27 09:31:14 GMT 2019

refused_code_in_responses = true was set before having the problems with openbsd.org.

dnscrypt-proxy config:

: ; egrep -v '^$|^ *#' /usr/local/etc/dnscrypt-proxy.toml
listen_addresses = ['127.0.0.1:50', '[::1]:50']
max_clients = 250
ipv4_servers = true
ipv6_servers = true
dnscrypt_servers = true
doh_servers = true
require_dnssec = true
require_nolog = true
require_nofilter = true
disabled_server_names = []
force_tcp = false
timeout = 2500
keepalive = 30
refused_code_in_responses = true
log_level = 0
log_file = '/var/log/dnscrypt-proxy.log'
cert_refresh_delay = 240
fallback_resolver = '9.9.9.9:53'
ignore_system_dns = true
netprobe_timeout = 60
log_files_max_size = 10
log_files_max_age = 7
log_files_max_backups = 1
block_ipv6 = false
cache = false
cache_size = 512
cache_min_ttl = 600
cache_max_ttl = 86400
cache_neg_min_ttl = 60
cache_neg_max_ttl = 600
[query_log]
  file = '/var/log/dnscrypt-query.log'
  format = 'tsv'
[nx_log]
  file = '/var/log/dnscrypt-nx.log'
  format = 'tsv'
[blacklist]
[ip_blacklist]
[whitelist]
[schedules]
[sources]
  [sources.'public-resolvers']
  urls = ['https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v2/public-resolvers.md', 'https://download.dnscrypt.info/resolvers-list/v2/public-resolvers.md']
  cache_file = 'public-resolvers.md'
  minisign_key = 'RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3'
  refresh_delay = 72
  prefix = ''
[static]

[jedisct1]

Duplicate of #774

Fixed in 2.0.22.

[rprimus]

Mon Apr 1 08:20:47 BST 2019

Thanks for the fix. Could you answer the following (for future reference)?

1. What would cause this behaviour only for domains in the org tld?
2. How would you suggest I debug this further - should something similar happen again ?

[jedisct1]

Queries for just org (no domain at all, just org) never happen with real clients. This kind of query only happens when a recursive resolver is put in front of dnscrypt-proxy.

Which is why I never noticed that issue, and couldn't initially reproduce it.

When asking just for org and with DNSSEC enabled, the response is huge. It probably wasn't originally that big, but recently got bigger.

In fact, too big to fit in a normal, UDP packet, without some extra care. So, the proxy sends a truncated response, as defined in the DNS protocol. And when a truncated response is received, the client, or here, dnsmasq, should retry using TCP. Because TCP is slower, but can accept larger packets without special care.

The problem was the truncated response got cached. So even when you retried using TCP, you received a truncated response, and not the full response.

dnsmasq having its own cache, the actual issue is not straightforward to understand. You can restart dnscrypt-proxy, but dnsmasq still has the truncated response in its own cache. In this context, using ping or a web browser, sitting between these two layers to diagnose the behavior, is not very helpful as the side effects of that bug appear not well-defined and quite unpredictable.

By far the best way to debug this would have been to enable query logging in dnscrypt-proxy. You would have seen two queries for ORG (just ORG) in a row when the problem starts to happen.
Then, using a DNS client such as dig or drill, send a query for org directly to the proxy, not to dnsmasq.
You'll have seen a truncated response. Having known that would have made the issue obvious.
And if you send the same query over and over again, even over TCP, you would have seen that the same truncated response was sent.

[rprimus]

Mon Apr 1 10:49:33 BST 2019

@jedisct1

Thank you for the EXCELLENT explanation. As shown in the initial issue above (in the expandable sections), I did perform all the tests (hence concluding it was dnscrypt-proxy cache) - just not fully understanding the reasoning.

Cheers!