FTL 🚀 - FTL crashed (in_zone+0x22)

the last pihole.log entries:

May 12 12:53:15 dnsmasq[709]: forwarded mel-efz.office.com to 2606:4700:4700::1001
May 12 12:53:15 dnsmasq[32471]: query[AAAA] mel-efz.office.com from 2001:xxx:xxx:0:3c56:f617:6559:9855
May 12 12:53:15 dnsmasq[32471]: forwarded mel-efz.office.com to 2606:4700:4700::1001
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:1::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:14::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:1000::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:15::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:29::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:2a::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:28::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:2b::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:816::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:860::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:200:23::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:200:21::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:200:2e::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:200:20::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:200:2d::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:200:2c::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:83e::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:815::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:82a::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:2c::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:201:2d::2
May 12 12:53:15 dnsmasq[32471]: reply mel-efz.office.com is 2603:1016:402:801::2
May 12 12:53:16 dnsmasq[709]: query[A] outlook.ms-acdc.office.com from 2001:xxx:xxx:0:3c56:f617:6559:9855
May 12 12:58:51 dnsmasq[534]: started, version pi-hole-2.81 cachesize 10000

ebugakov on 12 May 2020

This issue happened deeply inside dnsmasq code. Is the crash reproducible?

DL6ER on 12 May 2020

This issue happened deeply inside dnsmasq code. Is the crash reproducible?

not sure I can reproduce it. never seen this behavior before upgraded to 0.5 (with 0.47 it was stable for weeks)

ebugakov on 12 May 2020

We updated the embedded dnsmasq to a version which has not yet been widely deployed elsewhere. It is very likely that the bug is new.

If you can reproduce it, we need to reproduce it in debugging mode. Please see https://docs.pi-hole.net/ftldns/debugging/ for further instructions. Without this (and when I cannot reproduce this locally), we will not be able to identify the solution (or at least a workaround).

DL6ER on 12 May 2020

got another crash, looks like the previous one

[2020-05-14 10:43:15.778 8568] Note: FTL forked to handle TCP requests
[2020-05-14 10:43:15.946 534] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-14 10:43:15.946 534] ----------------------------> FTL crashed! <----------------------------
[2020-05-14 10:43:15.946 534] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-14 10:43:15.946 534] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-14 10:43:15.946 534] and include in your report already the following details:
[2020-05-14 10:43:15.946 534] FTL has been running for 164664 seconds
[2020-05-14 10:43:15.946 534] FTL branch: master
[2020-05-14 10:43:15.946 534] FTL version: v5.0
[2020-05-14 10:43:15.946 534] FTL commit: 3d7c095
[2020-05-14 10:43:15.946 534] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-14 10:43:15.946 534] FTL user: started as pihole, ended as pihole
[2020-05-14 10:43:15.946 534] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-05-14 10:43:15.946 534] Received signal: Segmentation fault
[2020-05-14 10:43:15.946 534] at address: (nil)
[2020-05-14 10:43:15.946 534] with code: Unknown (128)
[2020-05-14 10:43:15.946 534] Backtrace:
[2020-05-14 10:43:15.946 534] ------ Listing content of directory /dev/shm ------
[2020-05-14 10:43:15.946 534] File Mode User:Group Filesize Filename
[2020-05-14 10:43:15.946 534] rwxrwxrwx root:root 260 .
[2020-05-14 10:43:15.946 534] rwxr-xr-x root:root 3K ..
[2020-05-14 10:43:15.946 534] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-14 10:43:15.946 534] rw------- pihole:pihole 258K FTL-dns-cache
[2020-05-14 10:43:15.946 534] rw------- pihole:pihole 12K FTL-overTime
[2020-05-14 10:43:15.946 534] rw------- pihole:pihole 8M FTL-queries
[2020-05-14 10:43:15.946 534] rw------- pihole:pihole 4K FTL-upstreams
[2020-05-14 10:43:15.946 534] rw------- pihole:pihole 61K FTL-clients
[2020-05-14 10:43:15.947 534] rw------- pihole:pihole 295K FTL-domains
[2020-05-14 10:43:15.947 534] rw------- pihole:pihole 233K FTL-strings
[2020-05-14 10:43:15.947 534] rw------- pihole:pihole 12 FTL-settings
[2020-05-14 10:43:15.947 534] rw------- pihole:pihole 124 FTL-counters
[2020-05-14 10:43:15.947 534] rw------- pihole:pihole 48 FTL-lock
[2020-05-14 10:43:15.947 534] ---------------------------------------------------
[2020-05-14 10:43:15.947 534] Thank you for helping us to improve our FTL engine!
[2020-05-14 10:43:15.947 534] FTL terminated!

May 14 10:43:15 dnsmasq[8572]: query[AAAA] hel-efz.office.com from 2001:xxx:xxx:0:f9df:81ee:8e5e:e0fd
May 14 10:43:15 dnsmasq[8572]: forwarded hel-efz.office.com to 1.1.1.1
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:ca::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:b5::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:a3::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:28::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:2a::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:7d::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:53::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:91::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:92::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:15::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:63::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:52::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:29::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:7c::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:14::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:90::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:16::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:a2::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:67::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:dd::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:cb::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:78::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:f6::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:8b::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:3c::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:c9::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:f4::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:f2::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:51::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:14::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:3d::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:7:66::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:64::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:15::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:f3::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:b6::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:3c::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:f5::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:de::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:df::2
May 14 10:43:15 dnsmasq[8572]: reply hel-efz.office.com is 2603:1026:3:79::2

ebugakov on 14 May 2020

And again this strange *.office.com domain.

simphide on 14 May 2020

There is a certain chance that I can discover the bug by staring at the code for long enough even when I can not reproduce this bug myself. However, for this, I need a bugtrace showing me at which of the tens of thousands of code lines to stare.

I tried to reproduce it will all information I have so far (same domain, DNS upstream server 1.1.1.1, querying an AAAA record over IPv6).

With DNSSEC

May 15 00:31:55 dnsmasq[28620]: query[AAAA] hel-efz.office.com from fe80::9c:8d:5c:53
May 15 00:31:55 dnsmasq[28620]: forwarded hel-efz.office.com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DS] com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DNSKEY] . to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: reply . is DNSKEY keytag 48903, algo 8
May 15 00:31:55 dnsmasq[28620]: reply . is DNSKEY keytag 20326, algo 8
May 15 00:31:55 dnsmasq[28620]: reply com is DS keytag 30909, algo 8, digest 2
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DS] office.com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DNSKEY] com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: reply com is DNSKEY keytag 39844, algo 8
May 15 00:31:55 dnsmasq[28620]: reply com is DNSKEY keytag 30909, algo 8
May 15 00:31:55 dnsmasq[28620]: reply office.com is no DS
May 15 00:31:55 dnsmasq[28620]: validation result is INSECURE
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:92::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:15::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:63::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:52::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:29::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:7c::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:14::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:90::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:91::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:a2::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:67::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:dd::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:cb::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:79::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f6::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:8b::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:3c::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:c9::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f4::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f2::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:51::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:14::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:3d::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:66::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:64::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:15::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f3::2

Without DNSSEC

May 15 00:33:23 dnsmasq[28731]: query[AAAA] hel-efz.office.com from fe80::9c:8d:5c:53
May 15 00:33:23 dnsmasq[28731]: forwarded hel-efz.office.com to 1.1.1.1
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:15::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f3::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:b6::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:3c::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f5::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:de::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:df::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:78::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:ca::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:b5::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:a3::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:28::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:2a::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:7d::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:53::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:91::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:92::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:15::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:63::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:52::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:29::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:7c::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:14::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:90::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:16::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:a2::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:67::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:dd::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:cb::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:79::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f6::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:8b::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:3c::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:c9::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f4::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f2::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:51::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:14::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:3d::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:66::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:64::2

No crash happened for me.

@simphide @ebugakov @takerukoushirou Are you using a specific software during / before the crashes or are you using the online platform office.com itself? I have no Windows machines at hand, however, I somewhat doubt it could be related to the operating system of the requestor.

DL6ER on 15 May 2020

I'm using WireGuard and https://github.com/stevejenkins/pihole-cloudsync parallel to pi-hole.

the crash happend 50 minutes after pihole-cloudsyncs last pull, so I don't think that this was the reason for the crash.

simphide on 15 May 2020

There is a certain chance that I can discover the bug by staring at the code for long enough even when I can not reproduce this bug myself. However, for this, I need a bugtrace showing me at which of the tens of thousands of code lines to stare.

I tried to reproduce it will all information I have so far (same domain, DNS upstream server 1.1.1.1, querying an AAAA record over IPv6).

With DNSSEC

May 15 00:31:55 dnsmasq[28620]: query[AAAA] hel-efz.office.com from fe80::9c:8d:5c:53
May 15 00:31:55 dnsmasq[28620]: forwarded hel-efz.office.com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DS] com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DNSKEY] . to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: reply . is DNSKEY keytag 48903, algo 8
May 15 00:31:55 dnsmasq[28620]: reply . is DNSKEY keytag 20326, algo 8
May 15 00:31:55 dnsmasq[28620]: reply com is DS keytag 30909, algo 8, digest 2
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DS] office.com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: dnssec-query[DNSKEY] com to 1.1.1.1
May 15 00:31:55 dnsmasq[28620]: reply com is DNSKEY keytag 39844, algo 8
May 15 00:31:55 dnsmasq[28620]: reply com is DNSKEY keytag 30909, algo 8
May 15 00:31:55 dnsmasq[28620]: reply office.com is no DS
May 15 00:31:55 dnsmasq[28620]: validation result is INSECURE
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:92::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:15::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:63::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:52::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:29::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:7c::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:14::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:90::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:91::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:a2::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:67::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:dd::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:cb::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:79::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f6::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:8b::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:3c::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:c9::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f4::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f2::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:51::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:14::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:3d::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:7:66::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:64::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:15::2
May 15 00:31:55 dnsmasq[28620]: reply hel-efz.office.com is 2603:1026:3:f3::2

Without DNSSEC

May 15 00:33:23 dnsmasq[28731]: query[AAAA] hel-efz.office.com from fe80::9c:8d:5c:53
May 15 00:33:23 dnsmasq[28731]: forwarded hel-efz.office.com to 1.1.1.1
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:15::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f3::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:b6::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:3c::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f5::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:de::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:df::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:78::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:ca::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:b5::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:a3::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:28::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:2a::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:7d::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:53::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:91::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:92::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:15::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:63::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:52::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:29::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:7c::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:14::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:90::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:16::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:a2::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:67::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:dd::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:cb::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:79::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f6::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:8b::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:3c::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:c9::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f4::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:f2::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:51::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:14::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:3d::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:7:66::2
May 15 00:33:24 dnsmasq[28731]: reply hel-efz.office.com is 2603:1026:3:64::2

No crash happened for me.

@simphide @ebugakov @takerukoushirou Are you using a specific software during / before the crashes or are you using the online platform office.com itself? I have no Windows machines at hand, however, I somewhat doubt it could be related to the operating system of the requestor.

I don't have windows machines in the network either. I guess this was occasional synchronization of states from something link calendar or outlook running on one of the devices.

ebugakov on 15 May 2020

Hi,
I'm also having issues with FTL since updating everything to v5.0 a few days ago. When opening Pi-hole it will say "Lost connection to API" across the colored stat boxes. I'll try to collect all necessary debug informations. As of reproducibility I can't say what causes this issue and like the others already said, it happens kinda random and will make all devices in the network unable to connect to the internet (Android: Wifi bars with an exclamation mark).
On top of that my Raspberry Pi 1B (running Pi-hole) is most of the time under heavy load (Load: 1.38 1.04 0.85).

pi@raspberrypi:~ $ neofetch
  .',;:cc;,'.    .,;::c:,,.    pi@raspberrypi
 ,ooolcloooo:  'oooooccloo:    --------------
 .looooc;;:ol  :oc;;:ooooo'    OS: Raspbian GNU/Linux 9.11 (stretch) armv6l
   ;oooooo:      ,ooooooc.     Model: Raspberry Pi Model B Rev 2
     .,:;'.       .;:;'.       Kernel: 4.19.66+
     .... ..'''''. ....        Uptime: 1 day, 14 hours, 56 minutes
   .''.   ..'''''.  ..''.      Packages: 1324
   ..  .....    .....  ..      Shell: bash 4.4.12
  .  .'''''''  .''''''.  .     CPU: ARMv6-compatible rev 7 (v6l) (1) @ 0.7GHz
.'' .''''''''  .'''''''. ''.   Memory: 154MB / 432MB
'''  '''''''    .''''''  '''   
.'    ........... ...    .'.   ████████████████████████
  ....    ''''''''.   .''.     
  '''''.  ''''''''. .'''''
   '''''.  .'''''. .'''''.
    ..''.     .    .''..
          .'''''''
           ......

KLVN on 15 May 2020

@simphide @ebugakov @takerukoushirou Are you using a specific software during / before the crashes or are you using the online platform office.com itself? I have no Windows machines at hand, however, I somewhat doubt it could be related to the operating system of the requestor.

In my case it happened during the night, and regarding office.com it's most likely been a background sync on some device for Office 365. Nothing unusual but the daily routine in software being used or network utilisation over the last days. So far no crash since my report (gdb still being attached).

takerukoushirou on 15 May 2020

Hi,
I'm also having issues with FTL since updating everything to v5.0 a few days ago. When opening Pi-hole it will say "Lost connection to API" across the colored stat boxes. I'll try to collect all necessary debug informations. As of reproducibility I can't say what causes this issue and like the others already said, it happens kinda random and will make all devices in the network unable to connect to the internet (Android: Wifi bars with an exclamation mark).
On top of that my Raspberry Pi 1B (running Pi-hole) is most of the time under heavy load (Load: 1.38 1.04 0.85).

Same problem. This fixed it: in the admin panel, go to settings and click "flush network table" and "flush logs", then "restart dns resolver".

fhriley on 17 May 2020

Same problem. This fixed it: in the admin panel, go to settings and click "flush network table" and "flush logs", then "restart dns resolver".

Thanks, I'll give it a try. Currently my Pi-hole is most of the time disabled, because the heavy load on the Pi slowed down everything in our network until nothing worked anymore. Pretty annoying, but I'm still running gdb in the background, in case I get the FTL error.

KLVN on 17 May 2020

@KLVN Where does the heavy load come from? Typically, Pi-hole has been seen to work perfectly fine for some hundred clients even on a RPi device.

DL6ER on 17 May 2020

@DL6ER I have no clue. When Pi-hole is disabled my load average is about 0.64 0.78 0.69 but if enabled then it will be pretty fast above 1.0. To be fair, the Pi 1B is pretty old and I'm also running InfluxDB, Grafana and some Python Scripts in the background, but everything worked fine until last week when I've updated Pi-hole. For 2 years now I never had this problem before and I only have about 10 clients in the network.

KLVN on 17 May 2020

Maybe try disabling deep CNAME inspection. That should result in better performance

simphide on 18 May 2020

happened again to me:

May 19 15:36:12 dnsmasq[11633]: query[A] 30ec01c14838bb08b5f7b710bbf8b6b2.fp.measure.office.com from 2001:xxx:xxx:0:45bf:a943:9b72:1ec7
May 19 15:36:12 dnsmasq[11633]: forwarded 30ec01c14838bb08b5f7b710bbf8b6b2.fp.measure.office.com to 1.1.1.1
May 19 15:36:12 dnsmasq[11633]: reply 30ec01c14838bb08b5f7b710bbf8b6b2.fp.measure.office.com is
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.37.98
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.77.242
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.68.178
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.64.2
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.64.50
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.66.210
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.147.114
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.64.18
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.68.2
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.146.2
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.89.50
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.146.194
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.144.98
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.74.178
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.146.98
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.75.82
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.147.98
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.146.178
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.64.34
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.101.144.2
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.92.178
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.89.18
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.100.52.2
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.89.34
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.89.2
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.83.178
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 40.100.52.18
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.89.66
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.82.210
May 19 15:36:12 dnsmasq[11633]: reply hnd-efz.office.com is 52.98.79.162

May 19 15:36:12 dnsmasq[11634]: query[AAAA] 30ec01c14838bb08b5f7b710bbf8b6b2.fp.measure.office.com from 2001:xxx:xxx:0:45bf:a943:9b72:1ec7
May 19 15:36:12 dnsmasq[11634]: forwarded 30ec01c14838bb08b5f7b710bbf8b6b2.fp.measure.office.com to 1.1.1.1
May 19 15:36:12 dnsmasq[11634]: reply 30ec01c14838bb08b5f7b710bbf8b6b2.fp.measure.office.com is
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:100e::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1001::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1003::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1006::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:404::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1803::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:404:15::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:403:a::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1016::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:404:a::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1072::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1000::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:404:14::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1004::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:403::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:10b6::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1801::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:402::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1802::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1005::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:10d2::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1092::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1804::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:10ae::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:402:1::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1120::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:101e::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1032::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1002::2
May 19 15:36:12 dnsmasq[11634]: reply hnd-efz.office.com is 2603:1046:c09:1800::2

May 19 15:36:12 dnsmasq[11635]: query[AAAA] hnd-efz.office.com from 2001:xxx:xxx:0:45bf:a943:9b72:1ec7
May 19 15:36:12 dnsmasq[11635]: forwarded hnd-efz.office.com to 1.1.1.1
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1804::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:10ae::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:402:1::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1120::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:101e::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1032::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1002::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1005::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:10d2::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1001::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1003::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1006::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:404::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1803::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:404:15::2
May 19 15:36:12 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:403:a::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1016::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:404:a::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1072::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1000::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:404:14::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1004::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:403::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:10b6::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1801::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:402::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1802::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1800::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:1092::2
May 19 15:36:13 dnsmasq[11635]: reply hnd-efz.office.com is 2603:1046:c09:100e::2

[2020-05-19 15:36:13.960 8638] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-19 15:36:13.960 8638] ----------------------------> FTL crashed! <----------------------------
[2020-05-19 15:36:13.960 8638] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-19 15:36:13.960 8638] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-19 15:36:13.960 8638] and include in your report already the following details:
[2020-05-19 15:36:13.960 8638] FTL has been running for 449492 seconds
[2020-05-19 15:36:13.960 8638] FTL branch: master
[2020-05-19 15:36:13.960 8638] FTL version: v5.0
[2020-05-19 15:36:13.960 8638] FTL commit: 3d7c095
[2020-05-19 15:36:13.960 8638] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-19 15:36:13.960 8638] FTL user: started as pihole, ended as pihole
[2020-05-19 15:36:13.960 8638] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-05-19 15:36:13.960 8638] Received signal: Segmentation fault
[2020-05-19 15:36:13.960 8638] at address: (nil)
[2020-05-19 15:36:13.960 8638] with code: Unknown (128)
[2020-05-19 15:36:13.960 8638] Backtrace:
[2020-05-19 15:36:13.960 8638] ------ Listing content of directory /dev/shm ------
[2020-05-19 15:36:13.960 8638] File Mode User:Group Filesize Filename
[2020-05-19 15:36:13.961 8638] rwxrwxrwx root:root 260 .
[2020-05-19 15:36:13.961 8638] rwxr-xr-x root:root 3K ..
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 655K FTL-dns-cache
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 12K FTL-overTime
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 9M FTL-queries
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 4K FTL-upstreams
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 123K FTL-clients
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 295K FTL-domains
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 315K FTL-strings
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 12 FTL-settings
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 124 FTL-counters
[2020-05-19 15:36:13.961 8638] rw------- pihole:pihole 48 FTL-lock
[2020-05-19 15:36:13.961 8638] ---------------------------------------------------
[2020-05-19 15:36:13.961 8638] Thank you for helping us to improve our FTL engine!
[2020-05-19 15:36:13.961 8638] FTL terminated!

ebugakov on 19 May 2020

@ebugakov Could you attach the debugger? See https://github.com/pi-hole/FTL/issues/757#issuecomment-627557441

Your report again shows iface_check, I have seen this several times now but I'm still waiting for debugging output from someone so I can start looking what is going wrong for you.

DL6ER on 19 May 2020

Just to be sure: Am I debugging correctly, if the console looks like

[New Thread 0xb35ff470 (LWP 16595)]
[Thread 0xb35ff470 (LWP 16595) exited]
[New Thread 0xb35ff470 (LWP 16596)]
[Thread 0xb35ff470 (LWP 16596) exited]
[New Thread 0xb35ff470 (LWP 16597)]
[Thread 0xb35ff470 (LWP 16597) exited]
[New Thread 0xb35ff470 (LWP 16615)]
[Thread 0xb35ff470 (LWP 16615) exited]
[New Thread 0xb35ff470 (LWP 16616)]
[Thread 0xb35ff470 (LWP 16616) exited]
[New Thread 0xb35ff470 (LWP 16617)]
[Thread 0xb35ff470 (LWP 16617) exited]
[New Thread 0xb35ff470 (LWP 16618)]
[Thread 0xb35ff470 (LWP 16618) exited]
[New Thread 0xb35ff470 (LWP 16619)]
[Thread 0xb35ff470 (LWP 16619) exited]

after typing continue?

KLVN on 19 May 2020

@KLVN yes, it'll go away even you close the web interface, however, it also doesn't hurt

DL6ER on 19 May 2020

Thanks, just wanted to verify that it works correctly and that the debug output will be helpful.

KLVN on 19 May 2020

For those who cannot attach the debugger (which would still give more information about the crash source): Please run

pihole checkout ftl new/addr2line_master

This should (hopefully) generate logs with extended information for us to check.

DL6ER on 20 May 2020

Any update for me?

DL6ER on 27 May 2020

Sadly not. It crashed two more times in the last week, but only if gdb wasn't running. Had to restart the Pi, forgot to enable gdb and after a few hours it crashed again. Now, with gdb enabled, it's been running for 4 days without any crash.

KLVN on 27 May 2020

Same here, since I attached gdb about 2 weeks ago it's running fine.

takerukoushirou on 27 May 2020

since gdb was activated, no crashes happened (un)fortunately :) keeping eye on it.

ebugakov on 27 May 2020

Even when I have never seen a debugger influencing a crash, this seems to get more and more likely in this case. Even when I still cannot really believe this, we can still get down to the issue, it will just take more time as I will have to prepare sequential patches, each time giving more information until we have what we need.

So, as a first step, I looked again closely at the already available FTL crash reports and was able to compute the backtrace, even when this was a (slightly lengthy) manual task. I will note down the essential results in case you are interested, if not, skip straight to the bottom.

The result is

$ addr2line 0x41352 0x50025 0x6694b 0x687bc -e pihole-FTL-linux-x86_64 
/root/project/src/dnsmasq/auth.c:75
/root/project/src/dnsmasq/forward.c:1559
/root/project/src/dnsmasq/dnsmasq.c:1786
/root/project/src/dnsmasq/dnsmasq.c:1206

The SEGFAULT happens at

https://github.com/pi-hole/FTL/blob/3d7c095cb10a1b692c80dd28f6051dea91d45baa/src/dnsmasq/auth.c#L75

which means that either zone or zone->domain are NULL.

The full path to this function is (from top to bottom)
https://github.com/pi-hole/FTL/blob/3d7c095cb10a1b692c80dd28f6051dea91d45baa/src/dnsmasq/dnsmasq.c#L1206
https://github.com/pi-hole/FTL/blob/3d7c095cb10a1b692c80dd28f6051dea91d45baa/src/dnsmasq/dnsmasq.c#L1786
https://github.com/pi-hole/FTL/blob/3d7c095cb10a1b692c80dd28f6051dea91d45baa/src/dnsmasq/forward.c#L1559

This brings us closer to the source of the issue as I failed to reproduce this due to the lack of having configured authoritative zones.

@KLVN @takerukoushirou @ebugakov @simphide
I'd like to ask for your auth configuration so I can try to improve on my reproduction attempts. What are the contents of your configs in /etc/dnsmasq.conf and /etc/dnsmasq.d/* ?

As a second attempt, please run

pihole checkout ftl bughaunting_1

and run pihole-FTL without attaching the debugger. Immediately before the next crash there should be log lines with DEBUG: in your /etc/pihole/pihole-FTL.log. They should be helpful, however, I hope to be able to reproduce locally when I have seen your config files.

DL6ER on 27 May 2020

Thank you for your investigations!

/etc/dnsmasq.conf:
conf-dir=/etc/dnsmasq.d

/etc/dnsmasq.d/01-pihole.conf:

# Pi-hole: A black hole for Internet advertisements
# (c) 2017 Pi-hole, LLC (https://pi-hole.net)
# Network-wide ad blocking via your own hardware.
#
# Dnsmasq config for Pi-hole's FTLDNS
#
# This file is copyright under the latest version of the EUPL.
# Please see LICENSE file for your rights under this license.

###############################################################################
#      FILE AUTOMATICALLY POPULATED BY PI-HOLE INSTALL/UPDATE PROCEDURE.      #
# ANY CHANGES MADE TO THIS FILE AFTER INSTALL WILL BE LOST ON THE NEXT UPDATE #
#                                                                             #
#        IF YOU WISH TO CHANGE THE UPSTREAM SERVERS, CHANGE THEM IN:          #
#                      /etc/pihole/setupVars.conf                             #
#                                                                             #
#        ANY OTHER CHANGES SHOULD BE MADE IN A SEPARATE CONFIG FILE           #
#                    WITHIN /etc/dnsmasq.d/yourname.conf                      #
###############################################################################

addn-hosts=/etc/pihole/local.list
addn-hosts=/etc/pihole/custom.list


localise-queries


no-resolv



cache-size=10000

log-queries
log-facility=/var/log/pihole.log

local-ttl=2

log-async
server=8.8.8.8
server=1.1.1.1
domain-needed
bogus-priv
interface=wlan0
server=/fritz.box/192.168.178.1
server=/178.168.192.in-addr.arpa/192.168.178.1
server=/use-application-dns.net/

BTW: I am not using the debugger since I'm running pi-hole on a Pi-Zero and I am scared that pi-hole will lag.

edit: and had no crashes since then!

simphide on 27 May 2020

@simphide The debugger should have no influence on the performance as it only comes into play when a crash happens

DL6ER on 27 May 2020

👍1

/etc/dnsmasq.conf:
conf-dir=/etc/dnsmasq.d

/etc/dnsmasq.d/01-pihole.conf:

# Pi-hole: A black hole for Internet advertisements
# (c) 2017 Pi-hole, LLC (https://pi-hole.net)
# Network-wide ad blocking via your own hardware.
#
# Dnsmasq config for Pi-hole's FTLDNS
#
# This file is copyright under the latest version of the EUPL.
# Please see LICENSE file for your rights under this license.

###############################################################################
#      FILE AUTOMATICALLY POPULATED BY PI-HOLE INSTALL/UPDATE PROCEDURE.      #
# ANY CHANGES MADE TO THIS FILE AFTER INSTALL WILL BE LOST ON THE NEXT UPDATE #
#                                                                             #
#        IF YOU WISH TO CHANGE THE UPSTREAM SERVERS, CHANGE THEM IN:          #
#                      /etc/pihole/setupVars.conf                             #
#                                                                             #
#        ANY OTHER CHANGES SHOULD BE MADE IN A SEPARATE CONFIG FILE           #
#                    WITHIN /etc/dnsmasq.d/yourname.conf                      #
###############################################################################

addn-hosts=/etc/pihole/local.list
addn-hosts=/etc/pihole/custom.list


localise-queries


no-resolv



cache-size=10000

log-queries
log-facility=/var/log/pihole.log

local-ttl=2

log-async
server=1.1.1.1
server=1.0.0.1
server=2606:4700:4700::1111
server=2606:4700:4700::1001
interface=eth0
server=/fritz.box/192.168.178.1
server=/178.168.192.in-addr.arpa/192.168.178.1
server=/use-application-dns.net/

@simphide Interesting, we are both using a Fritz!Box (router) :D

Maybe off-topic/related: @simphide Have you had some weird issues with the Wifi of the Fritz!Box lately? A few times a day the Wifi utilization was extremely high but there was no traffic in the network. Every wireless device wasn't able to connect to the internet, wired/LAN devices worked just fine. I just noticed that since running Pi-hole with gdb attached (=> no FTL crashes) this never happened again. But probably it's just a coincidence...

KLVN on 27 May 2020

No, I have no problems...

simphide on 27 May 2020

Hi, I also have this issue, cross-posting from https://github.com/pi-hole/FTL/issues/805#issuecomment-640168852
Just wanted to add here that I also use Fritzbox (7490) with latest firmware, DHCP disabled on Fritzbox, so that Pihole also manages this

MichaelVoelkel on 7 Jun 2020

Thanks @MichaelVoelkel checking your bug report, you see a crash in iface_check indeed making #805 the correct place for your bug report. I will put put a comment for you there.

DL6ER on 7 Jun 2020

To all: Even when the debugger should not influence anything, it may still either do this or the bugs immediately disappeared as soon as you attached the debugger (out of 17 bug reports, I have seen only 2 coming back with a crash report from within the debugger, silence otherwise). I guess we all agree that the magic "fixed itself" is unlikely, so I went ahead and implemented more debugging output in a spacial variant of FTL.

Please run

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging

and then continue running Pi-hole as if nothing would have happened (don't attach the debugger as we want to reproduce the crash!). Even when this branch does not fix anything (only logging output is added), it should give us more details in the log file just before a crash happens.

The DEBUG: lines will be interesting, also those during startup like

[2020-06-08 09:16:02.153 8515] DEBUG option::interface arg 1 = 0x55555594970a 'enp2s0'
[2020-06-08 09:16:02.153 8515] DEBUG option::interface comma = (nil) '(null)'       
[2020-06-08 09:16:02.153 8515] DEBUG option::interface arg 2 = 0x55555594970a 'enp2s0'
[2020-06-08 09:16:02.153 8515] DEBUG option::interface new->name = 0x555555960c40 'enp2s0'
[2020-06-08 09:16:02.154 8515] DEBUG iface_allowed: daemon->if_names->name = 0x55555594f030 'lo'

Thanks for your continues support in trying to resolve this. We're pretty advanced with our Pi-hole v5.1 preparations and it would be absolutely awesome if we could get the fix for this still in there!

DL6ER on 8 Jun 2020

👍1

Regarding the running GDB session, I also set disable-randomization off a while ago to limit side effects, didn't make a difference though. Will switch to the fix/master-in_zone-and-iface_check_debugging branch but might have to do that outside of business hours.

My dnsmasq configuration:

# preamble omitted

addn-hosts=/etc/pihole/local.list
addn-hosts=/etc/pihole/custom.list

localise-queries

no-resolv

cache-size=10000

log-queries
log-facility=/var/log/pihole.log

local-ttl=2

log-async
server=10.200.0.1#53
server=fd00::f2b0:14ff:fe28:a335#53
interface=eth0
server=/fritz.box/10.200.0.1
server=/0.200.10.in-addr.arpa/10.200.0.1
server=/use-application-dns.net/

The Fritz!Box (model 5490) acts as the primary DHCP and publishes PiHole as the DNS resolver to clients; PiHole itself forwards requests to the Fritz!Box to also resolve local device names configured in the Fritz!Box.

takerukoushirou on 8 Jun 2020

I also installed the fix version but without crash so far (this bug hides well).

MichaelVoelkel on 9 Jun 2020

Looks like I am running into the same:

2020-06-17 17:25:27.292 362158] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-17 17:25:27.292 362158] ---------------------------->  FTL crashed!  <----------------------------
[2020-06-17 17:25:27.292 362158] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-17 17:25:27.292 362158] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-17 17:25:27.292 362158] and include in your report already the following details:
[2020-06-17 17:25:27.292 362158] FTL has been running for 24523 seconds
[2020-06-17 17:25:27.292 362158] FTL branch: master
[2020-06-17 17:25:27.292 362158] FTL version: v5.0
[2020-06-17 17:25:27.292 362158] FTL commit: 3d7c095
[2020-06-17 17:25:27.292 362158] FTL date: 2020-05-10 18:58:38 +0100
[2020-06-17 17:25:27.292 362158] FTL user: started as pihole, ended as pihole
[2020-06-17 17:25:27.292 362158] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-06-17 17:25:27.292 362158] Received signal: Segmentation fault
[2020-06-17 17:25:27.292 362158]      at address: (nil)
[2020-06-17 17:25:27.292 362158]      with code: Unknown (128)
[2020-06-17 17:25:27.292 362158] Backtrace:
[2020-06-17 17:25:27.292 362158] B[0000]: 0x55ae053536f9, /usr/bin/pihole-FTL(+0x316f9) [0x55ae053536f9]
[2020-06-17 17:25:27.292 362158] B[0001]: 0x7f83af8cb3c0, /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f83af8cb3c0]
[2020-06-17 17:25:27.292 362158] B[0002]: 0x55ae05363352, /usr/bin/pihole-FTL(in_zone+0x22) [0x55ae05363352]
[2020-06-17 17:25:27.292 362158] B[0003]: 0x55ae05372025, /usr/bin/pihole-FTL(receive_query+0x8e5) [0x55ae05372025]
[2020-06-17 17:25:27.292 362158] B[0004]: 0x55ae0538894b, /usr/bin/pihole-FTL(+0x6694b) [0x55ae0538894b]
[2020-06-17 17:25:27.292 362158] B[0005]: 0x55ae0538a7bc, /usr/bin/pihole-FTL(main_dnsmasq+0x129c) [0x55ae0538a7bc]
[2020-06-17 17:25:27.292 362158] B[0006]: 0x55ae053468ac, /usr/bin/pihole-FTL(main+0xdc) [0x55ae053468ac]
[2020-06-17 17:25:27.292 362158] B[0007]: 0x7f83af6eb0b3, /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f83af6eb0b3]
[2020-06-17 17:25:27.292 362158] B[0008]: 0x55ae05346a1a, /usr/bin/pihole-FTL(_start+0x2a) [0x55ae05346a1a]
[2020-06-17 17:25:27.292 362158] ------ Listing content of directory /dev/shm ------
[2020-06-17 17:25:27.292 362158] File Mode User:Group  Filesize Filename
[2020-06-17 17:25:27.293 362158] rwxrwxrwx root:root 280 .
[2020-06-17 17:25:27.293 362158] rwxr-xr-x root:root 4K ..
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 90K FTL-dns-cache
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 12K FTL-overTime
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 5M FTL-queries
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 4K FTL-upstreams
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 20K FTL-clients
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 197K FTL-domains
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 225K FTL-strings
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 12 FTL-settings
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 124 FTL-counters
[2020-06-17 17:25:27.293 362158] rw------- pihole:pihole 48 FTL-lock
[2020-06-17 17:25:27.293 362158] rwx------ root:root 80 multipath
[2020-06-17 17:25:27.293 362158] ---------------------------------------------------
[2020-06-17 17:25:27.293 362158] Thank you for helping us to improve our FTL engine!
[2020-06-17 17:25:27.293 362158] FTL terminated!

leonbrag on 18 Jun 2020

Just to I understand, there is no fix for this right?

and fix for https://github.com/pi-hole/pi-hole/issues/3502 does NOT apply to this bug.

Did I get it right?

leonbrag on 18 Jun 2020

@leonbrag Yes and No. We have a potential fix for a rather severe dnsmasq-internal bug discovered by @fhriley

To all: Please run

pihole checkout ftl fix/fhriley-fix_buf_overflow

and see if the crashes are resolved. As always, I very much appreciate testing as it is the only way to be sure we really got it fixed!

DL6ER on 18 Jun 2020

@DL6ER how do I restore pi hole to a release branch after testing? Will

pihole checkout master

do it?

leonbrag on 18 Jun 2020

Yes. You can also run the (quicker)

pihole checkout ftl master

before running

pihole -up

to get to Pi-hole v5.1 once it is released.

DL6ER on 18 Jun 2020

Is there an ETA on 5.1 release?

From: DL6ER notifications@github.com
Sent: Thursday, June 18, 2020 2:15 PM
To: pi-hole/FTL FTL@noreply.github.com
Cc: leonbrag leonbrag@hotmail.com; Mention mention@noreply.github.com
Subject: Re: [pi-hole/FTL] FTL crashed (in_zone+0x22) (#757)

Yes. You can also run the (quicker)

pihole checkout ftl master

before running

pihole -up

to get to Pi-hole v5.1 once it is released.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pi-hole/FTL/issues/757#issuecomment-646309217, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABILQ2JVJYKYZWN3NBZWJZTRXJ7UHANCNFSM4M6XELDA.

leonbrag on 18 Jun 2020

Sometime between now and the death of the universe.

dschaper on 19 Jun 2020

@DL6ER is this fixed in the 5.1 or 5.1.1? I don't see references to this bug in the release note: https://github.com/pi-hole/pi-hole/releases

Thanks

leonbrag on 17 Jul 2020

Yes, it is fixed in 5.1. The fix is in #822 (mentioned above).

DL6ER on 17 Jul 2020

👍1

FTL crashed (in_zone+0x22)

All 45 comments

Related issues