FTL crash (main_dnsmasq+0x129c)

[ric-dicle]

In raising this issue, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your issue:

• [X] I have read and understood the contributors guide.
• [X] The issue I am reporting can be replicated
• [] The issue I am reporting isn't a duplicate

How familiar are you with the codebase?:

_{replace this text with a number from 1 to 10, with 1 being not familiar, and 10 being very familiar}_

1

[BUG | ISSUE] Expected Behaviour:

**[BUG | ISSUE] Actual Behaviour: FTL crash PiHole not active

**[BUG | ISSUE] Steps to reproduce: time it happen once per day.

-
-
-
-

Log file output [if available]

[2020-05-17 14:55:11.987 946] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-17 14:55:11.987 946] ----------------------------> FTL crashed! <----------------------------
[2020-05-17 14:55:11.987 946] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-17 14:55:11.987 946] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-17 14:55:11.987 946] and include in your report already the following details:
[2020-05-17 14:55:11.987 946] FTL has been running for 9614 seconds
[2020-05-17 14:55:11.987 946] FTL branch: master
[2020-05-17 14:55:11.987 946] FTL version: v5.0
[2020-05-17 14:55:11.987 946] FTL commit: 3d7c095
[2020-05-17 14:55:11.987 946] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-17 14:55:11.987 946] FTL user: started as pihole, ended as pihole
[2020-05-17 14:55:11.987 946] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-05-17 14:55:11.987 946] Received signal: Segmentation fault
[2020-05-17 14:55:11.987 946] at address: (nil)
[2020-05-17 14:55:11.987 946] with code: Unknown (128)
[2020-05-17 14:55:11.987 946] Backtrace:
[2020-05-17 14:55:11.987 946] ------ Listing content of directory /dev/shm ------
[2020-05-17 14:55:11.987 946] File Mode User:Group Filesize Filename
[2020-05-17 14:55:11.987 946] rwxrwxrwx root:root 260 .
[2020-05-17 14:55:11.987 946] rwxr-xr-x root:root 4K ..
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 25K FTL-dns-cache
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 12K FTL-overTime
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 786K FTL-queries
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 4K FTL-upstreams
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 20K FTL-clients
[2020-05-17 14:55:11.987 946] rw------- pihole:pihole 98K FTL-domains
[2020-05-17 14:55:11.988 946] rw------- pihole:pihole 45K FTL-strings
[2020-05-17 14:55:11.988 946] rw------- pihole:pihole 12 FTL-settings
[2020-05-17 14:55:11.988 946] rw------- pihole:pihole 124 FTL-counters
[2020-05-17 14:55:11.988 946] rw------- pihole:pihole 48 FTL-lock
[2020-05-17 14:55:11.988 946] ---------------------------------------------------
[2020-05-17 14:55:11.988 946] Thank you for helping us to improve our FTL engine!
[2020-05-17 14:55:11.988 946] FTL terminated!

Device specifics

Hardware Type: Behive on FreeNAS-11.3-U2.1
OS: Ubuntu 18.04.4 LTS

_This template was created based on the work of udemy-dl._

[DL6ER]

What are the log lines immediately before the

[2020-05-17 14:55:11.987 946] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-17 14:55:11.987 946] ----------------------------> FTL crashed! <----------------------------
[2020-05-17 14:55:11.987 946] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

?

[ric-dicle]

Here the lines before:

[2020-05-17 12:14:57.160 941] Successfully accessed setupVars.conf
[2020-05-17 12:14:57.163 946] PID of FTL process: 946
[2020-05-17 12:14:57.163 946] Listening on port 4711 for incoming IPv4 telnet connections
[2020-05-17 12:14:57.163 946] Listening on port 4711 for incoming IPv6 telnet connections
[2020-05-17 12:14:57.163 946] Listening on Unix socket
[2020-05-17 12:14:57.165 946] Reloading DNS cache
[2020-05-17 12:14:57.165 946] Blocking status is enabled
[2020-05-17 12:14:57.168 946] INFO: No regex blacklist entries found
[2020-05-17 12:14:57.168 946] INFO: No regex whitelist entries found
[2020-05-17 12:14:57.170 946] Compiled 0 whitelist and 0 blacklist regex filters in 2.7 msec
[2020-05-17 12:14:59.388 946] New upstream server: 1.1.1.1 (8/128)
[2020-05-17 11:25:34.013 946] Resizing "/FTL-dns-cache" from 4096 to 8192
[2020-05-17 11:26:42.149 1400] Note: FTL forked to handle TCP requests
[2020-05-17 11:37:16.114 946] Resizing "/FTL-strings" from 24576 to 28672
[2020-05-17 11:46:42.222 1597] Note: FTL forked to handle TCP requests
[2020-05-17 11:48:23.963 946] Resizing "/FTL-dns-cache" from 8192 to 12288
[2020-05-17 11:54:51.846 946] Resizing "/FTL-strings" from 28672 to 32768
[2020-05-17 11:56:42.168 1686] Note: FTL forked to handle TCP requests
[2020-05-17 12:05:28.432 946] Resizing "/FTL-queries" from 262144 to 524288
[2020-05-17 12:18:09.294 946] Resizing "/FTL-dns-cache" from 12288 to 16384
[2020-05-17 12:20:09.439 946] Resizing "/FTL-strings" from 32768 to 36864
[2020-05-17 13:19:16.827 946] Resizing "/FTL-strings" from 36864 to 40960
[2020-05-17 13:25:55.430 946] Resizing "/FTL-dns-cache" from 16384 to 20480
[2020-05-17 13:32:33.952 946] Resizing "/FTL-queries" from 524288 to 786432
[2020-05-17 13:35:19.503 946] Resizing "/FTL-strings" from 40960 to 45056
[2020-05-17 14:06:10.741 946] New upstream server: 2606:4700:4700::1111 (9/128)
[2020-05-17 14:06:42.322 3223] Note: FTL forked to handle TCP requests
[2020-05-17 14:12:19.099 946] Resizing "/FTL-dns-cache" from 20480 to 24576
[2020-05-17 14:26:42.303 3502] Note: FTL forked to handle TCP requests
[2020-05-17 14:46:42.411 3774] Note: FTL forked to handle TCP requests

[DL6ER]

Okay, so we've seen

[2020-05-17 14:26:42.303 3502] Note: FTL forked to handle TCP requests
[2020-05-17 14:46:42.411 3774] Note: FTL forked to handle TCP requests

also in the other recent crash reports. It seems we're hitting an edge case somewhere. The parallelization within FTL is rather complex (threads and forks existing simultaneously). Interestingly enough, this did never show up during the beta testing period, however, we will eventually come down to it.

Can you follow the instructions here?
https://docs.pi-hole.net/ftldns/debugging/

[ric-dicle]

Just set up the gdb, at the next crash I will post the infos!
thank you

[ric-dicle]

Hello, it crashed this morning, but it was no possible to enter in ssh so I had to reboot it.
Just check the pihole-FTL.log, it starting to log after my reboot.
How I can recover the infos of the crash?
thanks!

[DL6ER]

I don't think it is preserved over a reboot as thegdb output is only in the terminal itself and not written to a log file. However, I did note that you had crashes once per day initially and now no crash for 6 days? There are also no more crash reports from others coming in so it may have been an external influence triggering the crash which got fixed upstream?

By any chance, do you use Cloudflare as upstream DNS? We've seen a misconfiguration in their servers some months ago which was causing issues when DNSSEC was enabled. This issue got fixed, however, there may be something else here which did now disappear because they fixed some other issue on their servers?

So far, we can still only speculate what is going on.

[ric-dicle]

I am using cloudflare, I will restart gdb, and let you know.
Thanks!

[ric-dicle]

Hello here my last FTL log and gbd terminal, hope this helps.

[2020-05-25 00:22:25.468 1184] Resizing "/FTL-dns-cache" from 61440 to 65536
[2020-05-25 02:38:04.910 1184] IPv4 telnet error: Interrupted system call (4)
[2020-05-25 08:26:35.573 1184] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-25 08:26:35.573 1184] ----------------------------> FTL crashed! <----------------------------
[2020-05-25 08:26:35.573 1184] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-25 08:26:35.573 1184] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-25 08:26:35.573 1184] and include in your report already the following details:
[2020-05-25 08:26:35.573 1184] FTL has been running for 80287 seconds
[2020-05-25 08:26:35.573 1184] FTL branch: master
[2020-05-25 08:26:35.573 1184] FTL version: v5.0
[2020-05-25 08:26:35.573 1184] FTL commit: 3d7c095
[2020-05-25 08:26:35.573 1184] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-25 08:26:35.573 1184] FTL user: started as pihole, ended as pihole
[2020-05-25 08:26:35.573 1184] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-05-25 08:26:35.573 1184] Received signal: Segmentation fault
[2020-05-25 08:26:35.573 1184] at address: (nil)
[2020-05-25 08:26:35.573 1184] with code: Unknown (128)
[2020-05-25 08:26:35.587 1184] Backtrace:
[2020-05-25 08:26:35.587 1184] ------ Listing content of directory /dev/shm ------
[2020-05-25 08:26:35.587 1184] File Mode User:Group Filesize Filename
[2020-05-25 08:26:35.587 1184] rwxrwxrwx root:root 260 .
[2020-05-25 08:26:35.587 1184] rwxr-xr-x root:root 4K ..
[2020-05-25 08:26:35.587 1184] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-25 08:26:35.587 1184] rw------- pihole:pihole 66K FTL-dns-cache
[2020-05-25 08:26:35.587 1184] rw------- pihole:pihole 12K FTL-overTime
[2020-05-25 08:26:35.587 1184] rw------- pihole:pihole 4M FTL-queries
[2020-05-25 08:26:35.587 1184] rw------- pihole:pihole 4K FTL-upstreams
[2020-05-25 08:26:35.588 1184] rw------- pihole:pihole 20K FTL-clients
[2020-05-25 08:26:35.588 1184] rw------- pihole:pihole 98K FTL-domains
[2020-05-25 08:26:35.588 1184] rw------- pihole:pihole 111K FTL-strings
[2020-05-25 08:26:35.588 1184] rw------- pihole:pihole 12 FTL-settings
[2020-05-25 08:26:35.588 1184] rw------- pihole:pihole 124 FTL-counters
[2020-05-25 08:26:35.588 1184] rw------- pihole:pihole 48 FTL-lock
[2020-05-25 08:26:35.588 1184] ---------------------------------------------------
[2020-05-25 08:26:35.588 1184] Thank you for helping us to improve our FTL engine!
[2020-05-25 08:26:35.588 1184] FTL terminated!

gdb

[Thread 0x7f7987fff700 (LWP 3963) exited]
[New Thread 0x7f7987fff700 (LWP 3964)]
[Thread 0x7f7987fff700 (LWP 3964) exited]
[New Thread 0x7f7987fff700 (LWP 3965)]
[Thread 0x7f7987fff700 (LWP 3965) exited]
[New Thread 0x7f7987fff700 (LWP 3966)]
[Thread 0x7f7987fff700 (LWP 3966) exited]
[New Thread 0x7f7987fff700 (LWP 3967)]
[Thread 0x7f7987fff700 (LWP 3967) exited]
[New Thread 0x7f7987fff700 (LWP 3968)]
[Thread 0x7f7987fff700 (LWP 3968) exited]
[New Thread 0x7f7987fff700 (LWP 3969)]
[Thread 0x7f7987fff700 (LWP 3969) exited]
[New Thread 0x7f7987fff700 (LWP 3970)]
[Thread 0x7f7987fff700 (LWP 3970) exited]
[New Thread 0x7f7987fff700 (LWP 3971)]
[Thread 0x7f7987fff700 (LWP 3971) exited]
[New Thread 0x7f7987fff700 (LWP 3972)]
[Thread 0x7f7987fff700 (LWP 3972) exited]
[New Thread 0x7f7987fff700 (LWP 3973)]
[Thread 0x7f7987fff700 (LWP 3973) exited]
[New Thread 0x7f7987fff700 (LWP 3974)]
[Thread 0x7f7987fff700 (LWP 3974) exited]
[New Thread 0x7f7987fff700 (LWP 3975)]
[Thread 0x7f7987fff700 (LWP 3975) exited]

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
in_zone (zone=zone@entry=0x6e69730000000000, name=0x562ebf1c2b90 "sin-efz.office.com", cut=cut@entry=0x0) at src/dnsmasq/auth.c:75
75 src/dnsmasq/auth.c: No such file or directory.continue
Continuing.
[New Thread 0x7f7987fff700 (LWP 20875)]
[Thread 0x7f7987fff700 (LWP 20875) exited]
[New Thread 0x7f79877fe700 (LWP 20876)]
[Thread 0x7f79877fe700 (LWP 20876) exited]
[Thread 0x7f798ca73700 (LWP 1190) exited]
[Thread 0x7f798d274700 (LWP 1189) exited]
[Thread 0x7f798da75700 (LWP 1188) exited]
[Thread 0x7f798e276700 (LWP 1187) exited]
[Thread 0x7f798ea77700 (LWP 1186) exited]
[Thread 0x7f798f278700 (LWP 1185) exited]
[Inferior 1 (process 1184) exited with code 01]

Please let me know if you need more infos.

[ric-dicle]

it just re-crashed here the output FTL:
[2020-05-25 10:02:55.677 22020] Resizing "/FTL-dns-cache" from 12288 to 16384
[2020-05-25 10:03:38.079 22020] Resizing "/FTL-strings" from 77824 to 81920
[2020-05-25 10:08:04.568 23183] Note: FTL forked to handle TCP requests
[2020-05-25 10:17:33.121 22020] Resizing "/FTL-dns-cache" from 16384 to 20480
[2020-05-25 10:18:04.607 23355] Note: FTL forked to handle TCP requests
[2020-05-25 10:38:04.764 23561] Note: FTL forked to handle TCP requests
[2020-05-25 10:38:04.824 22020] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-25 10:38:04.824 22020] ----------------------------> FTL crashed! <----------------------------
[2020-05-25 10:38:04.824 22020] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-05-25 10:38:04.824 22020] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-05-25 10:38:04.824 22020] and include in your report already the following details:
[2020-05-25 10:38:04.824 22020] FTL has been running for 6823 seconds
[2020-05-25 10:38:04.824 22020] FTL branch: master
[2020-05-25 10:38:04.824 22020] FTL version: v5.0
[2020-05-25 10:38:04.824 22020] FTL commit: 3d7c095
[2020-05-25 10:38:04.824 22020] FTL date: 2020-05-10 18:58:38 +0100
[2020-05-25 10:38:04.824 22020] FTL user: started as pihole, ended as pihole
[2020-05-25 10:38:04.824 22020] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-05-25 10:38:04.824 22020] Received signal: Segmentation fault
[2020-05-25 10:38:04.824 22020] at address: (nil)
[2020-05-25 10:38:04.824 22020] with code: Unknown (128)
[2020-05-25 10:38:04.824 22020] Backtrace:
[2020-05-25 10:38:04.824 22020] ------ Listing content of directory /dev/shm ------
[2020-05-25 10:38:04.824 22020] File Mode User:Group Filesize Filename
[2020-05-25 10:38:04.824 22020] rwxrwxrwx root:root 260 .
[2020-05-25 10:38:04.825 22020] rwxr-xr-x root:root 4K ..
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 20K FTL-dns-cache
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 12K FTL-overTime
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 4M FTL-queries
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 4K FTL-upstreams
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 20K FTL-clients
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 98K FTL-domains
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 82K FTL-strings
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 12 FTL-settings
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 124 FTL-counters
[2020-05-25 10:38:04.825 22020] rw------- pihole:pihole 48 FTL-lock
[2020-05-25 10:38:04.825 22020] ---------------------------------------------------
[2020-05-25 10:38:04.825 22020] Thank you for helping us to improve our FTL engine!
[2020-05-25 10:38:04.825 22020] FTL terminated!

and gdb

[New Thread 0x7f7987fff700 (LWP 3959)]
[Thread 0x7f7987fff700 (LWP 3959) exited]
[New Thread 0x7f7987fff700 (LWP 3960)]
[Thread 0x7f7987fff700 (LWP 3960) exited]
[New Thread 0x7f7987fff700 (LWP 3961)]
[Thread 0x7f7987fff700 (LWP 3961) exited]
[New Thread 0x7f7987fff700 (LWP 3962)]
[Thread 0x7f7987fff700 (LWP 3962) exited]
[New Thread 0x7f7987fff700 (LWP 3963)]
[Thread 0x7f7987fff700 (LWP 3963) exited]
[New Thread 0x7f7987fff700 (LWP 3964)]
[Thread 0x7f7987fff700 (LWP 3964) exited]
[New Thread 0x7f7987fff700 (LWP 3965)]
[Thread 0x7f7987fff700 (LWP 3965) exited]
[New Thread 0x7f7987fff700 (LWP 3966)]
[Thread 0x7f7987fff700 (LWP 3966) exited]
[New Thread 0x7f7987fff700 (LWP 3967)]
[Thread 0x7f7987fff700 (LWP 3967) exited]
[New Thread 0x7f7987fff700 (LWP 3968)]
[Thread 0x7f7987fff700 (LWP 3968) exited]
[New Thread 0x7f7987fff700 (LWP 3969)]
[Thread 0x7f7987fff700 (LWP 3969) exited]
[New Thread 0x7f7987fff700 (LWP 3970)]
[Thread 0x7f7987fff700 (LWP 3970) exited]
[New Thread 0x7f7987fff700 (LWP 3971)]
[Thread 0x7f7987fff700 (LWP 3971) exited]
[New Thread 0x7f7987fff700 (LWP 3972)]
[Thread 0x7f7987fff700 (LWP 3972) exited]
[New Thread 0x7f7987fff700 (LWP 3973)]
[Thread 0x7f7987fff700 (LWP 3973) exited]
[New Thread 0x7f7987fff700 (LWP 3974)]
[Thread 0x7f7987fff700 (LWP 3974) exited]
[New Thread 0x7f7987fff700 (LWP 3975)]
[Thread 0x7f7987fff700 (LWP 3975) exited]

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
in_zone (zone=zone@entry=0x6e69730000000000, name=0x562ebf1c2b90 "sin-efz.office.com", cut=cut@entry=0x0) at src/dnsmasq/auth.c:75
75 src/dnsmasq/auth.c: No such file or directory.
[New Thread 0x7f7987fff700 (LWP 20875)]
[Thread 0x7f7987fff700 (LWP 20875) exited]
[New Thread 0x7f79877fe700 (LWP 20876)]
[Thread 0x7f79877fe700 (LWP 20876) exited]
[Thread 0x7f798ca73700 (LWP 1190) exited]
[Thread 0x7f798d274700 (LWP 1189) exited]
[Thread 0x7f798da75700 (LWP 1188) exited]
[Thread 0x7f798e276700 (LWP 1187) exited]
[Thread 0x7f798ea77700 (LWP 1186) exited]
[Thread 0x7f798f278700 (LWP 1185) exited]
[Inferior 1 (process 1184) exited with code 01]

[DL6ER]

Thread 1 "pihole-FTL" received signal SIGSEGV, Segmentation fault.
in_zone (zone=zone@entry=0x6e69730000000000, name=0x562ebf1c2b90 "sin-efz.office.com", cut=cut@entry=0x0) at src/dnsmasq/auth.c:75

Thanks, that's already very helpful. The error happens when dnsmasq tries to access zone->domain which seems to be NULL:

[2020-05-25 10:38:04.824 22020] Received signal: Segmentation fault
[2020-05-25 10:38:04.824 22020] at address: (nil)

Can you, next time it crashes, type and copy the result of the following debugger commands?

where
p zone
p *zone
p name
p cut
p *cut

This will hopefully shine some more light on this. So far, I'm not yet able to trace the entire route. in_zone() is called from 14 places in the code so we first have to isolate the exact path.

[DL6ER]

(main_dnsmasq+0x129c)

Just for the record: This is the same bug also reported as #783 and #770.

[ric-dicle]

Hello, here the check you asked:

Thread 1 "pihole-FTL" received signal SIGHUP, Hangup.
0x00007f0ad5c9dbf9 in __GI___poll (fds=0x55a3bcc9e150, nfds=18, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29      in ../sysdeps/unix/sysv/linux/poll.c


(gdb) where
#0  0x00007f0ad5c9dbf9 in __GI___poll (fds=0x55a3bcc9e150, nfds=18, timeout=timeout@entry=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x000055a3bb03042a in poll (__timeout=-1, __nfds=<optimized out>, __fds=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  do_poll (timeout=timeout@entry=-1) at src/dnsmasq/poll.c:78
#3  0x000055a3bb05072b in main_dnsmasq (argc=<optimized out>, argv=<optimized out>) at src/dnsmasq/dnsmasq.c:1125
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) p zone
No symbol "zone" in current context.
(gdb) p *zone
No symbol "zone" in current context.
(gdb) p name
Cannot access memory at address 0x7f0ad5f792a0
(gdb) p cut
No symbol "cut" in current context.
(gdb) p *cut
No symbol "cut" in current context.
(gdb) 


Moreover I tried to restartdns but it did not work:
/var/log$ pihole restartdns
  [✗] Job for pihole-FTL.service failed because the control process exited with error code.
See "systemctl status pihole-FTL.service" and "journalctl -xe" for details.

then I checked as requested both  "systemctl status pihole-FTL.service" and "journalctl -xe":


/var/log$ systemctl status pihole-FTL.service
● pihole-FTL.service - LSB: pihole-FTL daemon
   Loaded: loaded (/etc/init.d/pihole-FTL; generated)
   Active: failed (Result: exit-code) since Sun 2020-05-31 08:46:17 UTC; 26s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 21572 ExecStop=/etc/init.d/pihole-FTL stop (code=exited, status=1/FAILURE)
  Process: 22009 ExecStart=/etc/init.d/pihole-FTL start (code=exited, status=1/FAILURE)

May 31 08:46:12 pihole systemd[1]: Starting LSB: pihole-FTL daemon...
May 31 08:46:17 pihole pihole-FTL[22009]: .....
May 31 08:46:17 pihole pihole-FTL[22009]: Not stopped; may still be shutting down or shutdown may have failed, killing now
May 31 08:46:17 pihole systemd[1]: pihole-FTL.service: Control process exited, code=exited status=1
May 31 08:46:17 pihole systemd[1]: pihole-FTL.service: Failed with result 'exit-code'.
May 31 08:46:17 pihole systemd[1]: Failed to start LSB: pihole-FTL daemon.

and "journalctl -xe":

May 31 08:39:31 pihole sudo[21784]: www-data : TTY=unknown ; PWD=/var/www/html/admin ; USER=root ; COMMAND=/usr/local/bin/pihole status web
May 31 08:39:31 pihole sudo[21784]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 31 08:39:31 pihole sudo[21784]: pam_unix(sudo:session): session closed for user root
May 31 08:39:32 pihole sudo[21792]: www-data : TTY=unknown ; PWD=/var/www/html/admin ; USER=root ; COMMAND=/usr/local/bin/pihole status web
May 31 08:39:32 pihole sudo[21792]: pam_unix(sudo:session): session opened for user root by (uid=0)
May 31 08:39:32 pihole sudo[21792]: pam_unix(sudo:session): session closed for user root
May 31 08:40:01 pihole CRON[21795]: pam_unix(cron:session): session opened for user root by (uid=0)
May 31 08:40:01 pihole CRON[21796]: (root) CMD (   PATH="$PATH:/usr/local/bin/" pihole updatechecker local)
May 31 08:40:01 pihole CRON[21795]: pam_unix(cron:session): session closed for user root
May 31 08:40:28 pihole sshd[21414]: pam_unix(sshd:session): session closed for user tom
May 31 08:40:28 pihole systemd-logind[802]: Removed session 1307.
-- Subject: Session 1307 has been terminated
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- Documentation: https://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A session with the ID 1307 has been terminated.
May 31 08:42:25 pihole sshd[21827]: Accepted password for tom from 192.168.1.87 port 53316 ssh2
May 31 08:42:25 pihole sshd[21827]: pam_unix(sshd:session): session opened for user tom by (uid=0)
May 31 08:42:25 pihole systemd[1]: Started Session 1310 of user tom.
-- Subject: Unit session-1310.scope has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit session-1310.scope has finished starting up.
-- 
-- The start-up result is RESULT.
May 31 08:42:25 pihole systemd-logind[802]: New session 1310 of user tom.
-- Subject: A new session 1310 has been created for user tom
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- Documentation: https://www.freedesktop.org/wiki/Software/systemd/multiseat
-- 
-- A new session with the ID 1310 has been created for the user tom.
-- 
-- The leading process of the session is 21827.
May 31 08:42:42 pihole sudo[21928]:      tom : TTY=pts/0 ; PWD=/home/tom ; USER=root ; COMMAND=/bin/bash /usr/local/bin/pihole status
May 31 08:42:42 pihole sudo[21928]: pam_unix(sudo:session): session opened for user root by tom(uid=0)
May 31 08:42:42 pihole sudo[21928]: pam_unix(sudo:session): session closed for user root
May 31 08:45:48 pihole sudo[21970]:      tom : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash /usr/local/bin/pihole -restart
May 31 08:45:48 pihole sudo[21970]: pam_unix(sudo:session): session opened for user root by tom(uid=0)
May 31 08:45:48 pihole sudo[21970]: pam_unix(sudo:session): session closed for user root
May 31 08:45:58 pihole sudo[21977]:      tom : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash /usr/local/bin/pihole restardnst
May 31 08:45:58 pihole sudo[21977]: pam_unix(sudo:session): session opened for user root by tom(uid=0)
May 31 08:45:58 pihole sudo[21977]: pam_unix(sudo:session): session closed for user root
May 31 08:46:04 pihole sudo[21989]:      tom : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash /usr/local/bin/pihole restardns
May 31 08:46:04 pihole sudo[21989]: pam_unix(sudo:session): session opened for user root by tom(uid=0)
May 31 08:46:04 pihole sudo[21989]: pam_unix(sudo:session): session closed for user root
May 31 08:46:12 pihole sudo[21996]:      tom : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash /usr/local/bin/pihole restartdns
May 31 08:46:12 pihole sudo[21996]: pam_unix(sudo:session): session opened for user root by tom(uid=0)
May 31 08:46:12 pihole systemd[1]: Starting LSB: pihole-FTL daemon...
-- Subject: Unit pihole-FTL.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit pihole-FTL.service has begun starting up.
May 31 08:46:17 pihole pihole-FTL[22009]: .....
May 31 08:46:17 pihole pihole-FTL[22009]: Not stopped; may still be shutting down or shutdown may have failed, killing now
May 31 08:46:17 pihole systemd[1]: pihole-FTL.service: Control process exited, code=exited status=1
May 31 08:46:17 pihole sudo[21996]: pam_unix(sudo:session): session closed for user root
May 31 08:46:17 pihole systemd[1]: pihole-FTL.service: Failed with result 'exit-code'.
May 31 08:46:17 pihole systemd[1]: Failed to start LSB: pihole-FTL daemon.
-- Subject: Unit pihole-FTL.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit pihole-FTL.service has failed.
-- 
-- The result is RESULT.

I had to reboot the system.

[actng]

came here from #801

pihole crashed third day in a row... although this time it happened at 3pm ET while the previous two days it was 6am ET.

here is the entire pihole-FTL.log from just now

pihole@pihole:/var/log$ cat pihole-FTL.log | more
[2020-06-01 00:09:08.623 203877] Note: FTL forked to handle TCP requests
[2020-06-01 00:52:02.174 207911] Note: FTL forked to handle TCP requests
[2020-06-01 00:52:44.259 207972] Note: FTL forked to handle TCP requests
[2020-06-01 02:11:04.360 136387] Resizing "/FTL-dns-cache" from 45056 to 49152
[2020-06-01 02:30:43.020 136387] Resizing "/FTL-strings" from 73728 to 77824
[2020-06-01 03:29:39.553 222925] Note: FTL forked to handle TCP requests
[2020-06-01 03:32:41.435 136387] Resizing "/FTL-dns-cache" from 49152 to 53248
[2020-06-01 06:07:12.895 237584] Note: FTL forked to handle TCP requests
[2020-06-01 06:20:03.505 238809] Note: FTL forked to handle TCP requests
[2020-06-01 08:46:34.539 252583] Note: FTL forked to handle TCP requests
[2020-06-01 09:10:27.310 254819] Note: FTL forked to handle TCP requests
[2020-06-01 09:20:28.892 136387] Resizing "/FTL-strings" from 77824 to 81920
[2020-06-01 09:27:37.055 136387] Resizing "/FTL-dns-cache" from 53248 to 57344
[2020-06-01 11:57:37.217 136387] Resizing "/FTL-strings" from 81920 to 86016
[2020-06-01 12:10:42.742 271466] Note: FTL forked to handle TCP requests
[2020-06-01 12:11:15.222 136387] Resizing "/FTL-dns-cache" from 57344 to 61440
[2020-06-01 12:44:31.908 136387] Resizing "/FTL-strings" from 86016 to 90112
[2020-06-01 12:57:49.850 136387] Resizing "/FTL-dns-cache" from 61440 to 65536
[2020-06-01 13:11:02.936 136387] Resizing "/FTL-strings" from 90112 to 94208
[2020-06-01 13:50:17.744 136387] Resizing "/FTL-dns-cache" from 65536 to 69632
[2020-06-01 14:22:20.575 136387] Resizing "/FTL-strings" from 94208 to 98304
[2020-06-01 15:11:07.416 291814] Note: FTL forked to handle TCP requests
[2020-06-01 15:13:36.612 136387] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-01 15:13:36.612 136387] ----------------------------> FTL crashed! <----------------------------
[2020-06-01 15:13:36.612 136387] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-01 15:13:36.612 136387] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-01 15:13:36.612 136387] and include in your report already the following details:
[2020-06-01 15:13:36.612 136387] FTL has been running for 91294 seconds
[2020-06-01 15:13:36.612 136387] FTL branch: master
[2020-06-01 15:13:36.612 136387] FTL version: v5.0
[2020-06-01 15:13:36.612 136387] FTL commit: 3d7c095
[2020-06-01 15:13:36.612 136387] FTL date: 2020-05-10 18:58:38 +0100
[2020-06-01 15:13:36.612 136387] FTL user: started as pihole, ended as pihole
[2020-06-01 15:13:36.612 136387] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-06-01 15:13:36.612 136387] Received signal: Segmentation fault
[2020-06-01 15:13:36.612 136387] at address: (nil)
[2020-06-01 15:13:36.612 136387] with code: Unknown (128)
[2020-06-01 15:13:36.612 136387] Backtrace:
[2020-06-01 15:13:36.613 136387] ------ Listing content of directory /dev/shm ------
[2020-06-01 15:13:36.613 136387] File Mode User:Group Filesize Filename
[2020-06-01 15:13:36.613 136387] rwxrwxrwx root:root 280 .
[2020-06-01 15:13:36.614 136387] rwxr-xr-x root:root 4K ..
[2020-06-01 15:13:36.614 136387] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 70K FTL-dns-cache
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 12K FTL-overTime
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 4M FTL-queries
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 4K FTL-upstreams
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 20K FTL-clients
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 98K FTL-domains
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 98K FTL-strings
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 12 FTL-settings
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 124 FTL-counters
[2020-06-01 15:13:36.615 136387] rw------- pihole:pihole 48 FTL-lock
[2020-06-01 15:13:36.615 136387] rwx------ root:root 80 multipath
[2020-06-01 15:13:36.615 136387] ---------------------------------------------------
[2020-06-01 15:13:36.615 136387] Thank you for helping us to improve our FTL engine!
[2020-06-01 15:13:36.615 136387] FTL terminated!

[DL6ER]

@ric-dicle This is not a crash but was triggered due to list reloading.

Thread 1 "pihole-FTL" received signal SIGHUP, Hangup.

Did you follow step 4 of the documentation? https://docs.pi-hole.net/ftldns/debugging/
It will tell the debugger not to stop at expected interrupt points (such as reloading the lists).

This is also why there was no useful output in the following debugger instructions you typed.

---

To all: We have another report coming in here relating a crash to either DoH or Cloudflared (or the combination of both). Do you use a similar combination?

[actng]

To all: We have another report coming in here relating a crash to either DoH or Cloudflared (or the combination of both). Do you use a similar combination?

i use cloudflare!

[actng]

crashed again this morning
pihole@pihole:/var/log$ cat pihole-FTL.log
[2020-06-04 00:09:55.583 699861] Resizing "/FTL-dns-cache" from 12288 to 16384
[2020-06-04 00:21:58.501 713540] Note: FTL forked to handle TCP requests
[2020-06-04 00:26:13.028 699861] Resizing "/FTL-strings" from 49152 to 53248
[2020-06-04 00:27:19.958 714319] Note: FTL forked to handle TCP requests
[2020-06-04 01:03:00.988 699861] Resizing "/FTL-dns-cache" from 16384 to 20480
[2020-06-04 01:22:38.571 722567] Note: FTL forked to handle TCP requests
[2020-06-04 02:09:00.129 699861] Resizing "/FTL-strings" from 53248 to 57344
[2020-06-04 02:54:51.157 699861] Resizing "/FTL-dns-cache" from 20480 to 24576
[2020-06-04 02:59:36.077 736432] Note: FTL forked to handle TCP requests
[2020-06-04 03:27:47.417 740367] Note: FTL forked to handle TCP requests
[2020-06-04 06:21:43.335 699861] Resizing "/FTL-strings" from 57344 to 61440
[2020-06-04 06:21:43.421 699861] Resizing "/FTL-dns-cache" from 24576 to 28672
[2020-06-04 06:23:54.052 764132] Note: FTL forked to handle TCP requests
[2020-06-04 06:25:33.232 699861] Resizing "/FTL-dns-cache" from 28672 to 32768
[2020-06-04 06:28:46.067 764650] Note: FTL forked to handle TCP requests
[2020-06-04 07:24:27.325 699861] Resizing "/FTL-dns-cache" from 32768 to 36864
[2020-06-04 07:24:34.343 699861] Resizing "/FTL-strings" from 61440 to 65536
[2020-06-04 08:38:18.631 699861] Resizing "/FTL-dns-cache" from 36864 to 40960
[2020-06-04 08:59:22.140 775055] Note: FTL forked to handle TCP requests
[2020-06-04 09:07:12.166 699861] Resizing "/FTL-strings" from 65536 to 69632
[2020-06-04 09:09:15.994 699861] Resizing "/FTL-dns-cache" from 40960 to 45056
[2020-06-04 09:15:57.087 776219] Note: FTL forked to handle TCP requests
[2020-06-04 09:15:57.606 699861] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-04 09:15:57.606 699861] ----------------------------> FTL crashed! <----------------------------
[2020-06-04 09:15:57.606 699861] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2020-06-04 09:15:57.606 699861] Please report a bug at https://github.com/pi-hole/FTL/issues
[2020-06-04 09:15:57.606 699861] and include in your report already the following details:
[2020-06-04 09:15:57.606 699861] FTL has been running for 37604 seconds
[2020-06-04 09:15:57.606 699861] FTL branch: master
[2020-06-04 09:15:57.606 699861] FTL version: v5.0
[2020-06-04 09:15:57.606 699861] FTL commit: 3d7c095
[2020-06-04 09:15:57.606 699861] FTL date: 2020-05-10 18:58:38 +0100
[2020-06-04 09:15:57.606 699861] FTL user: started as pihole, ended as pihole
[2020-06-04 09:15:57.606 699861] Compiled for x86_64 (compiled on CI) using gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
[2020-06-04 09:15:57.606 699861] Received signal: Segmentation fault
[2020-06-04 09:15:57.606 699861] at address: (nil)
[2020-06-04 09:15:57.606 699861] with code: Unknown (128)
[2020-06-04 09:15:57.608 699861] Backtrace:
[2020-06-04 09:15:57.608 699861] ------ Listing content of directory /dev/shm ------
[2020-06-04 09:15:57.609 699861] File Mode User:Group Filesize Filename
[2020-06-04 09:15:57.609 699861] rwxrwxrwx root:root 280 .
[2020-06-04 09:15:57.609 699861] rwxr-xr-x root:root 4K ..
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 4K FTL-per-client-regex
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 45K FTL-dns-cache
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 12K FTL-overTime
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 7M FTL-queries
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 4K FTL-upstreams
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 20K FTL-clients
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 98K FTL-domains
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 70K FTL-strings
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 12 FTL-settings
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 124 FTL-counters
[2020-06-04 09:15:57.609 699861] rw------- pihole:pihole 48 FTL-lock
[2020-06-04 09:15:57.609 699861] rwx------ root:root 80 multipath
[2020-06-04 09:15:57.609 699861] ---------------------------------------------------
[2020-06-04 09:15:57.609 699861] Thank you for helping us to improve our FTL engine!
[2020-06-04 09:15:57.609 699861] FTL terminated!
pihole@pihole:/var/log$

[DL6ER]

@actng Can you attach the debugger per description in the linked docs article? Only with access to the variables (or me being able to reproduce this locally), I can try to find out what is causing this and how to fix it.

[actng]

@actng Can you attach the debugger per description in the linked docs article? Only with access to the variables (or me being able to reproduce this locally), I can try to find out what is causing this and how to fix it.

ok i just installed screen and gdb and ran all those commands
i'm guessing it will be quite a while before it crashes so i just leave it running until then?

[DL6ER]

@actng Sure, the debugger doesn't affect anything otherwise (except a bit more memory consumption). You could even run it forever without noticing any adverse effect.

[pralor-bot]

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/pihole-crashes-daily/33952/5

[DL6ER]

@actng Please check also for the output of the following lines when you see a crash.

where
p dnsmasq_daemon
p *dnsmasq_daemon
p tmp
p *tmp
p name
p *name

[moeskerv]

Comming from https://github.com/pi-hole/FTL/issues/803

• started debug session
• I'm using cloudflare upstream DNS servers

[actng]

ummm... so pihole crashed this time and wouldn't restart without a reboot... and i lost the terminal where screen/gdb was running... and backtrace is not a known command... epic fail... any logs i can upload still ?

[DL6ER]

any logs i can upload still ?

No, unfortunately not :-/

---

To all: Even when the debugger should not influence anything, it may still either do this or the bugs immediately disappeared as soon as you touched the debugger (out of 17 bug reports, I have seen only 2 coming back with a crash report from within the debugger, silence otherwise). I guess we all agree that the magic "fixed itself" is unlikely, so I went ahead and implemented more debugging output in a spacial variant of FTL.

Please run

pihole checkout ftl fix/master-in_zone-and-iface_check_debugging

and then continue running Pi-hole as if nothing would have happened (don't attach the debugger as we want to reproduce the crash!). Even when this branch does not fix anything (only logging output is added), it should give us more details in the log file just before a crash happens.

The DEBUG: lines will be interesting, also those during startup like

[2020-06-08 09:16:02.153 8515] DEBUG option::interface arg 1 = 0x55555594970a 'enp2s0'
[2020-06-08 09:16:02.153 8515] DEBUG option::interface comma = (nil) '(null)'       
[2020-06-08 09:16:02.153 8515] DEBUG option::interface arg 2 = 0x55555594970a 'enp2s0'
[2020-06-08 09:16:02.153 8515] DEBUG option::interface new->name = 0x555555960c40 'enp2s0'
[2020-06-08 09:16:02.154 8515] DEBUG iface_allowed: daemon->if_names->name = 0x55555594f030 'lo'

Thanks for your continues support in trying to resolve this. We're pretty advanced with our Pi-hole v5.1 preparations and it would be absolutely awesome if we could get the fix for this still in there!

[ric-dicle]

Hello here my gdb of today:

[New Thread 0x7f07d3c09700 (LWP 3036)]
[Thread 0x7f07d3c09700 (LWP 3036) exited]
[New Thread 0x7f07d3c09700 (LWP 3037)]
[Thread 0x7f07d3c09700 (LWP 3037) exited]
[New Thread 0x7f07d3c09700 (LWP 3038)]
[Thread 0x7f07d3c09700 (LWP 3038) exited]
[New Thread 0x7f07d3c09700 (LWP 3050)]
[Thread 0x7f07d3c09700 (LWP 3050) exited]

Thread 1 "pihole-FTL" received signal SIGHUP, Hangup.
0x00007f07d7c26bf9 in __GI___poll (fds=0x557835a382c0, nfds=18, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29 in ../sysdeps/unix/sysv/linux/poll.c
(gdb) backtrace

0 0x00007f07d7c26bf9 in __GI___poll (fds=0x557835a382c0, nfds=18, timeout=timeout@entry=-1)

at ../sysdeps/unix/sysv/linux/poll.c:29

1 0x000055783347b42a in poll (__timeout=-1, __nfds=, __fds=)

at /usr/include/x86_64-linux-gnu/bits/poll2.h:46

2 do_poll (timeout=timeout@entry=-1) at src/dnsmasq/poll.c:78

3 0x000055783349b72b in main_dnsmasq (argc=, argv=) at src/dnsmasq/dnsmasq.c:1125

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

I hope this helps.

[DL6ER]

@ric-dicle Sorry, I didn't see your last reply. SIGHUP is expected and signals FTL to reload the DNS cache, e.g., after a pihole -g run. This is perfectly fine. Actually, step 4 of https://docs.pi-hole.net/ftldns/debugging/#prerequirements-only-required-once should have configured gdb to ignore this and other expected signals altogether. Do you still experience issues?

Simon Kelley, the maintainer of dnsmasq is aware of this bug and is about to release dnsmasq v2.82 finally fixing this. However, we already have the fix merged into our development branch so Pi-hole should already be "bugfree".