Ftl: [4.1.1] FTL crashed

[tronyx]

In raising this issue, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your issue:

• [X] I have read and understood the contributors guide.
• [X] The issue I am reporting can be replicated
• [X] The issue I am reporting isn't a duplicate

How familiar are you with the codebase?:
1

---

[ISSUE] Expected Behaviour:
FTL to not crash.

[ISSUE] Actual Behaviour:
FTL crashing after ~15 seconds.

[ISSUE] Steps to reproduce:
Start/restart pihole-FTL service or reboot Pi.

Log file output [if available]

[2018-12-21 13:32:00.478] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-12-21 13:32:00.478] ---------------------------->  FTL crashed!  <----------------------------
[2018-12-21 13:32:00.478] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-12-21 13:32:00.478] Please report a bug at https://github.com/pi-hole/FTL/issues
[2018-12-21 13:32:00.478] and include in your report already the following details:

[2018-12-21 13:32:00.478] FTL has been running for 16 seconds
[2018-12-21 13:32:00.478] FTL branch: master
[2018-12-21 13:32:00.478] FTL version: v4.1.1
[2018-12-21 13:32:00.478] FTL commit: f5f331b
[2018-12-21 13:32:00.478] FTL date: 2018-12-21 10:32:10 -0600
[2018-12-21 13:32:00.479] FTL user: pihole
[2018-12-21 13:32:00.479] Received signal: Segmentation fault
[2018-12-21 13:32:00.479]      at address: 0
[2018-12-21 13:32:00.479]      with code: SEGV_MAPERR (Address not mapped to object)
[2018-12-21 13:32:00.479] Backtrace:
[2018-12-21 13:32:00.479] B[0000]: /usr/bin/pihole-FTL(+0x1a09a) [0x45109a]
[2018-12-21 13:32:00.479] B[0001]: /lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer+0) [0x76d606c0]
[2018-12-21 13:32:00.480] Thank you for helping us to improve our FTL engine!
[2018-12-21 13:32:00.480] FTL terminated!

Device specifics

Hardware Type: rPi
OS: Raspbian GNU/Linux 9 (stretch)

Just updated from 4.1 to 4.1.1 and now FTL won't stay running. I've tried repairing with and without sudo to no avail, including replacing the 127.0.0.1 nameserver in the /etc/resolv.conf file.

[KRiusd]

Same here, is there any way to get back to 4.1 without full backup?
If it helps, my debug log: w18b6wvdpa

[mrfrange]

Dito. As a workaround I replaced /usr/bin/pihole-FTL with v4.1 and it's working again.

[AzureMarker]

You can run pihole checkout ftl v4.1 to go back to v4.1.

[tronyx]

You can run pihole checkout ftl v4.1 to go back to v4.1.

Not without changing 127.0.0.1 with your preferred DNS Server address in the /etc/resolv.conf file first.

I can confirm that the above did resolve my issue, or rather put a band-aid on it.

[dschaper]

Are you both running the arm-linux-gnueabihf variant?

[tronyx]

Also, rolling back to 4.1 may have fixed FTL crashing, but none of my internal DNS is resolving.

[MarkRWatts]

Add myself to the list of crashes on 4.1.1 (I've seen 2 since updating earlier this evening).
I am using pihole-FTL-arm-linux-gnueabi (Pi Zero W)

"systemctl restart pihole-FTL" gave me about 20 minutes of working before crashing.

[AzureMarker]

Follow these instructions to help debug the crash: https://docs.pi-hole.net/ftldns/debugging/

[ahettlin]

Wow. I was just literally setting up pihole on my RaspberryPi for the very first time and ran into this issue. Thanks for posting just at the right time. My FTL was crashing due to segmentation faults less than a minute after starting and just after parsing the block lists. I reverted to 4.1 per the suggestion above and I am up and running now.

[jacobsalmela]

Sorry for the trouble. We are looking into the problem.

[rmuehlbauer]

Same here:

[2018-12-21 20:31:05.965] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-12-21 20:31:05.965] ----------------------------> FTL crashed! <----------------------------
[2018-12-21 20:31:05.965] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-12-21 20:31:05.965] Please report a bug at https://github.com/pi-hole/FTL/issues
[2018-12-21 20:31:05.965] and include in your report already the following details:

[2018-12-21 20:31:05.965] FTL has been running for 22 seconds
[2018-12-21 20:31:05.966] FTL branch: master
[2018-12-21 20:31:05.966] FTL version: v4.1.1
[2018-12-21 20:31:05.966] FTL commit: f5f331b
[2018-12-21 20:31:05.966] FTL date: 2018-12-21 10:32:10 -0600
[2018-12-21 20:31:05.966] FTL user: pihole
[2018-12-21 20:31:05.966] Received signal: Segmentation fault
[2018-12-21 20:31:05.966] at address: 0
[2018-12-21 20:31:05.966] with code: SEGV_MAPERR (Address not mapped to object)
[2018-12-21 20:31:05.967] Backtrace:
[2018-12-21 20:31:05.967] Thank you for helping us to improve our FTL engine!
[2018-12-21 20:31:05.967] FTL terminated!

[jvandervorm]

FTL 4.1.1 crashes for me after ~20 seconds on x86_64. Happens every restart. On 4.1 everything was fine. Different backtrace though:

[2018-12-21 20:24:00.181] FTL has been running for 24 seconds
[2018-12-21 20:24:00.181] FTL branch: master
[2018-12-21 20:24:00.181] FTL version: v4.1.1
[2018-12-21 20:24:00.181] FTL commit: f5f331b
[2018-12-21 20:24:00.181] FTL date: 2018-12-21 10:32:10 -0600
[2018-12-21 20:24:00.181] FTL user: pihole
[2018-12-21 20:24:00.181] Received signal: Segmentation fault
[2018-12-21 20:24:00.181] at address: 0
[2018-12-21 20:24:00.181] with code: SEGV_MAPERR (Address not mapped to object)
[2018-12-21 20:24:00.186] Backtrace:
[2018-12-21 20:24:00.187] Thank you for helping us to improve our FTL engine!
[2018-12-21 20:24:00.187] FTL terminated!

[dschaper]

Thanks for the backtrace, it looks like we've identified the situation that causes the segfault and we're working on a fix that should remedy everything.

Thanks everyone for the reports and for your understanding.

[CollinChaffin]

Is this related to the forwarders? Just curious I opened an issue over here with my log but I ask because I notice no matter what it never seems to load more than two forwarders (conditional or not).

Note the (to me) strange way it' counting - two normal forwarders the first labeled (0/0) the 2nd though is (1/4)? Then if I limited down to only one additional conditional (indicated correctly with this line Known forward destinations: 3), the first conditional is for some reason then (2/4)? Then note, my first conditional (the 3rd total) never loads but should be right about the time of the crash which is why I asked above. So firstly, I thought I could have far more than 4 conditionals if needed, so where is this odd counting coming from and second - is it just coincidence that the crash happens before or is it related? I can obviously test further for some answers but since you said you've nailed it down I thought I'd ask.

[2018-12-21 14:37:05.151] New forward server: 8.8.8.8 (0/0)
[2018-12-21 14:37:05.151] Notice: Increasing forwarded struct size from 0 to 4
[2018-12-21 14:37:05.151] Notice: Increasing overTime struct size from 0 to 100
[2018-12-21 14:37:05.151] New forward server: 1.0.0.1 (1/4)
[2018-12-21 14:37:05.151] New forward server: 192.168.1.1 (2/4)
[2018-12-21 14:37:05.151] Imported 541 queries from the long-term database
[2018-12-21 14:37:05.151]  -> Total DNS queries: 541
[2018-12-21 14:37:05.151]  -> Cached DNS queries: 346
[2018-12-21 14:37:05.151]  -> Forwarded DNS queries: 195
[2018-12-21 14:37:05.151]  -> Exactly blocked DNS queries: 0
[2018-12-21 14:37:05.151]  -> Unknown DNS queries: 0
[2018-12-21 14:37:05.151]  -> Unique domains: 61
[2018-12-21 14:37:05.151]  -> Unique clients: 7
[2018-12-21 14:37:05.151]  -> Known forward destinations: 3
[2018-12-21 14:37:05.151] Successfully accessed setupVars.conf
[2018-12-21 14:37:05.153] PID of FTL process: 2843
[2018-12-21 14:37:05.153] Listening on port 4711 for incoming IPv4 telnet connections
[2018-12-21 14:37:05.153] Listening on Unix socket
[2018-12-21 14:37:05.153] INFO: No whitelist file found
[2018-12-21 14:37:05.153] Compiled 0 Regex filters and 0 whitelisted domains in 0.0 msec (0 errors)
[2018-12-21 14:37:05.315] /etc/pihole/gravity.list: parsed 136336 domains (took 161.8 ms)
[2018-12-21 14:38:00.215] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[2018-12-21 14:38:00.215] ---------------------------->  FTL crashed!  <----------------------------
[2018-12-21 14:38:00.215] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

https://github.com/pi-hole/pi-hole/issues/2559

[dschaper]

It actually came down to a string comparison between a string and a (sometimes) null value. The fix was to bypass the section of code that caused a NULL and handle the failure.

[dschaper]

The numbering is zero based and shows (number of entries/memory allocated for entries).

So 8.8.8.8 (0/0) means adding 8.8.8.8 with no entries already existing and no memory allocated.
1.0.0.1 (1/4) means 1 entry already existing (8.8.8.8) and 4 memory sections allocated.

Once all the memory is used, then a new construct is created with more memory available. You can have as many forwarders as you need, we'll just allocate more memory for them to be stored.

[CollinChaffin]

Ahh, makes total sense but I wouldn't have ever guessed it without parsing code so thanks for explaining! And yes read the commit right after it hit and back up and running since. Thanks for quick fix!