Pi-hole: Dns resolution fails precisely once a week (sat > sun)

[fsantiago07044]

Versions

• Pi-hole: 5.1.1
• AdminLTE: 5.1
• FTL: 5.1

Platform

• OS and version: raspian 9.13
  
• Platform: Raspberry pi 3 model b+

Expected behavior

For dns resolution to always work consistently.

Actual behavior / bug

Once a week sometime sat night > sun morning, dns resolution stops working. The web ui I don’t believe shows an issue. But restarting the dns resolution service corrects it. It will once again work for another week.

Steps to reproduce

I’m not sure. It just stops working after the week. This started after my most recent pi-hole update which was some time ago.

Debug Token

• URL: https://tricorder.pi-hole.net/krs6vwdbo4

Screenshots

N/a

Additional context

N/a

[yubiuser]

At sunday morning (between 3 and 4 am) pihole runs its weekly update of the gravity database by fetching all your adlists.
Can you reproduce the failure by running pihole -g (triggers manually gravity update)?

[fsantiago07044]

At the moment it seems to be working, albeit a little slower. The on screen log from -g only had one apparent error:

[i] Target: https://hosts-file.net/ad_servers.txt [✗] Status: Connection Refused
[✗] List download failed: no cached list available

[yubiuser]

This list doesn't exist anymore. You can remove it from your adlist.

[fsantiago07044]

Ok it’s gone. Do you believe or know that to have been the cause? Or just something you noticed?

[yubiuser]

I don't think that's the cause, I just noticed. I don't know what caused the problem. There is (at least one) other user experiencing a similar issue.
https://discourse.pi-hole.net/t/dns-resolution-unexpectedly-stops-after-upgrade-to-5-0/34678

[fsantiago07044]

Ok your test definitely caused resolution to stop working again. It wasn’t immediate but soon after it started failing. Another dns service restart fixed it.

[AtjonTV]

Sounds like an issue that the database isnt reloaded correctly :thinking:

Just from reading it seems that the database is not loaded correctly and the resolution list is empty, so no requests are successful, yet with a response ..

When you manually send a DNS request to the DNS (host -t A google.com <pihole-ip>) what is your response? Does it timeout or is it returning an empty entry.

If it is the database then it should give an empty entry. :thinking:

When the issue happens, you should also check if the DNS still has its port (netstat -tulpn | grep LISTEN).

If there is no Port used by the DNS, it might be a DNS Server issue, if it is there and the manual request timed out .. it could then be a bug somewhere else (maby the OS?)

I am running 5.1.1 since Saturday now, so I cannot confirm this issue yet. Though I am running on a CentOS 7 KVM instance.

---

Please note that I do not have knowledge about how Pihole works in the backend, I am just bringing out my thoughs where the issues could be located at ;D

[yubiuser]

Please have a look at /var/log/pihole-FTL.log
Do you see any errors or crashes?
It would be also interesting if there is a long delay between

Reloading DNS cache

And the subsequent entries.

[ElTopo]

I still see this problem after I upgrade to:
Pi-hole version is v5.1.2 (Latest: v5.1.2)
AdminLTE version is v5.1.1 (Latest: v5.1.1)
FTL version is v5.2 (Latest: v5.2)

In fact, manually running this command (from /etc/cron.d/pihole) as root
PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log
causes this problem right away, I have to use
pihole restartdns
to make pihole working again.

Currently I commented out this line from /etc/cron.d/pihole.

[dschaper]

Does /var/log/pihole_updateGravity.log have any information in it?

[ElTopo]

Here's my /var/log/pihole_updateGravity.log:

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [i] Received 58343 domains

  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✓] Status: No changes detected
  [i] Received 26853 domains

  [i] Target: http://sysctl.org/cameleon/hosts
  [✓] Status: No changes detected
  [i] Received 20567 domains

  [i] Target: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
  [✓] Status: Retrieval successful
  [i] Received 0 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  [✓] Status: No changes detected
  [i] Received 34 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: No changes detected
  [i] Received 2701 domains

  [i] Target: https://dbl.oisd.nl
  [✓] Status: Retrieval successful
  [i] Received 1242547 domains

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [i] Number of gravity domains: 1351045 (1276390 unique domains)
  [i] Number of exact blacklisted domains: 3
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 6
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

[C0D3-IT]

I got the exact same issue.....

[fl0wfr]

Same issue on 5.2. 4.x releases where much more stable :/

[cgergen]

Same issue here.

Pi-hole v5.1.1
Web Interface v5.1
FTL v5.1

[dschaper]

Pi-hole v5.1.1
Web Interface v5.1
FTL v5.1

Update your install.

[fl0wfr]

Pi-hole v5.1.1
Web Interface v5.1
FTL v5.1

Update your install.

That won't change a thing, same issue on 5.2...

[dschaper]

Okay, tried to help, see help is unwanted. Will stop further help.

Have a nice day!

[fl0wfr]

Okay, tried to help, see help is unwanted. Will stop further help.

Have a nice day!

Help is wanted but please read the whole thread and not the last message. And I don't think that doing the upset one will help either :/
Anyway, a nice day to you too!

[dschaper]

My advice was to the person that posted they were on a previous version without the most recent bugfixes.

But I do appreciate you jumping in to respond for them.

[fl0wfr]

My advice was to the person that posted they were on a previous version without the most recent bugfixes.

But I do appreciate you jumping in to respond for them.

By the way, this behaviour seems similar to the one in the issue #3507
In both cases, no DNS resolution, and the only way to resolve the issue immediately is to restart Pi-Hole.

[cgergen]

Pi-hole v5.1.1
Web Interface v5.1
FTL v5.1

Update your install.

Did that immediately after posting here. Now running Pi-hole v5.1.2 and FTL v5.2.

Also applied all available updates for Raspberry Pi OS via apt full-upgrade, including a kernel update to Linux Pi 5.4.51-v8+

Will report back in a week.

[dzaikos]

I was also experiencing the issue of DNS resolution failing from Saturday to Sunday (occurring when the lists are updated). My Pi-hole install runs in an LXC container (Debian 10.5 for both the container and the host). I narrowed it down to /usr/bin/pkill not being installed. The solution for me was to install the procps package. Since then things have been working fine with no failures.

[dschaper]

@DL6ER Do we need to add pkill|pgrep to the dependencies?

https://github.com/pi-hole/pi-hole/pull/3599/commits/17aabf26f7859600bd3f234530f01c85043c3d59

[DL6ER]

Every distribution I checked had procps installed. I was assuming that no sane system would miss out this package. They will also not have, for instance, ps, top and uptime

Package: procps

/proc file system utilities

This package provides command line and full screen utilities for browsing procfs, a "pseudo" file system dynamically generated by the kernel to provide information about the status of entries in its process table (such as whether the process is running, stopped, or a "zombie").

It contains free, kill, pkill, pgrep, pmap, ps, pwdx, skill, slabtop, snice, sysctl, tload, top, uptime, vmstat, w, and watch.

edit @dschaper The commit you quoted replaced ps by pkill. If ps was available on these systems but pkill isn't they seem to be inconsistent themselves. What do you think?

[dschaper]

If pkill is required for operation then we should check for it's existence.

How to resolve it not being there it the bigger question.

[dzaikos]

I was assuming that no sane system would miss out this package.

I should think it would be part of a normal installation. I noticed it was missing when two things happened:

1. Pi-hole DNS resolution began failing weekly.
2. I was missing top.

While I can't be 100% sure, I think procps was removed on my installation during the upgrade of the container from Debian 9 to 10. There may have been a dependency conflict or something; I don't know, I was doing a bunch of upgrades at the time and it was awhile ago (Pi-hole only recently started needing this command).

Regardless, pkill's presence should be checked because without it Pi-hole fails in a rather non-obvious manner.

[DL6ER]

Yes, we actually have it as a RPM dependency, it is just missing for the DEB world.

https://github.com/pi-hole/pi-hole/blob/6b536b7428a1f57ff34ddc444ded6d3a62b00a38/automated%20install/basic-install.sh#L364

I'm not sure this would have made any difference here because when you upgrade your container after installing Pi-hole, the installer cannot notice that a package was removed.

[dzaikos]

I'm not sure this would have made any difference here because when you upgrade your container after installing Pi-hole, the installer cannot notice that a package was removed.

That makes sense. Are dependencies checked during upgrade? I would have upgraded the container at some point, then later run pihole -up. It doesn't matter either way. For all I know this package was missing when I created the container, I simply have no way of knowing how it came about to not be there. As long as it's checked for during the initial installation, that should be fine. If it a future operating system upgrade breaks it that's not really Pi-hole's fault and this GitHub issue is here and people should be able to find the solution relatively quickly.

Separately, it would be beneficial if @fsantiago07044 could see if the same solution works for them.

[DL6ER]

Are dependencies checked during upgrade?

Yes

[xxRockOnxx]

Experiencing this one as well. Read the whole thread but it seems like there isn't anything helpful so here's another debug log in case it might help: https://tricorder.pi-hole.net/gkjw00u1s0

[cgergen]

Pi-hole v5.1.1
Web Interface v5.1
FTL v5.1

Update your install.

Did that immediately after posting here. Now running Pi-hole v5.1.2 and FTL v5.2.

Also applied all available updates for Raspberry Pi OS via apt full-upgrade, including a kernel update to Linux Pi 5.4.51-v8+

Will report back in a week.

Forgot to come back and report in. I have had no issues since updating Pi-Hole, FTL, and my Raspberry Pi OS.