DietPi Software | PiHole: "Queries over time" show only graph 1 hour

Created on 26 Mar 2017  ·  21Comments  ·  Source: MichaIng/DietPi

170326-0001

Tested devices:

  • RPi Zero W
  • NanoPi Neo
  • RPi3
  • OrangePi One

DietPi Version:

  • v1.46
  • v1.47

Installed software:

  • SSH Server: Dropbear
  • File Server: None
  • Log System: DietPi-Ramlog #2
  • Webserver Preference: Lighttpd
  • User Data Location: SD/EMMC | /mnt/dietpi_userdata

Install PiHole via dietpi-software

PiHole Version:

~# pihole -v
::: Pi-hole version is v2.13.2 (Latest version is v2.13.2)
::: Web-Admin version is v2.5.2 (Latest version is v2.5.2)

PiHole logfile:

root@NanoPi-Neo:~# cat /var/log/pihole.log
Mar 26 16:17:01 dnsmasq[9486]: gestartet, Version 2.72, Cachegröße 10000
Mar 26 16:17:01 dnsmasq[9486]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect
Mar 26 16:17:01 dnsmasq[9486]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 26 16:17:01 dnsmasq[9486]: Benutze Namensserver 192.168.1.1#53
Mar 26 16:17:01 dnsmasq[9486]: Benutze Namensserver 192.168.1.55#53
Mar 26 16:17:01 dnsmasq[9486]: /etc/hosts gelesen - 6 Adressen
Mar 26 16:17:01 dnsmasq[9486]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 26 16:17:10 dnsmasq[9486]: /etc/pihole/gravity.list gelesen - 347296 Adressen
Mar 26 16:21:41 dnsmasq[9486]: query[A] github.com from 192.168.1.102
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.1
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.55
Mar 26 16:21:41 dnsmasq[9486]: query[AAAA] github.com from 192.168.1.102
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.1
Mar 26 16:21:41 dnsmasq[9486]: forwarded github.com to 192.168.1.55
Mar 26 16:21:41 dnsmasq[9486]: reply github.com is 192.30.253.113
Mar 26 16:21:41 dnsmasq[9486]: reply github.com is 192.30.253.112
.....

Fiddle around a lot, e.g. fresh install pihole, different devices, starting with new images, install by hand or dietpi.txt, Ramlog 1 or 2 or no, but with no success. 😢

I think it is a problem in dietpi-logclear with dnsmasq:

https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-logclear#L108

because every hour at xx:17:01 dnsmasq log start new again.

Wondering, why no other DietPi user notice graphing issue in the PiHole web interface ... ❓

_BTW: my main productive system, with a old and outdated PiHole installation, it works like a charm ..._

170326-0002

Bug
Source
picture k-plan
❤1 👍1

Most helpful comment

@k-plan

Only copy your command from here 😃

Yep, my bad 👍

picture Fourdee on 30 Mar 2017
😄1 👍1

All 21 comments

@k-plan

Thanks for the report 👍

There is an exclusion for PiHole's log file clearing with DietPi-Ramlog. It should allow for 24 hour logs. Let me run some tests here and check source.

edit:

Log System: DietPi-Ramlog #2

Most likely this, Mode 2 probably moving the log contents, before 24h check is done.

Notes:

  • Option 1: exclude PiHole logs from backup in Mode 2
  • Option 2: 24h timer before backup of logs and clear in Mode 2. Chance of loosing <24h of PiHole logs if power failure.
Fourdee picture Fourdee on 26 Mar 2017
🎉1 👍1

@Fourdee

yes, I noticed this, but can't understand or debug, what your script is doing ... sorry for my disability. 😃

On the outdated PiHole installation, with Ramlog 2, graphing will works like a charm.
Noticed as well, that here is no "preamble" in the log-file and it will always start at 00:00:xx :

root@Odroid-C2:~# cat /var/log/pihole.log
Mar 26 00:00:35 dnsmasq[32401]: query[AAAA] apple-mobile.query.yahooapis.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.3
Mar 26 00:00:35 dnsmasq[32401]: query[A] apple-mobile.query.yahooapis.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded apple-mobile.query.yahooapis.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: query[AAAA] guzzoni.apple.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded guzzoni.apple.com to 192.168.1.1
Mar 26 00:00:35 dnsmasq[32401]: query[A] guzzoni.apple.com from 192.168.1.112
Mar 26 00:00:35 dnsmasq[32401]: forwarded guzzo
.....
k-plan picture k-plan on 26 Mar 2017
👍1

@k-plan

On the outdated PiHole installation, with Ramlog 2, graphing will works like a charm.

Yep, checked code, looks fine. Shouldn't trigger a clear unless previous day.

Noticed as well, that here is no "preamble" in the log-file and it will always start at 00:00:xx

Thanks, i'll check 👍

Fourdee picture Fourdee on 26 Mar 2017
👍1

Yep, checked code, looks fine. Shouldn't trigger a clear unless previous day.

The first screenshot is taken from this device (NanoPi v146)

root@NanoPi-Neo:~# uptime
 21:04:54 up 7 days,  8:57,  1 user,  load average: 0,01, 0,04, 0,08

But it's the same, as well on RPi Zero W or OPi One (latest image version) updated to v147,
running Pi-hole v2.13.2 with Admin-LTE v2.5.2

k-plan picture k-plan on 26 Mar 2017
👍1

Tests with Ramlog mode 2:

  • 🈯️ Confirmed worked on DietPi v142 system. Possible log format has changed in later version of PiHole, which will break scrape. Will test.
  • 🈯️ Fresh install of v147 system
    image
Fourdee picture Fourdee on 27 Mar 2017

@k-plan

Really strange, I cant seem to replicate this on RAMlog mode 1/2. PiHole logs are only cleared if day != today, or 5MB limit.

Just wondering if your PiHole log file is hitting 5MB> in size? in which case, it will be cleared to free up RAMlog (20MB tmpfs max):
https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-logclear#L128-L136

Other thing could be locale related, causing a failed scrape. Running en_GB.UTF-8? Others should be fine, as long as en_GB.UTF-8 is still installed.

Fourdee picture Fourdee on 27 Mar 2017

@Fourdee

Just wondering if your PiHole log file is hitting 5MB> in size?

No, it's only my personal test device for my single computer.
Rest of the hole network use the old PiHole installation on a other device.

root@NanoPi-Neo:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 102K Mär 27 16:13 /var/log/pihole.log

Other thing could be locale related, causing a failed scrape.

root@NanoPi-Neo:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX

root@NanoPi-Neo:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

But anyway ... shall we start form the scratch? _(hope it is the last time, I do this ...)_
Which device? NanoPi Neo (v.1), OPi One or RPi Zero (W)?
Last image version? Update to last DietPi?
RAM-Log 2? May I change locale to de_DE?
Install by hand or full automated?

k-plan picture k-plan on 27 Mar 2017

@Fourdee

OPi One installation:

  • image: DietPi_v145_NanoPiNEO-armv7-(Jessie).img

Software:

  • RAM Log 2
  • lighttp
  • mc
  • PiHole
root@OPi-One:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX


root@OPi-One:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
  • first PiHole installation on OPi-One failed/stop (before dialog for asking to install Admin-LTE), show in terminal Fail in /DietPi/dietpi/dietpi-software line 7xxx (if i remember correct),
    follow by a automated reboot, PiHole was not installed (tested: pihohle -v - not found)
    but PiHole is shown in dietpi-software as installed!

So choose for uninstall and after this run install via dietpi-software again.

After automated reboot, PiHole works.

Setup Pihole:
cp /etc/pihole/adlists.default /etc/pihole/adlists.list
nano /etc/pihole/adlists.list
pihole -g

Tested and work:

root@OPi-One:~# cat /var/log/pihole.log
Mar 27 17:30:16 dnsmasq[1011]: started, version 2.72 cachesize 10000
Mar 27 17:30:16 dnsmasq[1011]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 17:30:16 dnsmasq[1011]: warning: ignoring resolv-file flag because no-resolv is set
Mar 27 17:30:16 dnsmasq[1011]: using nameserver 192.168.1.1#53
Mar 27 17:30:16 dnsmasq[1011]: using nameserver 192.168.1.55#53
Mar 27 17:30:16 dnsmasq[1011]: read /etc/hosts - 2 addresses
Mar 27 17:30:16 dnsmasq[1011]: read /etc/pihole/local.list - 4 addresses
Mar 27 17:30:36 dnsmasq[1011]: read /etc/pihole/gravity.list - 204894 addresses
Mar 27 17:31:09 dnsmasq[1011]: query[A] api.github.com from 127.0.0.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.55
Mar 27 17:31:09 dnsmasq[1011]: query[AAAA] api.github.com from 127.0.0.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.1
Mar 27 17:31:09 dnsmasq[1011]: forwarded api.github.com to 192.168.1.55
Mar 27 17:31:09 dnsmasq[1011]: reply api.github.com is 192.30.253.116
Mar 27 17:31:09 dnsmasq[1011]: reply api.github.com is 192.30.253.117
Mar 27 17:31:10 dnsmasq[1011]: query[A] api.github.com from 127.0.0.1
Mar 27 17:31:10 dnsmasq[1011]: cached api.github.com is 192.30.253.117
Mar 27 17:31:10 dnsmasq[1011]: cached api.github.com is 192.30.253.116
Mar 27 17:31:10 dnsmasq[1011]: query[AAAA] api.github.com from 127.0.0.1
....

Graphing in "Queries over time" start working after 10 min.

After 47 min:

170327-0001

root@OPi-One:~# cat /var/log/pihole.log
Mar 27 18:17:02 dnsmasq[5818]: started, version 2.72 cachesize 10000
Mar 27 18:17:02 dnsmasq[5818]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 18:17:02 dnsmasq[5818]: warning: ignoring resolv-file flag because no-resolv is set
Mar 27 18:17:02 dnsmasq[5818]: using nameserver 192.168.1.1#53
Mar 27 18:17:02 dnsmasq[5818]: using nameserver 192.168.1.55#53
Mar 27 18:17:02 dnsmasq[5818]: read /etc/hosts - 2 addresses
Mar 27 18:17:02 dnsmasq[5818]: read /etc/pihole/local.list - 4 addresses
Mar 27 18:17:10 dnsmasq[5818]: read /etc/pihole/gravity.list - 343244 addresses
Mar 27 18:18:01 dnsmasq[5818]: query[A] raw.githubusercontent.com from 192.168.1.50
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.1
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.55
Mar 27 18:18:01 dnsmasq[5818]: query[AAAA] raw.githubusercontent.com from 192.168.1.50
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.1
Mar 27 18:18:01 dnsmasq[5818]: forwarded raw.githubusercontent.com to 192.168.1.55
Mar 27 18:18:01 dnsmasq[5818]: reply raw.githubusercontent.com is <CNAME>
Mar 27 18:18:01 dnsmasq[5818]: reply github.map.fastly.net is 151.101.112.133
Mar 27 18:18:01 dnsmasq[5818]: reply raw.githubusercontent.com is <CNAME>
Mar 27 18:18:01 dnsmasq[5818]: reply github.map.fastly.net is NODATA-IPv6
Mar 27 18:18:30 dnsmasq[5818]: query[A] api.github.com from 192.168.1.102
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.1
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:18:30 dnsmasq[5818]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 18:18:30 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:18:30 dnsmasq[5818]: reply api.github.com is 192.30.253.116
Mar 27 18:18:30 dnsmasq[5818]: reply api.github.com is 192.30.253.117
Mar 27 18:19:34 dnsmasq[5818]: query[A] api.github.com from 192.168.1.102
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.1
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:19:34 dnsmasq[5818]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 18:19:34 dnsmasq[5818]: forwarded api.github.com to 192.168.1.55
Mar 27 18:19:34 dnsmasq[5818]: reply api.github.com is 192.30.253.117
Mar 27 18:19:34 dnsmasq[5818]: reply api.github.com is 192.30.253.116
....

👎

Now installation on RPi-Zero-W is running ....

k-plan picture k-plan on 27 Mar 2017

@Fourdee

RPi-Zero-W installation:

  • image: DietPi_v145_RPi-armv6-(Jessie).img

Software:

  • RAM Log 2
  • lighttp
  • mc
  • PiHole
root@RPi-Zero-W:~# locale -a
C
C.UTF-8
de_DE.utf8
en_GB.utf8
POSIX

root@RPi-Zero-W:~# locale
LANG=de_DE.UTF-8
LANGUAGE=
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=
  • PiHole installation on RPi-Zero-W works without a problem.

After automated reboot, PiHole works.

Setup Pihole:
cp /etc/pihole/adlists.default /etc/pihole/adlists.list
nano /etc/pihole/adlists.list
pihole -g

170327-0003

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 18:20:15 dnsmasq[946]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 18:20:15 dnsmasq[946]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntra$
Mar 27 18:20:15 dnsmasq[946]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 18:20:15 dnsmasq[946]: Benutze Namensserver 192.168.1.55#53
Mar 27 18:20:15 dnsmasq[946]: Benutze Namensserver 192.168.1.66#53
Mar 27 18:20:15 dnsmasq[946]: /etc/hosts gelesen - 2 Adressen
Mar 27 18:20:15 dnsmasq[946]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 18:20:29 dnsmasq[946]: /etc/pihole/gravity.list gelesen - 204894 Adressen
Mar 27 18:28:27 dnsmasq[946]: query[PTR] lb._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.LAN to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.LAN to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] lb._dns-sd._udp.0.10.16.172.in-addr.arpa from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.0.10.16.172.in-addr.arpa to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded lb._dns-sd._udp.0.10.16.172.in-addr.arpa to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] b._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded b._dns-sd._udp.LAN to 192.168.1.55
Mar 27 18:28:27 dnsmasq[946]: forwarded b._dns-sd._udp.LAN to 192.168.1.66
Mar 27 18:28:27 dnsmasq[946]: query[PTR] db._dns-sd._udp.LAN from 192.168.1.102
Mar 27 18:28:27 dnsmasq[946]: forwarded db._dns-sd._udp.LAN to 192.168.1.55
.......
  • After 57 min:

170327-0004

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 19:17:02 dnsmasq[4170]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 19:17:02 dnsmasq[4170]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 19:17:02 dnsmasq[4170]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 19:17:02 dnsmasq[4170]: Benutze Namensserver 192.168.1.55#53
Mar 27 19:17:02 dnsmasq[4170]: Benutze Namensserver 192.168.1.66#53
Mar 27 19:17:02 dnsmasq[4170]: /etc/hosts gelesen - 2 Adressen
Mar 27 19:17:02 dnsmasq[4170]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 19:17:24 dnsmasq[4170]: /etc/pihole/gravity.list gelesen - 343244 Adressen
Mar 27 19:17:29 dnsmasq[4170]: query[A] gitter.im from 192.168.1.102
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.55
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.66
Mar 27 19:17:29 dnsmasq[4170]: query[AAAA] gitter.im from 192.168.1.102
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.55
Mar 27 19:17:29 dnsmasq[4170]: forwarded gitter.im to 192.168.1.66
Mar 27 19:17:29 dnsmasq[4170]: reply gitter.im is 54.209.199.250
Mar 27 19:17:29 dnsmasq[4170]: reply gitter.im is 54.174.166.63
Mar 27 19:17:39 dnsmasq[4170]: query[A] github.com from 192.168.1.102
Mar 27 19:17:39 dnsmasq[4170]: forwarded github.com to 192.168.1.55
Mar 27 19:17:39 dnsmasq[4170]: query[AAAA] github.com from 192.168.1.102
Mar 27 19:17:39 dnsmasq[4170]: forwarded github.com to 192.168.1.55
Mar 27 19:17:39 dnsmasq[4170]: reply github.com is 192.30.253.112
Mar 27 19:17:39 dnsmasq[4170]: reply github.com is 192.30.253.113
Mar 27 19:17:41 dnsmasq[4170]: query[A] live.github.com from 192.168.1.102
Mar 27 19:17:41 dnsmasq[4170]: forwarded live.github.com to 192.168.1.55
Mar 27 19:17:41 dnsmasq[4170]: query[AAAA] live.github.com from 192.168.1.102
Mar 27 19:17:41 dnsmasq[4170]: forwarded live.github.com to 192.168.1.55
Mar 27 19:17:41 dnsmasq[4170]: reply live.github.com is 192.30.253.124
Mar 27 19:17:41 dnsmasq[4170]: reply live.github.com is 192.30.253.125
Mar 27 19:18:41 dnsmasq[4170]: query[A] api.github.com from 192.168.1.102
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.55
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.66
Mar 27 19:18:41 dnsmasq[4170]: query[AAAA] api.github.com from 192.168.1.102
Mar 27 19:18:41 dnsmasq[4170]: forwarded api.github.com to 192.168.1.55
Mar 27 19:18:41 dnsmasq[4170]: reply api.github.com is 192.30.253.117
Mar 27 19:18:41 dnsmasq[4170]: reply api.github.com is 192.30.253.116
......

Do you noticed "Every Hour at 17 Minutes " on ALL tested devices❓

170327-0005

_update: 20:17:00_

root@RPi-Zero-W:~# cat /var/log/pihole.log
Mar 27 20:17:02 dnsmasq[8208]: gestartet, Version 2.72, Cachegröße 10000
Mar 27 20:17:02 dnsmasq[8208]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 27 20:17:02 dnsmasq[8208]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 27 20:17:02 dnsmasq[8208]: Benutze Namensserver 192.168.1.55#53
Mar 27 20:17:02 dnsmasq[8208]: Benutze Namensserver 192.168.1.66#53
Mar 27 20:17:02 dnsmasq[8208]: /etc/hosts gelesen - 2 Adressen
Mar 27 20:17:02 dnsmasq[8208]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 27 20:17:25 dnsmasq[8208]: /etc/pihole/gravity.list gelesen - 343244 Adressen
Mar 27 20:21:24 dnsmasq[8208]: query[A] github.com from 192.168.1.102
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.55
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.66
Mar 27 20:21:24 dnsmasq[8208]: query[AAAA] github.com from 192.168.1.102
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.55
Mar 27 20:21:24 dnsmasq[8208]: forwarded github.com to 192.168.1.66
Mar 27 20:21:24 dnsmasq[8208]: reply github.com is 192.30.253.113
Mar 27 20:21:24 dnsmasq[8208]: reply github.com is 192.30.253.112
Mar 27 20:21:28 dnsmasq[8208]: query[A] avatars3.githubusercontent.com from 192.168.1.102
Mar 27 20:21:28 dnsmasq[8208]: forwarded avatars3.githubusercontent.com to 192.168.1.66
Mar 27 20:21:28 dnsmasq[8208]: query[AAAA] avatars3.githubusercontent.com from 192.168.1.102
Mar 27 20:21:28 dnsmasq[8208]: forwarded avatars3.githubusercontent.com to 192.168.1.66
Mar 27 20:21:28 dnsmasq[8208]: reply avatars3.githubusercontent.com is <CNAME>
Mar 27 20:21:28 dnsmasq[8208]: reply github.map.fastly.net is 151.101.112.133
Mar 27 20:21:28 dnsmasq[8208]: reply avatars3.githubusercontent.com is <CNAME>
Mar 27 20:21:28 dnsmasq[8208]: reply github.map.fastly.net is NODATA-IPv6
Mar 27 20:21:41 dnsmasq[8208]: query[A] avatars2.githubusercontent.com from 192.168.1.102
Mar 27 20:21:41 dnsmasq[8208]: forwarded avatars2.githubusercontent.com to 192.168.1.66
Mar 27 20:21:41 dnsmasq[8208]: query[AAAA] avatars2.githubusercontent.com from 192.168.1.102
Mar 27 20:21:41 dnsmasq[8208]: forwarded avatars2.githubusercontent.com to 192.168.1.66
Mar 27 20:21:41 dnsmasq[8208]: reply avatars2.githubusercontent.com is <CNAME>
Mar 27 20:21:41 dnsmasq[8208]: reply github.map.fastly.net is 151.101.112.133
Mar 27 20:21:41 dnsmasq[8208]: reply avatars2.githubusercontent.com is <CNAME>
Mar 27 20:21:41 dnsmasq[8208]: reply github.map.fastly.net is NODATA-IPv6
....

Yes, /etc/cron.hourly/ works ....

k-plan picture k-plan on 27 Mar 2017

for testing:

  • unselect locale de_DE on same RPi Zero W and reboot
root@RPi-Zero-W:~# locale -a
C
C.UTF-8
en_GB.utf8
POSIX

root@RPi-Zero-W:~# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

170327-0006

Really strange, I cant seem to replicate this on RAMlog mode 1/2. PiHole logs are only cleared if day != today, or 5MB limit.

No, choose, add and switch locale=de_DE to default and you can see the magic ... 😃

BTW: _Hope, with "Pi-hole FTL engine" all become better._ 😃

Coming Soon To A Pi-hole Near You: FTL

Pi-hole FTL engine

k-plan picture k-plan on 27 Mar 2017

@k-plan

No, choose, add and switch locale=de_DE to default and you can see the magic ...

Thanks, i'll check 👍

FTL

Looks like a change from php to C: https://github.com/pi-hole/FTL/blob/master/main.c should make for much reduced query times, espically with use of multithreading: https://github.com/pi-hole/FTL/blob/master/main.c#L43-L49

If IO on log file has any impact on query time, it "should" run faster on DietPi (RAMlog) 😉

Fourdee picture Fourdee on 28 Mar 2017

@k-plan

https://github.com/pi-hole/FTL#pi-hole-ftl

This project is copyright under the latest version of the EUPL.

Yep, and Sonic is copyright of SEGA lmao

Fourdee picture Fourdee on 28 Mar 2017

Only change default locale=en_GB didn't help. Must remove all other language support.

If IO on log file has any impact on query time, it "should" run faster on DietPi (RAMlog) 😉

Flat file support will decease. Running database in RAMlog? 😉

FTL will help remediate the immediate need for a faster interface. The second component on the roadmap would be using a database instead of trying to query the entire text of the log file in it’s raw format.

k-plan picture k-plan on 28 Mar 2017
👍1

@k-plan

Got it:

  • en_GB default:
root@DietPi:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 29
  • de_DE default with en_GB installed:
root@DietPi:~# LANG=en_GB.UTF-8 date +'%b %e'
Mär 29

This will cause scrape to miss those entries, removing them.

I'll try a fix.

  • 🈯️ 
root@DietPi:~# LC_ALL=en_GB.UTF-8 date +'%b %e'
Mar 29
Fourdee picture Fourdee on 29 Mar 2017
😄1 👍1

@k-plan

This commit should do it:
https://github.com/Fourdee/DietPi/commit/3a6c6a3b1cf5fff8d69103a9fd78507b43b80419

Fourdee picture Fourdee on 29 Mar 2017

@Fourdee

Test is running on one device.
We have to wait a little bit and see what's happens.

Thanks.

k-plan picture k-plan on 29 Mar 2017
👍1

Testing:

  • switch locale language back to my default de_DE:

170330-0001

  • apply your commit by hand:
root@OPi-One:~# cat /DietPi/dietpi/dietpi-logclear | grep 'local month_day_today'
                                        ## local month_day_today=$(LANG=en_GB.UTF-8 date +'%b %e')
                                        local month_day_today=$(LC_ALL=en_GB.UTF-8 date +'%b %e')

  • reboot the device

  • de_DE default with en_GB installed:

root@OPi-One:~# date
Do 30. Mär 13:00:27 CEST 2017

root@OPi-One:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30

root@OPi-One:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
Mär 30


root@OPi-One:~# cat /var/log/pihole.log
Mar 30 10:58:42 dnsmasq[1061]: started, version 2.72 cachesize 10000
Mar 30 10:58:42 dnsmasq[1061]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 30 10:58:42 dnsmasq[1061]: warning: ignoring resolv-file flag because no-resolv is set
Mar 30 10:58:42 dnsmasq[1061]: using nameserver 192.168.1.56#53
Mar 30 10:58:42 dnsmasq[1061]: read /etc/hosts - 2 addresses
Mar 30 10:58:42 dnsmasq[1061]: read /etc/pihole/local.list - 4 addresses
Mar 30 10:59:03 dnsmasq[1061]: read /etc/pihole/gravity.list - 343242 addresses
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] lb._dns-sd._udp.LAN from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: forwarded lb._dns-sd._udp.LAN to 192.168.1.56
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] lb._dns-sd._udp.0.1.168.192.in-addr.arpa from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: forwarded lb._dns-sd._udp.0.1.168.192.in-addr.arpa to 192.168.1.56
Mar 30 11:26:38 dnsmasq[1061]: query[PTR] 102.1.168.192.in-addr.arpa from 192.168.1.102
Mar 30 11:26:38 dnsmasq[1061]: config 192.168.1.102 is NXDOMAIN
....

root@OPi-One:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 101K Mär 30 13:06 /var/log/pihole.log
  • AdminLTE:

170330-0003

  • checkout "testing branch" on a other device:
root@NanoPi-Neo:~# cat /DietPi/dietpi/.version
148


root@NanoPi-Neo:~# date
Do 30. Mär 13:07:46 CEST 2017

root@NanoPi-Neo:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30

root@NanoPi-Neo:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
Mär 30

root@NanoPi-Neo:~# cat /DietPi/dietpi/dietpi-logclear | grep 'local month_day'
                                        local month_day_today=$(LC_ALL=en_GB.UTF-8 date +'%b %e')


Mar 30 10:35:35 dnsmasq[1075]: gestartet, Version 2.72, Cachegröße 10000
Mar 30 10:35:35 dnsmasq[1075]: Übersetzungsoptionen: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntr$
Mar 30 10:35:35 dnsmasq[1075]: Warnung: Ignoriere "resolv-file", weil "no-resolv" aktiv ist
Mar 30 10:35:35 dnsmasq[1075]: Benutze Namensserver 192.168.1.1#53
Mar 30 10:35:35 dnsmasq[1075]: Benutze Namensserver 192.168.1.55#53
Mar 30 10:35:35 dnsmasq[1075]: /etc/hosts gelesen - 6 Adressen
Mar 30 10:35:35 dnsmasq[1075]: /etc/pihole/local.list gelesen - 4 Adressen
Mar 30 10:36:01 dnsmasq[1075]: /etc/pihole/gravity.list gelesen - 347304 Adressen
Mar 30 10:36:01 dnsmasq[1075]: query[A] github.com from 192.168.1.102
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.1
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.55
Mar 30 10:36:01 dnsmasq[1075]: query[AAAA] github.com from 192.168.1.102
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.1
Mar 30 10:36:01 dnsmasq[1075]: forwarded github.com to 192.168.1.55
Mar 30 10:36:01 dnsmasq[1075]: reply github.com is 192.30.253.113
Mar 30 10:36:01 dnsmasq[1075]: reply github.com is 192.30.253.112
....

root@NanoPi-Neo:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 131K Mär 30 13:09 /var/log/pihole.log
  • AdminLTE:

170330-0004

Different language in logfile content depends on device (NanoPi Neo vs. OrangePi One), but it seem to be equal.

k-plan picture k-plan on 30 Mar 2017
👍1

@k-plan

Thanks, looks to be working with the change?

I believe you need to remove the ;, else, doesn't get applied to next command

LC_ALL=en_GB.UTF-8 date +'%b %e'

I left my C2 running today with the code change, de_DE default locale, seems to be working, logs from 01:00 onwards:
image

Think we can mark this one as closed?

Fourdee picture Fourdee on 30 Mar 2017

@Fourdee

I believe you need to remove the ;, else, doesn't get applied to next command

Only copy your command from here 😃

But yes, you're right:

root@RPi-Zero-W:~# date
2017年 03月 30日 星期四 14:22:28 CEST

root@RPi-Zero-W:~# LC_ALL=en_GB.UTF-8; date +'%b %e'
3月 30

root@RPi-Zero-W:~# LC_ALL=en_GB.UTF-8 date +'%b %e'
Mar 30

root@RPi-Zero-W:~# LANG=en_GB.UTF-8 date +'%b %e'
Mar 30
  • My new testing default locale (zh_CN.utf8):
root@RPi-Zero-W:~# locale -a -v

locale: zh_CN.utf8      archive: /usr/lib/locale/locale-archive
-------------------------------------------------------------------------------
    title | Chinese locale for Peoples Republic of China
    email | [email protected]
 language | Chinese
territory | P.R. of China
 revision | 0.1
     date | 2000-07-25
  codeset | UTF-8

locale: en_GB.utf8      archive: /usr/lib/locale/locale-archive
-------------------------------------------------------------------------------
    title | English locale for Britain
   source | RAP
  address | Sankt Jørgens Alle 8, DK-1615 København V, Danmark
  contact | Keld Simonsen
    email | [email protected]
 language | English
territory | Great Britain
 revision | 1.0
     date | 2000-06-28
  codeset | UTF-8

😄 _( ... and British form a Dane - nice!)_ :rofl:

Thanks, looks to be working with the change?

yes, I think so. It seem to be working on my Chinese locale RPi-Zero_W now.

170330-0005

root@RPi-Zero-W:~# ls -lah /var/log/pihole.log
-rwxrwxr-x 1 dnsmasq www-data 81K 3月  30 14:39 /var/log/pihole.log

root@RPi-Zero-W:~# apt-get upgrade
正在读取软件包列表... 完成
正在分析软件包的依赖关系树
正在读取状态信息... 完成
正在对升级进行计算... 完成
下列软件包的版本将保持不变：
  libraspberrypi-bin libraspberrypi0 raspberrypi-bootloader raspberrypi-kernel
升级了 0 个软件包，新安装了 0 个软件包，要卸载 0 个软件包，有 4 个软件包未被升级。

:rofl:

Think we can mark this one as closed?

Yes, if it will run well on your C2 too.

Failure with locales in scripting, just as difficult every time.
Sorry, better to remove language option in DietPi ... 😺

k-plan picture k-plan on 30 Mar 2017
😄1 👍1

@Fourdee

Testing results:

  • RPi-Zero W (zh_CN.utf8):

170330-0006

  • OrangePi One (de_DE.utf8):

170330-0007

  • NanoPi Neo v.1 (de_DE.utf8):

170330-0008

  • _Production system (Odroid C2)_ - non-competitive

170330-0009

k-plan picture k-plan on 30 Mar 2017
👍1

@k-plan

Only copy your command from here 😃

Yep, my bad 👍

Fourdee picture Fourdee on 30 Mar 2017
😄1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

MichaIng picture MichaIng  ·  3Comments
Invictaz picture Invictaz  ·  3Comments
Fourdee picture Fourdee  ·  3Comments
1021683053 picture 1021683053  ·  3Comments
aesirteam picture aesirteam  ·  3Comments