Adguardhome: Adguard Home has high cpu usage, high memory usage, constantly spikes the cpu and remains unresponsive during filterlist/allowlist or blocklist changes - insanely more so than pihole.

Created on 24 Aug 2020  ·  15Comments  ·  Source: AdguardTeam/AdGuardHome

Prerequisites

  • [x] I am running the latest version
  • [x] I checked the documentation and found no answer
  • [x] I checked to make sure that this issue has not already been filed

Issue Details

  • Version of AdGuard Home server:

    • 0.103.3 , raspi 32 bit arm version

  • How did you setup DNS configuration:

    • dnsmasq handing it out as DHCP option

  • If it's a router or IoT, please write device model:

    • Router: Zbox Nano with Fedora, Adguard Home: Raspberry Pi 3B

  • Operating system and version:

    • Raspian current 32bit, Fedora 32 on Raspi4 with rpi4-uefi

Expected Behavior

When comparing adguard home to pihole i expected the following:

For the same amount of Filterlists it should remain as responsive, even if there are a lot
Remain fast when i add or delete entries to the blocklists/allowlists
Startup fast-ish
To not constantly use cpu time

Actual Behavior

When booting the pi or restarting adguard home it takes ~2 Minutes to initialize and build the filterlists.
The same goes for when i add or remove blocklist entires and adguard entirely stops responding for up to several minutes.
This also shows in the "average time to process" since those queries are then listed as taking 3+ Minutes.

Adguard, after boot, constantly spikes on the cpu. It entirely goes 100% for 2 Minutes while rebuilding filterlists and keeps constantly using a surprisingly huge amount of cpu time afterwards. This also drives the pi into much higher thermals than with pihole.

Of course, entirely not responding via dns during filterlist rebuild - or boot - is less than an optimal experience.

And to top the above it uses several times the memory of pihole, making it only feasible to be run on a larger vm/raspi4 because the system already starts swapping on the pi3.

I expected it to be a smooth swap-over to get Dnscrypt/DoH/DoT but in this state, especially since debugging takes 2-3 minutes to rebuild with no dns response of any kind during the time when i add black/whitelists its not as easy to go entirely in on Adguard Home. Especially since everything here is setup the same way. Same dns servers (just with DoH/DoT/Dnscrypt), same filterlists as pihole before.

Screenshots

You can clearly see when i switched on adguard.

Screenshots:

Cpu vs pihole
The spike on the left is a pihole filterlist update, the spike on the right are adguards'.

adguardcpu

Cpu spiking closeup
adguardcpu-closeup

Memory over time vs pihole
adguardmemory

Full Config:

bind_host: 0.0.0.0
bind_port: 80
users:
- name: [omitted]
  password: [omitted]
http_proxy: ""
language: ""
rlimit_nofile: 0
debug_pprof: false
web_session_ttl: 720
dns:
  bind_host: 0.0.0.0
  port: 53
  statistics_interval: 90
  querylog_enabled: true
  querylog_file_enabled: true
  querylog_interval: 90
  querylog_size_memory: 1000
  anonymize_client_ip: false
  protection_enabled: true
  blocking_mode: default
  blocking_ipv4: ""
  blocking_ipv6: ""
  blocked_response_ttl: 10
  parental_block_host: family-block.dns.adguard.com
  safebrowsing_block_host: standard-block.dns.adguard.com
  ratelimit: 20
  ratelimit_whitelist: []
  refuse_any: true
  upstream_dns:
  - https://dns.digitale-gesellschaft.ch/dns-query
  - tls://dns2.digitalcourage.de#853
  - tls://dns.digitale-gesellschaft.ch#853
  - sdns://AQcAAAAAAAAAEzYyLjIxMC4xNzcuMTg5OjEwNTMgW8vytBGk6u3kvCpl4q88XjqW-w6JJiJ7QBObcFV7gYAfMi5kbnNjcnlwdC1jZXJ0Lm5zMS5pcmlzZWRlbi5mcg
  bootstrap_dns:
  - 46.182.19.48
  - 1.1.1.1
  all_servers: true
  fastest_addr: false
  allowed_clients: []
  disallowed_clients: []
  blocked_hosts: []
  cache_size: 4194304
  cache_ttl_min: 0
  cache_ttl_max: 0
  bogus_nxdomain: []
  aaaa_disabled: false
  enable_dnssec: true
  edns_client_subnet: false
  filtering_enabled: true
  filters_update_interval: 24
  parental_enabled: false
  safesearch_enabled: false
  safebrowsing_enabled: false
  safebrowsing_cache_size: 1048576
  safesearch_cache_size: 1048576
  parental_cache_size: 1048576
  cache_time: 30
  rewrites:
  - domain: [omitted]
    answer: [omitted]
  - domain: [omitted]
    answer: [omitted]
  - domain: [omitted]
    answer: [omitted]
  blocked_services:
  - skype
  - origin
  - epic_games
  - vk
  - snapchat
  - facebook
  - whatsapp
  - mail_ru
  - instagram
  - discord
  - ok
  - tiktok
tls:
  enabled: false
  server_name: ""
  force_https: false
  port_https: 443
  port_dns_over_tls: 853
  allow_unencrypted_doh: false
  strict_sni_check: false
  certificate_chain: ""
  private_key: ""
  certificate_path: ""
  private_key_path: ""
filters:
- enabled: true
  url: https://adaway.org/hosts.txt
  name: https://adaway.org/hosts.txt
  id: 1
- enabled: true
  url: https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt
  name: https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt
  id: 2
- enabled: true
  url: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
  name: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
  id: 3
- enabled: true
  url: https://block.energized.pro/basic/formats/hosts.txt
  name: https://block.energized.pro/basic/formats/hosts.txt
  id: 4
- enabled: true
  url: https://blocklistproject.github.io/Lists/abuse.txt
  name: https://blocklistproject.github.io/Lists/abuse.txt
  id: 5
- enabled: true
  url: https://blocklistproject.github.io/Lists/ads.txt
  name: https://blocklistproject.github.io/Lists/ads.txt
  id: 6
- enabled: true
  url: https://blocklistproject.github.io/Lists/fraud.txt
  name: https://blocklistproject.github.io/Lists/fraud.txt
  id: 7
- enabled: true
  url: https://blocklistproject.github.io/Lists/ransomware.txt
  name: https://blocklistproject.github.io/Lists/ransomware.txt
  id: 8
- enabled: true
  url: https://blocklistproject.github.io/Lists/scam.txt
  name: https://blocklistproject.github.io/Lists/scam.txt
  id: 9
- enabled: true
  url: https://blocklistproject.github.io/Lists/tracking.txt
  name: https://blocklistproject.github.io/Lists/tracking.txt
  id: 10
- enabled: true
  url: https://dbl.oisd.nl/
  name: https://dbl.oisd.nl/
  id: 11
- enabled: true
  url: https://gist.githubusercontent.com/BBcan177/b6df57cef74e28d90acf1eec93d62d3b/raw/f0996cf5248657ada2adb396f3636be8716b99eb/MS-4
  name: https://gist.githubusercontent.com/BBcan177/b6df57cef74e28d90acf1eec93d62d3b/raw/f0996cf5248657ada2adb396f3636be8716b99eb/MS-4
  id: 12
- enabled: true
  url: https://github.com/AdAway/adaway.github.io/blob/master/hosts.txt
  name: https://github.com/AdAway/adaway.github.io/blob/master/hosts.txt
  id: 13
- enabled: true
  url: https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-hosts.txt
  name: https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-hosts.txt
  id: 14
- enabled: true
  url: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
  name: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
  id: 15
- enabled: true
  url: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  name: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  id: 16
- enabled: true
  url: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt
  name: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt
  id: 17
- enabled: true
  url: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list.txt
  name: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list.txt
  id: 18
- enabled: true
  url: https://gnuzilla.gnu.org/filters/blacklist.txt
  name: https://gnuzilla.gnu.org/filters/blacklist.txt
  id: 19
- enabled: true
  url: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
  name: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
  id: 20
- enabled: true
  url: https://hosts.nfz.moe/basic/hosts
  name: https://hosts.nfz.moe/basic/hosts
  id: 21
- enabled: true
  url: https://mirror1.malwaredomains.com/files/justdomains
  name: https://mirror1.malwaredomains.com/files/justdomains
  id: 22
- enabled: true
  url: https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt
  name: https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt
  id: 23
- enabled: true
  url: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
  name: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
  id: 24
- enabled: true
  url: https://paulgb.github.io/BarbBlock/blacklists/domain-list.txt
  name: https://paulgb.github.io/BarbBlock/blacklists/domain-list.txt
  id: 25
- enabled: true
  url: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext
  name: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext
  id: 26
- enabled: true
  url: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
  name: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
  id: 27
- enabled: true
  url: https://phishing.army/download/phishing_army_blocklist_extended.txt
  name: https://phishing.army/download/phishing_army_blocklist_extended.txt
  id: 28
- enabled: true
  url: https://raw.github.com/notracking/hosts-blocklists/master/hostnames.txt
  name: https://raw.github.com/notracking/hosts-blocklists/master/hostnames.txt
  id: 29
- enabled: true
  url: https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
  name: https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
  id: 30
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/adobeblock.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/adobeblock.txt
  id: 31
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/cryptomine.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/cryptomine.txt
  id: 32
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/fakenewsde.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/fakenewsde.txt
  id: 33
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/gamefake.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/gamefake.txt
  id: 34
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/jbfake.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/jbfake.txt
  id: 35
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nintendoblock.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nintendoblock.txt
  id: 36
- enabled: true
  url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nomsdata.txt
  name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nomsdata.txt
  id: 37
- enabled: true
  url: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
  name: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
  id: 38
- enabled: true
  url: https://raw.githubusercontent.com/autinerd/anti-axelspringer-hosts/master/axelspringer-hosts
  name: https://raw.githubusercontent.com/autinerd/anti-axelspringer-hosts/master/axelspringer-hosts
  id: 39
- enabled: true
  url: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
  name: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
  id: 40
- enabled: true
  url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Baidu.txt
  name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Baidu.txt
  id: 41
- enabled: true
  url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/HP.txt
  name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/HP.txt
  id: 42
- enabled: true
  url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/LG.txt
  name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/LG.txt
  id: 43
- enabled: true
  url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Synology.txt
  name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Synology.txt
  id: 44
- enabled: true
  url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Ubisoft.txt
  name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Ubisoft.txt
  id: 45
- enabled: true
  url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Xiaomi.txt
  name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Xiaomi.txt
  id: 46
- enabled: true
  url: https://raw.githubusercontent.com/buggerman/SwitchBlockerForPiHole/master/Paranoid.txt
  name: https://raw.githubusercontent.com/buggerman/SwitchBlockerForPiHole/master/Paranoid.txt
  id: 47
- enabled: true
  url: https://raw.githubusercontent.com/cbuijs/shallalist/master/adv/domains
  name: https://raw.githubusercontent.com/cbuijs/shallalist/master/adv/domains
  id: 48
- enabled: true
  url: https://raw.githubusercontent.com/cbuijs/shallalist/master/costtraps/domains
  name: https://raw.githubusercontent.com/cbuijs/shallalist/master/costtraps/domains
  id: 49
- enabled: true
  url: https://raw.githubusercontent.com/cbuijs/shallalist/master/religion/domains
  name: https://raw.githubusercontent.com/cbuijs/shallalist/master/religion/domains
  id: 50
- enabled: true
  url: https://raw.githubusercontent.com/cbuijs/shallalist/master/spyware/domains
  name: https://raw.githubusercontent.com/cbuijs/shallalist/master/spyware/domains
  id: 51
- enabled: true
  url: https://raw.githubusercontent.com/cbuijs/shallalist/master/tracker/domains
  name: https://raw.githubusercontent.com/cbuijs/shallalist/master/tracker/domains
  id: 52
- enabled: true
  url: https://raw.githubusercontent.com/CHEF-KOCH/Anti-Avast-Telemetry/master/HOSTS.txt
  name: https://raw.githubusercontent.com/CHEF-KOCH/Anti-Avast-Telemetry/master/HOSTS.txt
  id: 53
- enabled: true
  url: https://raw.githubusercontent.com/CHEF-KOCH/NSABlocklist/master/HOSTS/HOSTS
  name: https://raw.githubusercontent.com/CHEF-KOCH/NSABlocklist/master/HOSTS/HOSTS
  id: 54
- enabled: true
  url: https://raw.githubusercontent.com/CHEF-KOCH/PayWall-domains/master/HOSTS.txt
  name: https://raw.githubusercontent.com/CHEF-KOCH/PayWall-domains/master/HOSTS.txt
  id: 55
- enabled: true
  url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra.txt
  name: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra.txt
  id: 56
- enabled: true
  url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
  name: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
  id: 57
- enabled: true
  url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update.txt
  name: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update.txt
  id: 58
- enabled: true
  url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
  name: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
  id: 59
- enabled: true
  url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt
  name: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt
  id: 60
- enabled: true
  url: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
  name: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
  id: 61
- enabled: true
  url: https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt
  name: https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt
  id: 62
- enabled: true
  url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
  name: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
  id: 63
- enabled: true
  url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
  name: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
  id: 64
- enabled: true
  url: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt
  name: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt
  id: 65
- enabled: true
  url: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/nocoin.txt
  name: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/nocoin.txt
  id: 66
- enabled: true
  url: https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/Top500
  name: https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/Top500
  id: 67
- enabled: true
  url: https://raw.githubusercontent.com/KurzGedanke/kurzBlock/master/kurzBlock.txt
  name: https://raw.githubusercontent.com/KurzGedanke/kurzBlock/master/kurzBlock.txt
  id: 68
- enabled: true
  url: https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt
  name: https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt
  id: 69
- enabled: true
  url: https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list
  name: https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list
  id: 70
- enabled: true
  url: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/hosts.txt
  name: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/hosts.txt
  id: 71
- enabled: true
  url: https://raw.githubusercontent.com/mmotti/adguard-home-filters/master/filters.txt
  name: https://raw.githubusercontent.com/mmotti/adguard-home-filters/master/filters.txt
  id: 72
- enabled: true
  url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
  name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
  id: 73
- enabled: true
  url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
  name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
  id: 74
- enabled: true
  url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt
  name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt
  id: 75
- enabled: true
  url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
  name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
  id: 76
- enabled: true
  url: https://raw.githubusercontent.com/pirat28/IHateTracker/master/iHateTracker.txt
  name: https://raw.githubusercontent.com/pirat28/IHateTracker/master/iHateTracker.txt
  id: 77
- enabled: true
  url: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt
  name: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt
  id: 78
- enabled: true
  url: https://raw.githubusercontent.com/PoorPocketsMcNewHold/steamscamsites/master/steamscamsite.txt
  name: https://raw.githubusercontent.com/PoorPocketsMcNewHold/steamscamsites/master/steamscamsite.txt
  id: 79
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Corona-Blocklist
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Corona-Blocklist
  id: 80
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/crypto
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/crypto
  id: 81
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Fake-Science
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Fake-Science
  id: 82
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/gambling
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/gambling
  id: 83
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/malware
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/malware
  id: 84
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/notserious
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/notserious
  id: 85
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Phishing-Angriffe
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Phishing-Angriffe
  id: 86
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/samsung
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/samsung
  id: 87
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/spam.mails
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/spam.mails
  id: 88
- enabled: true
  url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Streaming
  name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Streaming
  id: 89
- enabled: true
  url: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
  name: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
  id: 90
- enabled: true
  url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts
  name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts
  id: 91
- enabled: true
  url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts
  name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts
  id: 92
- enabled: true
  url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts
  name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts
  id: 93
- enabled: true
  url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts
  name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts
  id: 94
- enabled: true
  url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts
  name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts
  id: 95
- enabled: true
  url: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  name: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  id: 96
- enabled: true
  url: https://raw.githubusercontent.com/vokins/yhosts/master/hosts
  name: https://raw.githubusercontent.com/vokins/yhosts/master/hosts
  id: 97
- enabled: true
  url: https://raw.githubusercontent.com/wlqY8gkVb9w1Ck5MVD4lBre9nWJez8/W10TelemetryBlocklist/master/W10TelemetryBlocklist
  name: https://raw.githubusercontent.com/wlqY8gkVb9w1Ck5MVD4lBre9nWJez8/W10TelemetryBlocklist/master/W10TelemetryBlocklist
  id: 98
- enabled: true
  url: https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
  name: https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
  id: 99
- enabled: true
  url: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  name: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  id: 100
- enabled: true
  url: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  name: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  id: 101
- enabled: true
  url: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  name: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  id: 102
- enabled: true
  url: https://someonewhocares.org/hosts/zero/hosts
  name: https://someonewhocares.org/hosts/zero/hosts
  id: 103
- enabled: true
  url: https://ssl.bblck.me/blacklists/hosts-file.txt
  name: https://ssl.bblck.me/blacklists/hosts-file.txt
  id: 104
- enabled: true
  url: https://sysctl.org/cameleon/hosts
  name: https://sysctl.org/cameleon/hosts
  id: 105
- enabled: true
  url: https://urlhaus.abuse.ch/downloads/hostfile/
  name: https://urlhaus.abuse.ch/downloads/hostfile/
  id: 106
- enabled: true
  url: https://v.firebog.net/hosts/BillStearns.txt
  name: https://v.firebog.net/hosts/BillStearns.txt
  id: 107
- enabled: true
  url: https://v.firebog.net/hosts/Easylist.txt
  name: https://v.firebog.net/hosts/Easylist.txt
  id: 108
- enabled: true
  url: https://v.firebog.net/hosts/Easyprivacy.txt
  name: https://v.firebog.net/hosts/Easyprivacy.txt
  id: 109
- enabled: true
  url: https://v.firebog.net/hosts/Prigent-Ads.txt
  name: https://v.firebog.net/hosts/Prigent-Ads.txt
  id: 110
- enabled: true
  url: https://v.firebog.net/hosts/Prigent-Malware.txt
  name: https://v.firebog.net/hosts/Prigent-Malware.txt
  id: 111
- enabled: true
  url: https://v.firebog.net/hosts/Prigent-Phishing.txt
  name: https://v.firebog.net/hosts/Prigent-Phishing.txt
  id: 112
- enabled: true
  url: https://v.firebog.net/hosts/Shalla-mal.txt
  name: https://v.firebog.net/hosts/Shalla-mal.txt
  id: 113
- enabled: true
  url: https://v.firebog.net/hosts/static/SamsungSmart.txt
  name: https://v.firebog.net/hosts/static/SamsungSmart.txt
  id: 114
- enabled: true
  url: https://v.firebog.net/hosts/static/w3kbl.txt
  name: https://v.firebog.net/hosts/static/w3kbl.txt
  id: 115
- enabled: true
  url: https://winhelp2002.mvps.org/hosts.txt
  name: https://winhelp2002.mvps.org/hosts.txt
  id: 116
- enabled: true
  url: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
  name: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
  id: 117
- enabled: true
  url: https://www.github.developerdan.com/hosts/lists/amp-hosts-extended.txt
  name: https://www.github.developerdan.com/hosts/lists/amp-hosts-extended.txt
  id: 118
- enabled: true
  url: https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt
  name: https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt
  id: 119
- enabled: true
  url: https://www.malwaredomainlist.com/hostslist/hosts.txt
  name: https://www.malwaredomainlist.com/hostslist/hosts.txt
  id: 120
- enabled: true
  url: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
  name: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
  id: 121
whitelist_filters: []
user_rules:
- [omitted]
- '@@||www.hcaptcha.com^$important'
- ""
dhcp:
  enabled: false
  interface_name: ""
  gateway_ip: ""
  subnet_mask: ""
  range_start: ""
  range_end: ""
  lease_duration: 86400
  icmp_timeout_msec: 1000
clients: []
log_compress: false
log_localtime: false
log_max_backups: 0
log_max_size: 100
log_max_age: 3
log_file: ""
verbose: false
schema_version: 6

Additional Information

question wontfix
Source
picture waffshappen
👍1

All 15 comments

Of course, entirely not responding via dns during filterlist rebuild - or boot - is less than an optimal experience.

AGH doesn't respond to DNS queries until the first initialization of filtering rules on startup - true. But when filters are being updated, added or removed AGH keeps responding to DNS queries as usual.

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C.
Further performance optimization of AGH may take much time to achieve, which means implementing less features. But of course we'll do what we can.

szolin picture szolin on 24 Aug 2020

@waffshappen Please

  1. What's the average load (requests per second)?
  2. What do you mean by "debugging"? Editing custom filtering rules or something else?

@szolin

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C.

The main part of the difference is due to using an encrypted DNS server vs plain DNS. Other than that, the difference is negligible.

What bothers me more is that we don't handle such configurations (dozens of blocklists with millions of rules) well enough. Let's see what usecases @waffshappen and check if we can avoid full engine re-init.

ameshkov picture ameshkov on 24 Aug 2020

The main part of the difference is due to using an encrypted DNS server vs plain DNS

Oh, I didn't notice he's comparing plain vs encrypted. In that case C vs Go doesn't matter of course.

szolin picture szolin on 24 Aug 2020

Of course, entirely not responding via dns during filterlist rebuild - or boot - is less than an optimal experience.

AGH doesn't respond to DNS queries until the first initialization of filtering rules on startup - true. But when filters are being updated, added or removed AGH keeps responding to DNS queries as usual.

Thats precisely the issue. When adding a blocklist while its running it is spinning for a solid minute and not answering any query during the time for me. (Web gui remains active)

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C.
Further performance optimization of AGH may take much time to achieve, which means implementing less features. But of course we'll do what we can.

The cpu spiking does look more than just c/go as difference. Something is keeping adguard active the entire time.

@waffshappen Please

  1. What's the average load (requests per second)?

Unless i'm running dnstorment i max out on 10-20 req/sec when all devices are being used in parallel and browse the web. But the entire time you can see up there it was <1req/sec

  1. What do you mean by "debugging"? Editing custom filtering rules or something else?

Any and all actions. Adding allow/blocklist entries, adding or removing rules, anything really.
The dns service doesnt respond at all during that time.
(Web gui remains up)

@szolin

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C.

The main part of the difference is due to using an encrypted DNS server vs plain DNS. Other than that, the difference is negligible.

What bothers me more is that we don't handle such configurations (dozens of blocklists with millions of rules) well enough. Let's see what usecases @waffshappen and check if we can avoid full engine re-init.

The ideal case would be to reduce the constant cpu load - but i am not sure what is generating it. I could try to selfbuild it and get some perf stats on it if needed and if it is not reproducible with my config posted above. And of course remaining responsive during the time it is rebuilding and ideally saving the cached last build so it starts faster.

waffshappen picture waffshappen on 24 Aug 2020

Screenshot during a blocklist update with nslookup timing out

adguardupdateblocklist

waffshappen picture waffshappen on 24 Aug 2020

@szolin

I guess this is actually several issues in one, we should investigate them all and create new tasks on GH for each of them.

  1. Unnecessary blocking on filtering engine reload;
  2. Constant CPU load -- I don't understand where it comes from; Try to repro with the @waffshappen's configuration.
ameshkov picture ameshkov on 24 Aug 2020

Created the two issues, but there's several more points i'd maybe bring up:

  • Possible caching of the generated database on reboots?
  • Possibly doing the same as pihole and building a deduplicated database in parallel, iirc they're using sqlite/something custom to do it?
  • And going on a stretch: Would it be possible to lessen database load and build times by adding the check if a domain is (possibly) in the blocklist with a bloom filter? Additionally to deduplicating entries from blocklists.

I'll selfbuild latest release in a moment and try to track down what keeps spiking.

waffshappen picture waffshappen on 24 Aug 2020
👍1

Possible caching of the generated database on reboots?

Possible, but I'd better avoid doing this if possible.

Possibly doing the same as pihole and building a deduplicated database in parallel, iirc they're using sqlite/something custom to do it?

What do you mean by "in parallel"?

And going on a stretch: Would it be possible to lessen database load and build times by adding the check if a domain is (possibly) in the blocklist with a bloom filter? Additionally to deduplicating entries from blocklists.

I think it would, but a straightforward approach to deduplication involves will require quite a lot of RAM.

ameshkov picture ameshkov on 28 Aug 2020

Possible caching of the generated database on reboots?

Possible, but I'd better avoid doing this if possible.

True, but it'd be a short-term solution

Possibly doing the same as pihole and building a deduplicated database in parallel, iirc they're using sqlite/something custom to do it?

What do you mean by "in parallel"?

pihole uses a sqlite database and then blasts the entries into it. That uses significantly less memory and has with free deduplication built into it since its an sql engine.

Plus it separates domains and user filters/regexes. So a change to the latter is basically instant by only requiring a single insert.

https://github.com/pi-hole/pi-hole/blob/master/gravity.sh

This would be doable straight in go, by loading the sqlite handler and swapping it in for all current storage methods using either a on-disk, or even in-memory database, with far less memory use than whatever the current system does, in multiple threads handling several lists at once.

If possible for adguard to support huge filterlists instead of building them i guess straight into memory with several parallel workers why not re-use existing validation filters instead of spawning new objects (as it seems right now from pprof) and then having multiple write references to an in-memory sqlite db, or even on-disk, that can do automatic deduplication of entries (INSERT OR IGNORE etc). That would possibly save a lot of memory, and the less objects spawn the less the need for manually calling memory flush, fixing that issue too.

And going on a stretch: Would it be possible to lessen database load and build times by adding the check if a domain is (possibly) in the blocklist with a bloom filter? Additionally to deduplicating entries from blocklists.

I think it would, but a straightforward approach to deduplication involves will require quite a lot of RAM.

pihole already pulls this off, by using sqlite, with minimal memory overuse.
In theory i could try adding this as Minimum Reproduction - replacing all references to the current storage method with sqlite calls and embedding it with go, but that'd also require CGO to be enabled, introducing c bindings and needing to build with musl libc and net_go tag to allow to keep the current amount of supported platforms. (pi-hole solves this by requiring the user to install sqlite and doing it so automatically in their install script)

Alternatively minimal storages like this exist in pure go. Would either request be evaluated or would any CGO requiring sqlite request be dropped?

waffshappen picture waffshappen on 28 Aug 2020

I'd like to avoid using any CGO to keep AGH written in pure Go.

instead of spawning new objects (as it seems right now from pprof) and then having multiple write references to an in-memory sqlite db, or even on-disk

These objects aren't long-living, in the end, we are keeping a simple index:

lookupTable map[uint32][]int64 // map for hosts hashes mapped to the list of rule indexes

Rule indexes point to the rule location in the file.

Unfortunately, Go is not too good at freeing memory. Even if we avoid creating these temporary structs, mere strings will be still allocated and that would still quite a lot of allocations.

Our further actions should depend on what exactly we're going to improve.

  1. Memory allocations on filters initialization: I don't think there's much that can be done about this if we keep AGH written in pure Go.
  2. Filters engine initialization performance:

    • This can be solved, DNSEngine just needs to support serialization/deserialization to a file.

    • Changes to custom filtering rules need to be handled independently and shouldn't lead to reloading the whole filtering engine.

ameshkov picture ameshkov on 28 Aug 2020

Same question,I'm running in Raspberrypi 2B

XrosSJ picture XrosSJ on 4 Sep 2020
👍1

@ameshkov 开发者你好
某一天我发现adh高cpu占用，进一步排查发现：上游dns服务器失效的时候，adh会出现高cpu占用，内存使用也随着时间增加而增加。不知道是不是个例。

sekaiacg picture sekaiacg on 28 Oct 2020

@sekaiacg 上游dns服务器失效的时候，AGH不能解析，它必须等每个解析查询，结果AGH做更多“goroutine“也用更多cpu时间。

如果你用一些上游dns服务器也在dns设置启用负载均衡，AGH就用更快的上游dns服务器。

ameshkov picture ameshkov on 28 Oct 2020

@sekaiacg 上游dns服务器失效的时候，AGH不能解析，它必须等每个解析查询，结果AGH做更多“goroutine“也用更多cpu时间。

如果你用一些上游dns服务器也在dns设置启用负载均衡，AGH就用更快的上游dns服务器。

好的谢谢解答

sekaiacg picture sekaiacg on 28 Oct 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] picture stale[bot] on 27 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

thb007 picture thb007  ·  3Comments
ammnt picture ammnt  ·  3Comments
ammnt picture ammnt  ·  3Comments
ameshkov picture ameshkov  ·  3Comments
alexpovel picture alexpovel  ·  3Comments