Packages: https-dns-proxy: high CPU/memory usage, crashes

[Mrnofish]

Maintainer: @stangri
Environment: mips 24Kc, TP-Link WR1043ND v1.8, OpenWRT 18.06.7

Description:

To reproduce: start a browser that will restore a saved session, performing a number of look-ups.

CPU usage will skyrocket. On occasion, memory use will also surge triggering an OoM crash that results in a zombie slowing down the router to a crawl (load reads 2 to 3)

Initially, I thought it was just the router having too little memory, however, after switching to stubby, which appears to have a similar memory footprint, everything has been working flawlessly.

[stangri]

@aarond10.

[jamesbo13]

Note - there was a recent patch in upstream to fix a memory leak (https://github.com/aarond10/https_dns_proxy/commit/40647ce94c62a47e9d53efae8018fb3142e277b9) ... perhaps this might be related

[stangri]

Thanks, PRs for the new version: https://github.com/openwrt/packages/pull/11819, https://github.com/openwrt/packages/pull/11820, https://github.com/openwrt/packages/pull/11821.
Once they are merged, the new package should be available in repos withing a few days, please test if it fixes the memory leak.

[dmascord]

This seems potentially related to the ipv4 vs ipv6 configuration.

https://openwrt.org/docs/guide-user/services/dns/doh_dnsmasq_https-dns-proxy suggests to use ipv4_resolvers setting, but the init scripts use the opposite configuration use_ipv6_resolvers_only.

I have modified my configuration to use use_ipv6_resolvers_only 0, rather than ipv4_resolvers 1, and will monitor to see how it behaves.

EDIT - OK, no difference at all, this is just a red herring.

Running apt update on a client machine, with a number of repo's activated often triggers this issue, both dnsmasq and https-dns-proxy consume 50% of the CPU each, and it essentially hangs,

8750 1 nobody R 5052 4% 48% /usr/sbin/https-dns-proxy -a 127.0.0.1 -p 5053 -b 1.1.1.1,1.0.0.1 -r https://cloudflare-dns.com/dns-query -u nobody -g nogroup -v -4
8742 1 dnsmasq R 9564 8% 47% /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid

I can reproduce pretty easily with https://github.com/jedisct1/dnsblast, and can confirm that stubby does not exhibit the same behaviour.

I was running strace on the router, and captured the strace output, and I can see some errors:

connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("104.16.248.249")}, 16) = -1 EINPROGRESS (Operation in progress)
poll([{fd=8, events=POLLOUT}], 1, 0) = 0 (Timeout)

You can see the full strace here - https://tuskernetau-my.sharepoint.com/:t:/g/personal/damien_tusker_net_au/ERmO8oY2i0BEu-UXlVyEQ60BUP20Tt-g0YHgqXDjWDSfUA?e=p0bG6A

[stangri]

@dmascord could you please post your full https-dns-proxy and dnsmasq configs?

[dmascord]

`
root@tp4320:/etc/config# cat https-dns-proxy

config main 'config'
option update_dnsmasq_config '*'

config https-dns-proxy
option bootstrap_dns '1.1.1.1,1.0.0.1'
option resolver_url 'https://cloudflare-dns.com/dns-query'
option listen_addr '127.0.0.1'
option listen_port '5053'
option user 'nobody'
option group 'nogroup'
option use_ipv6_resolvers_only '0'
option verbosity '1'

root@tp4320:/etc/config# cat dhcp

config dnsmasq
option domainneeded '1'
option boguspriv '1'
option filterwin2k '0'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option domain 'lan'
option expandhosts '1'
option nonegcache '0'
option authoritative '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option nonwildcard '1'
option localservice '1'
option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
option serversfile '/tmp/adb_list.overall'
option confdir '/tmp/dnsmasq.d'
option localuse '1'
list server '127.0.0.1#5053'
list server '0::1#5053'
option noresolv '1'

config dhcp 'lan'
option interface 'lan'
option start '100'
option limit '150'
option leasetime '12h'
option dhcpv6 'disabled'
list dhcp_option '6,10.0.0.1'
option ra 'disabled'

config dhcp 'wan'
option interface 'wan'
option ignore '1'

config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'
`

[stangri]

Which version of https-dns-proxy is that?

[dmascord]

strings /usr/sbin/https-dns-proxy:
https-dns-proxy-2020-04-09/src/main.c

[stangri]

Did you modify the init script in any way?

[willymilkytron]

Same issue here, https://github.com/aarond10/https_dns_proxy/commit/40647ce94c62a47e9d53efae8018fb3142e277b9 does not fix it for me.

I noticed memory usages usually increases when my ISP router reboots:

network -> openwrt (https-dns-proxy) -> ISP router -> internet

[aarond10]

Thanks @willymilkytron, that's a useful data point. There is at least one bad thing going on with libcurl but I haven't figured out exactly what yet. Worked fine for months, so I suspect it's triggered by either IP changes (DNS round robin?) or some http/2 server behavior.

[stangri]

@aarond10 do you think it may be beneficial to test with -x on the systems with abnormal CPU/memory usage?

[caliban511]

同样遇到 https_dns_proxy 占用CPU过高的问题。我的CPU是J4205，但这个进程一开，CPU平均占用就会上升到50%左右。这个问题不知道什么时候能解决一下？

[aarond10]

I'll need more details. Ideally some gprof output with a debug build or at
the least the output of the daemon running with -vvvv so I can try to infer
what it's doing.

On Mon, 28 Sep 2020 at 02:56, Caliban [e-mail] wrote:

同样遇到 https_dns_proxy
占用CPU过高的问题。我的CPU是J4205，但这个进程一开，CPU平均占用就会上升到50%左右。这个问题不知道什么时候能解决一下？

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openwrt/packages/issues/11410#issuecomment-699659949,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABTOXUVJSKWPEEUPKUVQETSH5VELANCNFSM4KZVP7CQ
.

[caliban511]

你好，请看下最上方占用25%CPU的那个，就这一条就占用掉25%，然后整个Openwrt的占用从7%左右上升到45%以上。

I'll need more details. Ideally some gprof output with a debug build or at the least the output of the daemon running with -vvvv so I can try to infer what it's doing.

[caliban511]

请问这两个任务是在做什么呢？为什么占用那么大？

I'll need more details. Ideally some gprof output with a debug build or at the least the output of the daemon running with -vvvv so I can try to infer what it's doing.

[caliban511]

I'll need more details. Ideally some gprof output with a debug build or at the least the output of the daemon running with -vvvv so I can try to infer what it's doing.

大佬你好，请问你用过passwall.ipk这个软件吗？我试过用最新版19.07.4的原版openwrt，安装passwall，会自动安装 https-dns-proxy ，然后只要它被装上就会立即引发这个问题，启用passwall就出现，百分之百重现，要不大佬你试试？这里是passwall的下载地址：https://t.me/passwallIPKnb

[caliban511]

我自己尝试编译lienol大佬的代码生成固件后，发现并没有这个问题。所以这应该是软件和原生固件之间的兼容性问题了。问题终结。

[aarond10]

It's an odd compatibility issue. 25% I assume is one of four cores so it might have been spinning trying to do something. I'll close for now, but feel free to reopen it if you think it's still a problem.

[aarond10]

On second thought, I'll leave this open for a little longer given the multiple reports. I've not been able to reproduce myself since the commit about a month ago now that forces curl to abort active HTTP2 sessions if the IP address of the resolver changes. I'm wondering if that was part of the problem here.

[Mrnofish]

Isn't there a rule that all submissions should be in English?

If there is, it should be enforced. If not, there should be one.

[caliban511]

Sorry, I don’t understand English. This is using Google Translate to reply. There are some unsatisfactory words. Please forgive me.
This problem of abnormal CPU usage occurred when I installed a software called passwall with the original openwrt firmware.
The source code of this software is here:
https://github.com/xiaorouji/openwrt-package/tree/master/lienol/luci-app-passwall
When I try to use the source code modified by the same person, compile it into openwrt firmware and use this software, the above problem does not appear. The revised firmware source code is here:
https://github.com/Lienol/openwrt
Because I don't understand these codes and only use them, I can't make any constructive comments. It can only be simply considered that this may be a compatibility issue between the original firmware and the software. I can't solve it, I can only avoid it.
If this troubles you, I'm really sorry.
Thank you for your continued work and dedication.

Isn't there a rule that all submissions should be in English?

If there is, it should be enforced. If not, there should be one.

[caliban511]

Sorry, I don’t understand English. This is using Google Translate to reply. There are some unsatisfactory words. Please forgive me.
This problem of abnormal CPU usage occurred when I installed a software called passwall with the original openwrt firmware.
The source code of this software is here:
https://github.com/xiaorouji/openwrt-package/tree/master/lienol/luci-app-passwall
When I try to use the source code modified by the same person, compile it into openwrt firmware and use this software, the above problem does not appear. The revised firmware source code is here:
https://github.com/Lienol/openwrt
Because I don't understand these codes and only use them, I can't make any constructive comments. It can only be simply considered that this may be a compatibility issue between the original firmware and the software. I can't solve it, I can only avoid it.
If this troubles you, I'm really sorry.
Thank you for your continued work and dedication.

On second thought, I'll leave this open for a little longer given the multiple reports. I've not been able to reproduce myself since the commit about a month ago now that forces curl to abort active HTTP2 sessions if the IP address of the resolver changes. I'm wondering if that was part of the problem here.

[obhasqr]

I found that setting up an unreachable doh server will cause high CPU usage.
I can‘t access Google doh server. If I set up Google doh server in https-dns-proxy, it will cause my router's CPU to be occupied by more than 50%.

Add Google doh:

Delete Google doh:

[stangri]

@aarond10 does the comment above shed any light on possible fix for high CPU utilization?

[stangri]

@caliban511 @obhasqr -- can you please test the 2020-08-21-1 version and see if you can still reproduce the problem.

[obhasqr]

Hi @stangri , 2020-08-21-1 can still reproduce this issue.

[nirui]

Ping https://github.com/aarond10/https_dns_proxy/issues/95

[aarond10]

Sorry for the slowness. I just removed the libcurl workaround code. It shouldn't be needed and looks like it was actually the cause of this.