core 🚀 - Home Assistant Memory Leak

Hey there @hunterjm, mind taking a look at this issue as its been labeled with an integration (onvif) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

probot-home-assistant[bot] on 2 Nov 2020

@McGiverGim - can you set log level to debug for ONVIF and include the resulting log here after you notice memory utilization rise again?

hunterjm on 2 Nov 2020

@hunterjm I will do, it seemed with my tests that it was the ONVIF integration, but now I'm not sure. I will confirm it and post the logs...

McGiverGim on 2 Nov 2020

It's possible that it is unrelated to ONVIF. I told this person (who doesn't use ONVIF) to open an issue here, but they opted not to: https://community.home-assistant.io/t/0-117-0-continual-memory-increase/240665/15

Edit: My recommendation would be to remove ONVIF devices and see if the issue still occurs for you. If it doesn't (removing restores memory utilization) then the logs would definitely be helpful. Otherwise, it may be an unrelated bug.

hunterjm on 2 Nov 2020

Several hours after removing ONVIF seems it does not fix the issue, so is clear that in my case the leak is at another place.

I have returned to 0.116.4, if not each 8 hours my system crash by the memory leak. But if I need to do some test I can install version 0.117.x again.

Is better that I rename this issue or that I open another? But without any suspicious log it will be difficult to find.

McGiverGim on 2 Nov 2020

Indeed not sure that the reason of this issue is causing by the onvif integration anymore, because on d'ont use it and i have the same kind of chart about the ram

But not sure where to start to find the issue ...

Gunth on 2 Nov 2020

@Gunth - Easiest way to start is working with @McGiverGim to narrow down a list of integrations you both use, then start disabling them one by one until you find the culprit.

hunterjm on 3 Nov 2020

My memory usage jumped to 41% for CORE after 12 hours, typically just a few percentage points. Rebooted, back to about 2% (8GB RAM on NUC), now at about 5.4% after a few hours. Not using ONVIF.

DanskerUS on 3 Nov 2020

Same issue here - For me it takes about 24 hours to fill up 2 gb of ram, my full swap file and then it starts increasing processor usage to 100% - making the system less and less responsive. I'm also not using ONVIF.

I'm running the supervisor in a esxi container and I still have the snapshot before any upgrade so I can still go back if needed.

EDIT: The integrations/add-ons that I use:

HACS:

tr1plus on 3 Nov 2020

I suppose the most probably is that is one of the included integrations, not any unofficial (HACS).
I don't have any of your HACS integrations.
Seeing your integrations I have the Brother and UPNP (I have more: mobile, CAST, and weather but I suppose this others are used by almost everybody so if there is a problem I suppose it will be massive):

There are others integrations configured using YAML, or that come as default, this is the list of all of them:

McGiverGim on 3 Nov 2020

I am experiencing the issues as well. I have the Brother integration as well. I have deleted it for now to see if there is something there.

llevering on 3 Nov 2020

👍1

I'll take another esxi snapshot and remove brother and UPNP.
Keep you posted in (hopefully) 24h :)

tr1plus on 3 Nov 2020

I have this problem, but no Brother integration. Sorry.

DanskerUS on 3 Nov 2020

i have the same issue. dont use brother or onvif. do have ring cameras which use ffmpeg. below snapshot of addons
downgrading to .116.4 to see if it makes a difference

ramyi on 3 Nov 2020

i have the same issue. dont use brother or onvif. do have ring cameras which use ffmpeg. below snapshot of addons
downgrading to .116.4 to see if it makes a difference

I think it was established that it isn't in the addons - what integrations are you running?

hmmbob on 3 Nov 2020

i have the same issue. dont use brother or onvif. do have ring cameras which use ffmpeg. below snapshot of addons
downgrading to .116.4 to see if it makes a difference

I think it was established that it isn't in the addons - what integrations are you running?

ramyi on 3 Nov 2020

👍1

i have the same issue. dont use brother or onvif. do have ring cameras which use ffmpeg. below snapshot of addons
downgrading to .116.4 to see if it makes a difference

I think it was established that it isn't in the addons - what integrations are you running?

there should be more. how do i get the simple table like McGiverGim

ramyi on 3 Nov 2020

there should be more. how do i get the simple table like McGiverGim

It's in the Configuration, Information.

McGiverGim on 3 Nov 2020

ramyi on 3 Nov 2020

Adding my full integration list to the mix too:

tr1plus on 3 Nov 2020

This is what i know. symptoms are that swap reaches 100% then ram starts to climb to 80-90% but never 100 after any heavy task.
i can replicate with a full snapshot which often results in a reboot after a while.
if i reboot the host after going to supervisor --> system-->reboot under host core then everything goes back to normal for hours until the next heavy task then spike of swap and increase of ram until it causes a reboot of the server not host which then reboots on a whim until you properly reboot the host again to make it stable. downgrading to .116.4 seems to make it more stable but still reboots. i suspect its the supervisor/hassos related.

if i dont reboot the host. after a crash and restart of the home assistant server but ram and swap are still high.

ramyi on 3 Nov 2020

@ramyi In my case 0.116.4 is totally stable. So maybe different issue than yours. This started to happen in 0.117.0 and has not been fixed in 0.117.2.

Is there a way to dump the memory and profile it? Maybe not me, that I use HA OS, that I suppose is more limited, but maybe some of you are able to do that.

McGiverGim on 3 Nov 2020

Are all using supervisor/HA OS?

hmmbob on 3 Nov 2020

I'm, on a raspberry pi4, ha os 5.4 64 bit booting from ssd. I don't know others.

McGiverGim on 3 Nov 2020

I am indeed using the supervisor Version 2020.10.1

What I do notice is that there is a ram difference between the System monitor sensor and the supervisor screen:

tr1plus on 3 Nov 2020

The supervisor only shows the memory used by the Home Assistant docker. The other is the total system if I'm not wrong.

McGiverGim on 3 Nov 2020

im using rpi 3b+ tried with sd card and ssd and issue remains.
i just moved to hyper-v to see if i can have it stable for now.

ramyi on 3 Nov 2020

I use the brother integration also on RPi3 on memory card.. same issue as you all swap to 100%, ram growing up then auto reboot.( can reboot 2,3 times by day ) Revert back to 116.4 is working correctly again ..

Gunth on 3 Nov 2020

I've seen that in 0.117.0 we have a new profiler integration: https://www.home-assistant.io/integrations/profiler/ and four days ago it was added the option for memory profiling: https://github.com/home-assistant/core/pull/42435

I suppose that the memory profiling is for version 0.118.0 but maybe we can add it as a custom_component and it will work? It will help with this issue?

McGiverGim on 3 Nov 2020

I suppose that the memory profiling is for version 0.118.0 but maybe we can add it as a custom_component and it will work? It will help with this issue?

It seems that yes, it can help added as custom_component, here is the post about it in the original ONVIF memory leak issue: https://github.com/home-assistant/core/issues/42390#issuecomment-717532291

I don't know if someone is able to test it. If not, I will try tomorrow, when I have time to let grow the memory for several hours before executing this profiling...

McGiverGim on 3 Nov 2020

The profiling I added currently only profiles a 60 second window. If the leak is slow (MB over hours) it won't help as is. If you want to run it as a custom component, it might be better to comment out lines 101 and 102 in __init__.py to get a full memory snapshot when run.

hunterjm on 3 Nov 2020

👍1

Thanks for the info @hunterjm I will try it tomorrow if nobody does it before. Maybe as suggestion, this can be an option in the data passed to the service

EDIT: I edit because I see in the doc that we can choose the number of seconds, but not the full memory snapshot.

McGiverGim on 3 Nov 2020

Hmm, it might actually be better to just set seconds: 3600 in the service call after HA starts. Then we won't get all the memory consumption from that, just what gets added over an hour.

hunterjm on 3 Nov 2020

For the rest of the group, Integrations are not just on the Integrations page in the UI. Anything you have in configuration.yaml is also relevant.

hunterjm on 3 Nov 2020

But all should be showing if you go to "settings" -> "info" -> "integrations", right?

hmmbob on 3 Nov 2020

@hmmbob - Yes

hunterjm on 3 Nov 2020

Having the same issue, first noticed it after i upgraded to 0.117.1, HA restarted randomly (so it seemed), after 1 day and 4 restarts I went backup to 0.116.4, no problems. Read about the ONVIF integration that could cause a mem leak, so removed that yesterday. This morning upgraded to 0.117.2 again, installed glances and put that data in influxdb/grafana to get a historical picture.
From restarting and installing glances it looks like the mem usage is rising steadily

Schermafbeelding 2020-11-03 183635

I'm running a esxi VM with HA OS, increased the memory from 2 to 3 GB
Schermafbeelding 2020-11-03 184002

Here you can see that at about 9:30 I updated to 0.117.2, the memory usage started increasing, until about 15:00 when I saw that the HA container used about 800Mb + and warnings started showing in glances that mem usage was above 70% (and rising). I stopped the VM and increased the memory to 3 GB and started it again.

Schermafbeelding 2020-11-03 184138

Stimpy68 on 3 Nov 2020

Hmm, it might actually be better to just set seconds: 3600 in the service call after HA starts. Then we won't get all the memory consumption from that, just what gets added over an hour.

True, it will be easy. I will do it in this way.

McGiverGim on 3 Nov 2020

Guys, can you try disabling ssdp component?

elupus on 4 Nov 2020

Sure, but how? Thanks.

DanskerUS on 4 Nov 2020

I have the same issue. I’m using docker version of home assistant in raspberry pi 3b+.
I have enlarged my swap file to 4gb and by what I can see the memory used in the container starts climbing up to 530mb. After that system starts swapping until it gets unresponsive after 30 hours.

Regarding integrations the only thing I have in common is deconz and brother.

cobirnm on 4 Nov 2020

Guys, can you try disabling ssdp component?

i've removd ssdp. will upgrade back to .117.2
should i remove upnp as well?
for others who want to try you can remove ssdp: and make sure you dont have default_config: in there.

ramyi on 4 Nov 2020

    @callback
    def _log_objects(*_):
        _LOGGER.debug("Most common types: %s", objgraph.most_common_types(limit=100))
        _LOGGER.debug("Growth: %s", objgraph.growth(limit=100))


    async_track_time_interval(hass, _log_objects, timedelta(seconds=30))

Maybe make a custom integration that uses objgraph to log the above. Then you can watch Growth over time to see what gets added.

bdraco on 4 Nov 2020

We are almost 24h later (like 21 I think) from my last message where I disabled upnp and brother. My ram has been a bit less aggressive(still climbing but much slower):

tr1plus on 4 Nov 2020

@tr1plus I also used the Brother integration, removed it yesterday in 0.117.2, but no real big difference, downgraded to 0.116.4, you can clearly see when I did that, also noticed that cpu is less choppy in 116.4. So maybe it's upnp?

Knipsel

Stimpy68 on 4 Nov 2020

Do you turn off your Brother printer at night or when you don't use it? If yes, please test this https://github.com/home-assistant/core/issues/42749#issuecomment-721214380

bieniu on 4 Nov 2020

@bieniu Only turn it on when I'm using it, and thats just a few times a week (it's Brother DCP-9020CDW laser printer/scanner). When done, turning it off.
But as I said, removing the Brother integration didn't make a significant difference in the mem usage, it still was climbing.

Stimpy68 on 4 Nov 2020

I have started a 60 minutes memory profiling. I will let you know when finished.
I have done one with 60 seconds to test that it works, but I suppose this will show almost nothing, here is the result (I don't know nothing about python or HA code, so I can't help with this):

And yes, I have it usually powered off Brother too, but I can't test your changes until I finish with the memory dump.

McGiverGim on 4 Nov 2020

We are almost 24h later (like 21 I think) from my last message where I disabled upnp and brother. My ram has been a bit less aggressive(still climbing but much slower):

I can confirm as well. It might be that it is no so much the brother integration as being an underlying library of course. But for me disabling the Brother integration is a work around.

@bieniu I can confirm that my brother is mostly turned off. I will try to use your version, however I am at work now so it will at first just be tonight to test. But it seems that you're definitely on to something :)

llevering on 4 Nov 2020

@llevering Probably brother integration causes memory leaking when device is turned off and restarting HA. If this scenario fits your usage, this test version may solve the problem.

bieniu on 4 Nov 2020

Upgraded again to 0.117.2 and disabled upnp (ssdp & zeroconf), Brother integration was removed yesterday.
Will keep an eye on memory usage in the coming hours.

Stimpy68 on 4 Nov 2020

Try creating a snapshot. It normally happens after that for me.

On Wed, 4 Nov 2020 at 7:12 pm, Stimpy68 notifications@github.com wrote:

Upgraded again to 0.117.2 and disabled upnp (ssdp & zeroconf), Brother
integration was removed yesterday.
Will keep an eye on memory usage in the coming hours.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/core/issues/42752#issuecomment-721579553,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AITMFIZB4FB37WRJSM4CHB3SOEEIRANCNFSM4THOP75Q
.

ramyi on 4 Nov 2020

@ramyi Did that now, now we wait ;-)
Core RAM usage stayed the same, Supervisor RAM usage increased a lot, but doesn't seem to be going down.... stays around 32% usage (3GB memory allocated), Core uses 9% at this moment.

But creating a snapshot seems to create errors:

20-11-04 08:29:14 INFO (MainThread) [supervisor.snapshots] Creating new full-snapshot with slug 2ba11093
20-11-04 08:29:14 INFO (MainThread) [supervisor.snapshots] Snapshotting 2ba11093 store Add-ons
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on a0d7b954_nodered
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon a0d7b954_nodered
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on core_configurator
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon core_configurator
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on core_ssh
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon core_ssh
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on core_samba
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon core_samba
20-11-04 08:29:14 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on a0d7b954_influxdb
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon a0d7b954_influxdb
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on a0d7b954_grafana
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon a0d7b954_grafana
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on 7ad98f9c_zigbee2mqtt
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon 7ad98f9c_zigbee2mqtt
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on core_mosquitto
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon core_mosquitto
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Building snapshot for add-on a0d7b954_glances
20-11-04 08:29:28 INFO (MainThread) [supervisor.addons.addon] Finish snapshot for addon a0d7b954_glances
20-11-04 08:29:28 INFO (MainThread) [supervisor.snapshots] Snapshotting 2ba11093 store folders
20-11-04 08:29:28 INFO (SyncWorker_2) [supervisor.snapshots.snapshot] Snapshot folder ssl
20-11-04 08:29:28 INFO (SyncWorker_2) [supervisor.snapshots.snapshot] Snapshot folder ssl done
20-11-04 08:29:28 INFO (SyncWorker_3) [supervisor.snapshots.snapshot] Snapshot folder homeassistant
20-11-04 08:30:14 ERROR (MainThread) [supervisor.snapshots] A snapshot/restore process is already running
20-11-04 08:30:47 INFO (SyncWorker_3) [supervisor.snapshots.snapshot] Snapshot folder homeassistant done
20-11-04 08:30:47 INFO (SyncWorker_2) [supervisor.snapshots.snapshot] Snapshot folder media
20-11-04 08:30:47 INFO (SyncWorker_2) [supervisor.snapshots.snapshot] Snapshot folder media done
20-11-04 08:30:47 INFO (SyncWorker_4) [supervisor.snapshots.snapshot] Snapshot folder share
20-11-04 08:30:48 INFO (SyncWorker_4) [supervisor.snapshots.snapshot] Snapshot folder share done
20-11-04 08:30:48 INFO (SyncWorker_1) [supervisor.snapshots.snapshot] Snapshot folder addons/local
20-11-04 08:30:48 INFO (SyncWorker_1) [supervisor.snapshots.snapshot] Snapshot folder addons/local done
20-11-04 08:30:48 INFO (MainThread) [supervisor.snapshots] Crating full-snapshot with slug 2ba11093 completed

Stimpy68 on 4 Nov 2020

Dump of 60 minute profiling ready, here it is:
heap_profile.1604474975861335.zip

Is clear that the memory had the leak during the test:

I don't know if some of the experts can look at it to see if it explains the problem @hunterjm @bdraco are you interested?

~~I'm trying to open it with the browser, but is slow, I will update this post later if I can get an image with it.~~
EDIT: This is the image produced by the Browser:

Now, I will test the Brother test version provided by @bieniu

McGiverGim on 4 Nov 2020

1 hour passed, and no noticeable increase in RAM usage after disabling upnp

Knipsel2

Just after 9:00 (the dip) disabled upnp and upgraded from 116.4 to 117.2

Stimpy68 on 4 Nov 2020

Creating snapshot bumps RAM usage from about 5% to 41%. Creating a second snapshot does not further increase RAM usage. Reboot restores RAM back to about 5%.

Running on a NUC w 8GB RAM.

DanskerUS on 4 Nov 2020

Maybe is too soon to comment, but with the updated test version of Brother by @bieniu my system seems stable in memory after two hours.

I have a slightly more memory usage, I don't know if this will continue growing in the next hours, seems stable. I will keep an eye on it. I have noticed too that in 0.117.2 I have more CPU usage (about 6%, with 0.116.4 it was about 3%).

Others have commented that after removing Brother the problem remains, so maybe there are two different memory leaks?

It is interesting to decide if the memory dump that I did points to Brother or not. If not maybe the tool used for this memory dump is not the correct one. We need one that can help developers to know where the problem is. Maybe the pysnmp OidOrderedDict is used by Brother? I think remember it uses SNMP and OIDs.

McGiverGim on 4 Nov 2020

@McGiverGim I removed the brother integration and the mem leak remained, I disabled upnp and it seems that this solved it (for now), memory usage is rising but MUCH more slowly than when upnp was enabled. But it has just been 2 hours now, so will get back after 24 hours.

Stimpy68 on 4 Nov 2020

@McGiverGim Memory leakage in Brother integration can only occur when restarting the HA when the printer is turned off/offline. Is this your case?

bieniu on 4 Nov 2020

@McGiverGim Memory leakage in Brother integration can only occur when restarting the HA when the printer is turned off/offline. Is this the case for you?

Yes, it is. I have usually it powered off, and I restarted at each update, so it seems my case.

McGiverGim on 4 Nov 2020

@McGiverGim Could you do one more test for me? Save __init__.py file from this gist https://gist.github.com/bieniu/096bd2e50b1befb6b8c6bea6616a8ba6 to /config/custom_components/brother, turn off the printer, restart HA and watch the memory usage.

bieniu on 4 Nov 2020

@McGiverGim Could you do one more test for me? Save __init__.py file from this gist https://gist.github.com/bieniu/096bd2e50b1befb6b8c6bea6616a8ba6 to /config/custom_components/brother, turn off the printer, restart HA and watch the memory usage.

Of course, I will do it and report in one or two hours...

McGiverGim on 4 Nov 2020

👍1

~~I feel it's either UPNP or Brother (my brother printer is always off when starting home assistant - I only turn it on when I need to print which is not often)~~

EDIT: Maybe not - I think my RAM started swapping - appologies

tr1plus on 4 Nov 2020

@McGiverGim Could you do one more test for me? Save init.py file from this gist https://gist.github.com/bieniu/096bd2e50b1befb6b8c6bea6616a8ba6 to /config/custom_components/brother, turn off the printer, restart HA and watch the memory usage.

@bieniu One thing more, only to be sure... do you want that I replace the file from your test version or that I remove your test version and I put this file there, all alone?

McGiverGim on 4 Nov 2020

@McGiverGim Please only replace __init__.py file.

bieniu on 4 Nov 2020

👍1

@bieniu with the gist init file, it seems the leak is here again:

If I'm not wrong this is the only difference between both init files:

McGiverGim on 4 Nov 2020

Yes, that's the only difference. Thanks for help with testing.

bieniu on 4 Nov 2020

👍1

I merged the fix for Brother and tagged it for 0.117.4 - Will keep this open since there are some reports in here of an issue without the Brother integration enabled.

hunterjm on 4 Nov 2020

🎉2 👍1

I also see the memory leak since 0.117.x but I don't have the Brother integration, see below for my integrations.
Will try some things in this thread.

rickkdotnet on 4 Nov 2020

Updated to 117.3. Making a snapshot still cause Supervisor RAM Usage to jump to 40.6 percent. Never had ONVIF or Brother integration in use here.

DanskerUS on 4 Nov 2020

Suggest to install 0.117.4, as it contains the brother fix:

https://github.com/home-assistant/core/releases/tag/0.117.4

hmmbob on 4 Nov 2020

Will do as soon as it become available, thank you.

DanskerUS on 4 Nov 2020

Updated to 117.4.

Same: making a snapshot makes Supervisor RAM Usage jump to 41.4%.

DanskerUS on 4 Nov 2020

Rebooting drops it back to 1.5%

DanskerUS on 4 Nov 2020

@DanskerUS snapshot memory usage doesn't seem related to this issue with leaking over time.

elupus on 4 Nov 2020

@elupus OK, should I open a separate ticket?

DanskerUS on 4 Nov 2020

👍1

1.5% after a rebooting is not indicative.
Reboot the system and run it without snapshotting it for a few days, the memory usage after a few days is going to be your baseline, e.g. 51% Take a snapshot. Does it jump to 92% and stays there? if yes, then open another ticket.

Adminiuga on 4 Nov 2020

After running some hours the baseline is around 5%. Soooo, should I turn off the nightly backup...?

Is it normal to see such a bump in memory use from a snapshot, did not notice this under 116.x or earlier?

DanskerUS on 4 Nov 2020

If you are taking snapshots at night as the backup, then no need to turn it off. Just take a screenshot of memory usage before and after the snapshot. Keep it till next night/next snapshot and take a screenshot before/after the 2nd night snapshot.
Open the ticket, post the details.

Adminiuga on 4 Nov 2020

24 hours later and upgrading to 0.117.3, some hours later to 0.117.4 (the dips in the chart) RAM usage stayed stable (upnp disabled)

Knipsel

Will enable upnp now and will keep an eye on RAM usage

Stimpy68 on 5 Nov 2020

1 hour after enabling upnp, memory usage seems stable (running 0.117.4)

Knipsel2

Turned on my Brother printer and will wait for HA to discover it, will add that.

Stimpy68 on 5 Nov 2020

Memory usage still stable after enabling UPNP again and installing the Brother integration. So issue seems te be fixed in 0.117.4
Maybe more people can verify this?

Stimpy68 on 5 Nov 2020

Have a PI3 running Hassos and while I'm not seeing the large mem increase that some see I do see a slight increase.. also running Adguard add-on. after 117 upgrade things become so slow i have to reboot to get speed back.. recently I disabled adguard and went back to router for DNS cause things are so slow. moved back to 116.2 and speed seems ok.. not getting the slowness I had on 117. For a while, I thought it was just adguard but I think its hass

sz0wxc on 5 Nov 2020

I have the exact same problem on 0.117.4 Memory slowly fills up and swap also fills up eventually grinding everything to a dead stop. Reverting back to 0.116.4 seems like a temporary solution.

thijsdejong on 5 Nov 2020

@sz0wxc and @thijsdejong when you say 0.117, did you try the latest release (currently 0.117.5)?

hmmbob on 5 Nov 2020

0.117.4 doesn't fix it for me, but that's expected as I don't use the Brother integration. I don't see any relevant fixes in 0.117.5.
It looks like everyone in this thread uses the esphome integration, so maybe that's one to look into..

rickkdotnet on 5 Nov 2020

@sz0wxc and @thijsdejong when you say 0.117, did you try the latest release (currently 0.117.5)?

0.117.4. But I don't see any change related to this in 0.117.5 vs .4 correct?

thijsdejong on 5 Nov 2020

👍1

0.117.4 doesn't fix it for me, but that's expected as I don't use the Brother integration. I don't see any relevant fixes in 0.117.5.
It looks like everyone in this thread uses the esphome integration, so maybe that's one to look into..

I don't use the esphome integration

thijsdejong on 5 Nov 2020

@rickkdotnet - I use ESP Home and am not seeing the same issues. There were also no changes to ESP Home in 0.117 other than translation string updates, so it is very unlikely to be the culprit.

hunterjm on 5 Nov 2020

I was hoping it's ONVIF, but after trying 0.117.4 yesterday, my automations again did not work this morning.
Here is the list of my integrations.

EricReiche on 6 Nov 2020

@sz0wxc and @thijsdejong when you say 0.117, did you try the latest release (currently 0.117.5)?

I had tried 117.4 but have not tried 117.5.
As for addons I only have Adguard, Duck DNS, Terminal & SSH, and Chrony.

sz0wxc on 6 Nov 2020

As for integrations I only have Adguard, Duck DNS, Terminal & SSH, and Chrony

This are addons, not integrations. Check Configuration -> Integrations.

McGiverGim on 6 Nov 2020

I also still have the memory issue, less then before but continue to reboot 1 / 24h ...

Here is my list of integrations

Gunth on 6 Nov 2020

Another HA with memory leak problems. HA will crash every 1-6 hours.

Home Assistant: 0.117.4
Supervisor: 2020.11.0
HassOS 4.15

JBS5 on 7 Nov 2020

Has anyone used objgraph to find the leak yet? I could probably put together a small leak detector integration if we are really stuck on this.

bdraco on 7 Nov 2020

To be honest I have not idea ho to use objgraph :-(

By the way, I tried with 0.117.5 but same issue

Gunth on 7 Nov 2020

Has anyone used objgraph to find the leak yet? I could probably put together a small leak detector integration if we are really stuck on this.

It will be great to have tools like this in the profiler integration. The added memory profiler is not enough?

McGiverGim on 7 Nov 2020

Has anyone used objgraph to find the leak yet? I could probably put together a small leak detector integration if we are really stuck on this.

It will be great to have tools like this in the profiler integration. The added memory profiler is not enough?

The memory profiler is great, but it doesn't provide the ability to dump objects in real time

bdraco on 7 Nov 2020

I've updated the profiler (https://github.com/bdraco/profiler) with two new services to watch object growth over time and dump interesting objects. I've also opened https://github.com/home-assistant/core/pull/42951 to merge the changes into core.

Example usage:

Screen Shot 2020-11-07 at 11 55 10 AM

Screen Shot 2020-11-07 at 11 55 22 AM

bdraco on 7 Nov 2020

I'm wondering if the issue really is within HA core, or maybe in the OS or Supervisor. As far as I can tell, all users seeing this issue are using a supervised setup in one shape or form. The few users I know using docker aren't seeing this issue (incl myself), nor did I see any venv user in this thread.

hmmbob on 8 Nov 2020

👍1

The supervisor shows the core and supervisor memory usage with two different bars. Easy to check.

McGiverGim on 8 Nov 2020

The supervisor shows the core and supervisor memory usage with two different bars. Easy to check.

Not my system, but that doesn't matter:

JBS5 on 8 Nov 2020

I've just installed the profiler, just waiting for the memory leak now.

Question how can i read the cprof file generated ?

Gunth on 8 Nov 2020

I've just installed the profiler, just waiting for the memory leak now.

Question how can i read the cprof file generated ?

https://www.home-assistant.io/integrations/profiler/

Also if you send me a copy of the callgrind.out.XXX file, I'm happy to analyze it -> [email protected]

bdraco on 8 Nov 2020

Here is the file, i don't know if you will find something relevant because I had to restart for the installation of the profiler but i'm already to the limit of the ram ( 83% ) and swap (100%).
profile.1604860760839893.zip

I try to analyse it but i'm a little bit lost on what i have to find :-)

I will try to make a new profiler later to be more closer to the crash ..

Gunth on 8 Nov 2020

@Gunth

It looks like you have an integration that is trying to setup but failing and may leak ram each time it fails.

Which integrations do you have logging about not being ready yet and will try again later in the log?

Screen Shot 2020-11-08 at 9 26 20 AM

bdraco on 8 Nov 2020

On a side note, snmp is also using quite a bit of cpu time because its recreating the engine each request. Should be able to rework it with https://stackoverflow.com/questions/34654132/reuse-getcmd-object-in-pysnmp

Edit: It looks like it is reusing the object so there might be a change to pysnmp needed to improve performance since its reloading the Mibs every time under the hood which seems unlikely since the project appears to be dormant upstream https://github.com/etingof/pysnmp/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc

Screen Shot 2020-11-08 at 9 33 42 AM

bdraco on 8 Nov 2020

Yes i have one, the brother integration which is related to the SNMP library which is strange because the issue has been resolved and I'm in the latest release.

I will disable it and check if it is better ...

Thanks for the investigations

Gunth on 8 Nov 2020

After running a few hours (NUC 117.5, w/8GB RAM):

Capture

After running a snapshot (automated or manual):

Capture2

Additional snapshots does not increase memory usage further. A reboot restores usage to first image.
I have a couple of additional systems running, prior to 117 I do not see this behavior.

DanskerUS on 9 Nov 2020

Each of my snapshots are about 32MB in size, -how can that consume about 3.2GB of memory...?
That is: 40% of 8GB of RAM.

DanskerUS on 10 Nov 2020

I can't reproduce on Pi3B. After a snapshot it uses about 12% of RAM and after next snapshot it uses 2% more, so under 15% of 2GB

Adminiuga on 10 Nov 2020

@Adminiuga Thank you for checking. I'm running HassOS 4.15, OS 5.4.72, supervisor 2020.11.0.

I have another system running under VirtualBox, same versions, no issue.

Not sure where to go with this?

DanskerUS on 10 Nov 2020

i think its more the swap space rather than ram. while i only have 1gb ram on my rpi3b+ it doesnt go above 85-90% but swap reaches 100% and after a while it reboots.
ive gone back go to core version 0.116 and its better but still overall unstable.
on supervisor 2020.11.0 and hassos 4.15

added uptime sensor and its rebooting way more than i expected.
my hunch this is either due to hassos or supervisor not core.
i dont know how to tell whats using all that swap space all of a sudden

ramyi on 10 Nov 2020

@Adminiuga Thank you for checking. I'm running HassOS 4.15, OS 5.4.72, supervisor 2020.11.0.

I have another system running under VirtualBox, same versions, no issue.

Not sure where to go with this?

You don't have a memory leak, your memory is just being used during a task where memory is needed. Memory will eventually be cleared when needed - remember that memory is designed to be used, not to be empty. See
https://www.linuxatemyram.com/

As long as you don't have similar issues (unexpected reboots) and your used ram graph doesn't top out at 100%, you do not have a memory leak and you are not affected by the same issue as the others here.

hmmbob on 10 Nov 2020

I'll stand down with my concerns.

Will remain a bit baffled by this being a new issue for me on only some installations, and the 100x magnitude of the usage.

DanskerUS on 10 Nov 2020

I have a raspberry pi 4 with 1gb ram. HA creates a +- 230MB swap file and once it is full (on 117.5) HA crashes or becomes extremely slow.

I reverted to 0.116.4 and there are no problems whatsoever.

basdejong95 on 10 Nov 2020

How large are y'all's log files when swap/ram are filling up? Im wondering if there is some severe logging going on in the background.

hmmbob on 10 Nov 2020

OZW.log is smallish. Where are the rest located?

DanskerUS on 11 Nov 2020

I’ve had the longest uptime so far. Log file is 4.6mb abs db is 280 mb. Sometimes the reboot happen within 5 mins or if I do a an intensive task like look at the camera dashboard which will replay 4 ring video feeds.

ramyi on 11 Nov 2020

Any update on this? I'm sticking with 0.116.4 for now. Anything higher constantly crashes HA.

basdejong95 on 15 Nov 2020

I have disabled some add-ons on HA environments with the memory leak probem and now the memory increasing is much less. Before a reboot was required every 1-4 hours, now once a day or even not. Pretty steady at 50-60% memory use now.

Disabled add-ons: ESPHome, File editor, Grafana, Google Drive Back-up, InfluxDB, Log Viewer, Mosquitto broker, UniFi Controller, SSH & Web Terminal.

The user is not started yet with enabling the add-ons one by one to see when the memor increasing starts again.

JBS5 on 16 Nov 2020

the new hassos 4.16 reduced this a little but still rebooting every little while.

disabled my ring cameras and it seems that its stable.
i was using a picture entity card with stream:live option for 4 cameras. never been an issue until recently.

ramyi on 17 Nov 2020

None of the more recent charts look like leaking memory. An increase in memory utilization perhaps, but not the steadily growing spikes we saw initially with this ticket.

With how large this is getting, it is really hard to see the signal through the noise. Perhaps a new issue can be created regarding issues with swap and/or general memory utilization?

hunterjm on 17 Nov 2020

Closing now in favor of creating new individual issues once the integrations causing issues can be isolated. This original ticket is resolved with #42833.

hunterjm on 17 Nov 2020

Core: Home Assistant Memory Leak

The problem

Environment

Problem-relevant `configuration.yaml`

Traceback/Error logs

Additional information

Most helpful comment

All 124 comments

Related issues

Core: Home Assistant Memory Leak

The problem

Environment

Problem-relevant configuration.yaml

Traceback/Error logs

Additional information

Most helpful comment

All 124 comments

Related issues

Problem-relevant `configuration.yaml`