Core: Memory issue

[CCOSTAN]

Home Assistant release (hass --version):
0.38.4

Python release (python3 --version):
3.4.2

Component/platform:
HASS / RPI3.

Description of problem:
Getting memory issue when trying to restart. This might be a local issue on my pi but just popped up with the upgrade to 38.4

Traceback (if applicable):

Feb 22 14:48:57 Carlo-Pi hass[629]: Traceback (most recent call last):
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/tasks.py", line 237, in _step
Feb 22 14:48:57 Carlo-Pi hass[629]: result = next(coro)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/core.py", line 10
18, in _event_to_service_call
Feb 22 14:48:57 Carlo-Pi hass[629]: yield from service_handler.func(service_call)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/__init
__.py", line 159, in async_handle_core_service
Feb 22 14:48:57 Carlo-Pi hass[629]: yield from conf_util.async_check_ha_config_file(hass)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/config.py", line
471, in async_check_ha_config_file
Feb 22 14:48:57 Carlo-Pi hass[629]: stdout=asyncio.subprocess.PIPE, loop=hass.loop)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/subprocess.py", line 226, in create_subprocess_exec

Feb 22 14:48:57 Carlo-Pi hass[629]: stderr=stderr, **kwds)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/base_events.py", line 866, in subprocess_exec
Feb 22 14:48:57 Carlo-Pi hass[629]: bufsize, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/unix_events.py", line 173, in _make_subprocess_trans
port
Feb 22 14:48:57 Carlo-Pi hass[629]: extra=extra, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/base_subprocess.py", line 31, in __init__
Feb 22 14:48:57 Carlo-Pi hass[629]: stderr=stderr, bufsize=bufsize, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/asyncio/unix_events.py", line 559, in _start
Feb 22 14:48:57 Carlo-Pi hass[629]: universal_newlines=False, bufsize=bufsize, **kwargs)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
Feb 22 14:48:57 Carlo-Pi hass[629]: restore_signals, start_new_session)
Feb 22 14:48:57 Carlo-Pi hass[629]: File "/usr/lib/python3.4/subprocess.py", line 1395, in _execute_child
Feb 22 14:48:57 Carlo-Pi hass[629]: restore_signals, start_new_session, preexec_fn)
Feb 22 14:48:57 Carlo-Pi hass[629]: OSError: [Errno 12] Cannot allocate memory

[CCOSTAN]

I think these issues might have been localized. unable to reproduce on 0.39.0.dev

[CCOSTAN]

HASS seems to be taking up 90% of my Pis memory now. :(

[pvizeli]

Please add a list of components and platform. On my raspbeery I have use only 20%

[grivers3]

Same Issue. After shutting components off one by one, I found the emulated hue is my culprit. Also, there is no history when my emulated hue is active.
emulated_hue:
type: google_home
listen_port: 80
off_maps_to_on_domains:
- script
- scene
expose_by_default: false

/w Emulated Hue (Taken at 1 minute intervals)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24822 homeass+ 20 0 424524 95692 11476 S 99.1 12.7 1:31.66 hass
24822 homeass+ 20 0 438716 109888 11476 S 101.7 14.6 2:30.35 hass
24822 homeass+ 20 0 450224 121780 11476 S 99.7 16.2 3:29.69 hass
24822 homeass+ 20 0 460648 130824 11476 S 99.4 17.4 4:30.18 hass

w/o Emulated Hue (Taken at 1 minute intervals)
24506 homeass+ 20 0 282328 59596 11568 S 7.0 7.9 0:35.59 hass
24506 homeass+ 20 0 282328 59720 11568 S 2.3 7.9 0:41.68 hass
24506 homeass+ 20 0 389708 60708 11568 S 27.7 8.1 0:45.75 hass
24506 homeass+ 20 0 283212 60800 11568 S 1.0 8.1 0:48.51 hass

[grivers3]

History issue might be browser cache. Take with a grain of salt..

[CCOSTAN]

I rolled back to 0.38.3 and the issue went away. been running fine for a few hours now.

The issue for me seems to have been 0.38.4

[Cinntax]

I have also seen this issue, and tracked it down to the emulated_hue component. The ram usage just seems to grow and grow. I'm running on a normal server, so it took a while, but eventually ate up the RAM on my server + swap.

I'm not quite sure where to go to provide more detail- I turned logging up to debug and didn't see anything terribly useful in the logs while this was all happening.

[pvizeli]

Do you have aiohttp 1.3.3 running? I think that is the Memory issue they fix with that. Mybe you need to remove the dept folder with inlcude a older version?

[grivers3]

Deleted all aiohttp entries in deps and restarted. aiohttp-1.3.3.dist-info added to deps. Still having issue with the emulated hue.

[ctaloi]

Confirming memory issue here since upgrading to 38.4

[pvizeli]

Okay from 0.38.3 to 0.38.4 have no change on aiohttp. Can you disable discovery? That have change the library

[ctaloi]

Removed discovery component and seems stable, proc and memory usage are normal again.

---

Christopher Aloi
Sent from my iPhone

On Feb 23, 2017, at 5:05 PM, Pascal Vizeli [e-mail] wrote:

Okay from 0.38.3 to 0.38.4 have no change on aiohttp. Can you disable discovery? That have change the library

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[Cinntax]

I agree- disabling discovery seemed to clear up the problem.

Also note that I tried to re-enable discovery, while at the time disabling my normal hue light component- thinking perhaps the issue was being caused by the hue component discovering the emulated_hue component. That did not make any difference - memory/CPU was still high.

[iambdud]

I'm seeing this issue in 0.38.3... and I already had the discovery component disabled... now that i think about it... i think the machine started eating memory when i told alexa to activate a scene via emulated_hue.

[deftdawg]

Also seeing this in 0.38.4

This starts to appear in my logs after a while (like 2 hours)
hass[983]: OSError: [Errno 12] Cannot allocate memory

hass is using more than 12GB (of 16GB) of ram

[iambdud]

So... I added back the "type" for my emulated_hue component and everything seems back to normal...

edit: I spoke too soon... it ran out of memory again.

[jer78]

Also have the memory issue since 0.38.4. Reverted back to 0.38.3 and everything is fine again.

[deftdawg]

I set this up to recycle 0.38.4

#!/usr/bin/env bash

# Usage sudo ./ha-watchdog.sh
while [ true ]; do
  sudo service home-assistant@${SUDO_USER} restart
  echo "Watchdog restarted Home Assistant at $(date)"
  journalctl -f -u home-assistant@${SUDO_USER} | grep "Cannot allocate memory" | head -1
done

sudo ./ha-watchdog.sh Watchdog restarted Home Assistant at Fri Feb 24 04:58:02 EST 2017 Feb 24 06:26:01 Chromebox hass[19432]: OSError: [Errno 12] Cannot allocate memory Watchdog restarted Home Assistant at Fri Feb 24 08:09:18 EST 2017 Feb 24 09:35:37 Chromebox hass[2903]: OSError: [Errno 12] Cannot allocate memory Watchdog restarted Home Assistant at Fri Feb 24 10:11:26 EST 2017 Feb 24 11:38:19 Chromebox hass[1462]: OSError: [Errno 12] Cannot allocate memory Watchdog restarted Home Assistant at Fri Feb 24 12:13:53 EST 2017

Unfortunately, it doesn't seem to work 100%, I just had to manually cycle it a little while ago because the same message was appearing again.

I briefly looked at emulated_hue component tagged in 0.38.1 to 0.38.4, doesn't seem like anything changed between point releases (last update was 17 days ago), wonder what is causing this.

[acarlo79]

Same issue here after the upgrade to 0.38.4, the only other action I performed was to chnage the emulated hue component from alexa to google_home.
Not sure yet if this is the culprit here:

17-02-25 00:00:10 ERROR (Thread-6) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/updater.py", line 83, in <lambda>
    hass, lambda _: check_newest_version(hass, huuid),
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/updater.py", line 91, in check_newest_version
    result = get_newest_version(huuid)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/updater.py", line 133, in get_newest_version
    import distro
  File "/home/hass/.homeassistant/deps/distro.py", line 1051, in <module>
    _distro = LinuxDistribution()
  File "/home/hass/.homeassistant/deps/distro.py", line 595, in __init__
    if include_lsb else {}
  File "/home/hass/.homeassistant/deps/distro.py", line 921, in _get_lsb_release_info
    stderr=subprocess.PIPE)
  File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1395, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

[deftdawg]

Seems to also be happening after update to 0.39.0

Feb 26 01:07:07 Chromebox hass[15729]: OSError: [Errno 12] Cannot allocate memory
Watchdog restarted Home Assistant at Sun Feb 26 01:17:14 EST 2017

[pvizeli]

To disable the updater does also not help? Only the discover (netdisco)?

[patrickeasters]

I am also still seeing this in 0.39.0. I'll be rolling back to 0.38.3 again.

[laf]

Also seeing the same issue, disabling discovery or emulated_hue stabilises things again.

Happy to provide remote access to this VM to one of the core devs if it's helpful.

[danodemano]

I was having the same issues on 0.39.0. DIsabling the "discovery" component seems to have resolved the issue.

[acockburn]

For the record I saw this on 0.38.4 and downgrading to 0.38.3 fixed it. I have not tried 0.39 yet.

I can't conveniently disable discovery because I have about 20 ZWave devices.

[danodemano]

@acockburn FWIW I don't believe discovery is required for z-wave. Mine are working just fine with it disabled (20+ devices) and it's not listed on the doc page: https://home-assistant.io/components/discovery/

[iambdud]

I must have something else going on... because I have discovery disabled and I still have this issue. But... i have this issue even if I go back to 0.38.0. I'm running HASS in a docker container... perhaps i'm doing something wrong...

[acockburn]

@danodeman good to know, thanks! I'll track this and may give it a try at the weekend if there isn;t a fix yet.

[markrennie71]

I got this memory problem in 0.39.0 and 0.39.1, back to 0.38.3 and stable again.

[vladm]

I'm on 39.1 under docker;
Commented out discovery, emulated_hue still seeing high CPU + 1.4gb ram usage.

[vladm]

okay, I've tracked down my high-cpu problem to using two hikvision binary_sensors defined as the following:

binary_sensor:
- platform: hikvision
  host: xx.xx.xx.10
  port: 80
  ssl: False
  username: aaa
  password: bbb

- platform: hikvision
  host: xx.xx.xx.20
  port: 80
  ssl: False
  username: aaa
  password: bbb

with both sensors defined in configuration, python process constantly takes 70% CPU. As long as I comment out the second sensor, everything goes back to normal.

@mezz64

[mezz64]

Interesting. I also have two cameras setup with hikvision binary with a docker install and don't have any high CPU or memory problems. I don't use emulated_hue though. Can you replicate this with a bare HASS instance and just the hikvision sensors defined?

[vladm]

@mezz64 - confirmed. Removed everything, leaving only sections "homeassistant" with basic settings, "frontend", "http" and "binary_sensor" defined as above.

Should I create a new issue as it seems a separate problem?

This is 0.39.1 official docker image.

[deftdawg]

Kind of hard for me to see what changed between 38.3 and 38.4+ on github, so today I fired up sourcetree and pull down the repo...

It looks like the only thing that changed was bumping netdisco from 0.8.2 to 0.8.3 and a bunch of UI changes in js/html files...

Looking at netdisco repo, support was added for Philips hue upnp and some other stuff in that release. My guess is whatever that code is doing it is causing crap ton of cpu / memory consumption...

To test this theory, I rolled back netdisco by doing the following:

cd ~/.homeassistant/deps
mv netdisco ../netdisco.0.8.3  # get it out of the deps directory
cp -r ../deps.bkp/netdisco .  # copy in previous version of the dep
# restart hass

So far my 2x CPUs have dropped from both hovering around 80% cpu to around 20% each. And memory seems to be stable at couple hundred MBs. I'll continue to watch it... but if others are experiencing this you may want to try rolling back netdisco too.

[mezz64]

@vladm You can create a seperate issue if you wish. I'm fairly certain something else is going here though. Was your "deps" folder empty? I was running on dev, but switched to the latest release container just now to verify from my end an am still not seeing anything significant... 5% cpu and about 150MB of memory being used. Anyone else with memory/cpu issues using hikvision binary sensors?

[tomusher]

@jumpkick I've submitted a PR to the netdisco repo which at least resolves the issue with the discovery component (https://github.com/home-assistant/netdisco/issues/99)

[deftdawg]

@mezz64 -- if you deps/netdisco folder has philips_hue_nupnp.py in it then it's 0.8.3, if not you have 0.8.2 or earlier and you won't see the issue.

@tomusher cool, I just put a patch to roll it back... Doesn't matter to me which way we go as long as this bug gets crushed out of my system 😄

[mezz64]

@jumpkick I did/do not personally experience the issue with 0.8.3 netdisco. I was just trying to figure out if there is something going wrong in my hikvision component that was mentioned by @vladm as a possible source of the problem. I can't replicate on my end so it's hard to troubleshoot.

[deftdawg]

@mezz64 @vladm hikvision did not change between 0.38.3 and 0.38.4, so it's not related to this issue, should spin up a different issue for that.

[kipwittchen]

Rolling back netdisco to 0.8.2 seems to have fixed my mqtt/memory leak issue! I'll check the log/see if mysensors are still reporting properly in the morning.

[balloob]

I have a proper Netdisco fix here: https://github.com/home-assistant/home-assistant/pull/6321

Would love to get some people testing this.

[iambdud]

Would love to test because this issue is driving me bonkers... but fairly new to this. I'm using docker. How can I help? Should i just pull the dev image in docker?

[jnewland]

Would love to get some people testing this.

Memory usage is low and stable with #6321 deployed. 👌

[kipwittchen]

I ran this command to install netdisco 0.9.0:
pip install netdisco==0.9.0 --target ~/.homeassistant/deps

So far, so good!