Core: Memory usage increase and Updating ping binary_sensoe error

[pierinho]

Home Assistant release with the issue:
0.95.4

Last working Home Assistant release (if known):
0.94.4

Operating environment (Hass.io/Docker/Windows/etc.):
Hass.io

Component/platform:

Description of problem:
After upgrade from 0.94.4 to 0.95.4 memory usage increase from 26% to 40-41% and have this error: Updating ping binary_sensor took longer than the scheduled update interval 0:00:30.
I have most ping.bynary.sensor, and all stop working. In 0.94.4 works perfectly.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:

[aegjoyce]

I am having the same issue. HassOS, rPI 3B+, 0.95.4

Not seeing quite the same problems with CPU usage as above but after a few hours my ping binary sensors stop working with the same error - took longer that scheduled update interval.

I have tried converting to command line binary sensors instead but this yields the same issue after working initially:

binary_sensor:
  - platform: command_line
    name: AV Receiver
    command: ping -W 1 -c 1 192.168.1.147 > /dev/null 2>&1 && echo success || echo fail
    scan_interval: 10
    command_timeout: 10
    payload_on: "success"
    payload_off: "fail"

Therefore I believe the issue is with the way Hassio handles any ping request, and not necessarily the specific ping components.

[darkkatarsis]

Same here, after few hours sensor stop working and status freezee

[aegjoyce]

Still can't make this work.

Have tried setting up ping sensing as binary_sensor.ping, binary_sensor.command_line and sensor.command_line but none of them work. In spite of setting scan_interval to 60 seconds and only having a count of 1, and with only two ping sensors (which should hardly be taxing from a networking POV), it still borks out with the same error after a few hours post-restart:

Updating command_line sensor took longer than the scheduled update interval 0:01:00
_6:07 PM helpers/entity_platform.py (WARNING) - message first occured at August 6, 2019, 9:21 PM and shows up 1227 times_

The really frustrating thing is not being able to diagnose why - even with logger for those specific entities set to debug, I am not getting any useful information back to help fix this.

My overall impression is that for some reason or another (most likely temporary network connection fluctuations) the ping component is unable to complete a request, but instead of timing-out graciously, it gets stuck.

I really hope he doesn't mind me tagging him in this (I know it's generally poor form, I'm really sorry) but @thomasloven - I know you did some work on reformulating the ping component a while back, can you suggest anything I can do to help further diagnose the problem? I just can't seem to glean any useful logs to get me in the right direction. Would it be worth doing some sort of packet-sniffing exercise to see if, after the errors start, the component is still actually sending out pings, or whether the first error then crashes it and stops it from sending further pings?

[aegjoyce]

More testing, more thoughts.

First, I set up a new sensor to ping my Mac and ran a tcpdump to detect incoming pings, then manually updated the sensor in Home Assistant. Pings came through instantaneously - so there is no problem with delays affecting initiation of commands or scripts in sensors.

Next - I have separated the ping command from the sensor by putting it into a bash file.

configuration.yaml:

sensor:
  - platform: command_line
    name: AV Receiver
    command: sh /config/av_receiver.sh
    command_timeout: 15
    scan_interval: 60

av_receiver.sh:

#!/bin/bash
ping -w 5 -c 1 192.168.1.147 > /dev/null 2>&1 && echo 'on' || echo 'off'

Interestingly, when the AV receiver was on, this all worked fine. As soon as I turned it off, the status wouldn't update to reflect it and I once again got:

Updating command_line sensor took longer than the scheduled update interval 0:01:00
8:45 PM helpers/entity_platform.py (WARNING) - message first occured at 8:35 PM and shows up 10 times

Whilst still borked out in this state, I can still execute the shell script without any problems using the Configurator add-on - it works exactly as expected:

Command executed: sh /config/av_receiver.sh: 0
off

So here is my new theory: ping works fine in itself, but the way its responses are processed by the sensor or binary_sensor elements of HA are problematic, particularly if no packets are received. Due to the -w 5 header, the script shouldn't (and doesn't, as proven by testing in Configurator) take more than 5 seconds to execute a response, and the sensor should timeout anyway after 15 seconds - but it doesn't. It lingers, stuck, apparently waiting for a response, indefinitely, until HA is restarted.

[darkkatarsis]

I have exactly the same behavior. @thomasloven can you help with this?

[aegjoyce]

Hi all

Hoping I don't jinx it by saying this but I think it's all fixed now. Tried many, many different ways to fix it but ended up settling with using my wifi device tracking and modifying the consider_home values on a per-device basis to get a good level of response.

For those suffering with lots of 'took longer than the scheduled update interval' problems - I eventually swore loudly and committed to a total reinstallation of HA. Immediately ran into issues with my SD card being corrupted - new card in, reinstalled, and now, now problems (I hope).

I was finding that ANY sensor requiring some sort of command line function was throwing up these errors - sensor.py, binary_sensor.py, ping.py, you name it - all had update interval issues.

I can only conclude that these errors were getting caused by either a) a bloated installation requiring a reinstall and refresh OR b) the corrupted blocks on my SD card.

So do consider reinstallation or replacing your SD cards!

EDIT: Nope, not fixed. Even with full wipe, new SD card, and rebuilding my Home Assistant from scratch, still getting problems. Errors with all different types of command_line sensor.

I can confirm that I also get a jump in RAM usage from about 28% to 34% consistently with the jump coinciding with the first time the error occurs.

It's really starting to seem like if a command_line sensor fails to update, the process is not stopped gracefully and continues to run - even if I had shutdown and timeout commands programmed in.

Really desperate for some help with this now!

[aegjoyce]

Brand new install, this time on an SSD using HassOS 3.5 and USB booting. So, so much faster and has fixed almost all the usual errors that would occur except this one.

Error first gets logged at 10:20pm. You can see the CPU and RAM spikes that coincide with it, but no apparent changes to network connectivity. The sensor that broke in this case was the CPU temperature sensor, which doesn't require any network connectivity anyway.

Here are all my command_line sensors, in case anyone can see what is causing the issue:

  - platform: command_line
    name: CPU Temperature
    command: "cat /sys/class/thermal/thermal_zone0/temp"
    unit_of_measurement: "°C"
    value_template: '{{ value | multiply(0.001) | round(1) }}'
  - platform: command_line
    name: electricity
    value_template: '{{ value_json.count }}'
    scan_interval: 600
    json_attributes: 
      - results
    command: 'sh /config/bash_scripts/electricity.sh'
  - platform: command_line
    name: gas
    value_template: '{{ value_json.count }}'
    scan_interval: 600
    json_attributes: 
      - results
    command: 'sh /config/bash_scripts/gas.sh'
  - platform: command_line
    name: Household Waste
    value_template: '{{ value.split("</td><td>")[2]|replace("</td>","") }}'
    scan_interval: 600
    command: 'sh /config/bash_scripts/household_waste.sh'
  - platform: command_line
    name: Recycling
    value_template: '{{ value.split("</td><td>")[2]|replace("</td>","") }}'
    scan_interval: 600
    command: 'sh /config/bash_scripts/recycling.sh'

[gsotiriou]

I have wake_on_lan sensors which update their state by using ping. I have had issues in 0.98.5 in Hass.io on Rpi where after a reboot they would work and after a while they would stop being able to update indicating that the ping could not go through. Unfortunately I cannot recall the last version that it used to work. This started unexpectedly for me.

[gsotiriou]

Any update on this? Has it been fixed in the recent 0.99.x release?

[gohm44]

I am on version 0.99.2 and issue is still valid.
I have tried an installation from scratch still the same.

[siom7]

Hi, I have same issue and I finally install Home Assistant in Python Virtual Environment on a Debian Stretch and no issues now all is working perfectly. I guess this issue is caused by Docker.

[gsotiriou]

Could someone pick this up and take a look as to why it doesn't work in docker installation as intended?

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[mynameisdaniel32]

I'm seeing this issue still on 0.103 - in my case it seems the ping sensor component can't handle multiple ping requests gracefully. When one of them times out with no response, it significantly delays other ping sensors.

I'm running HassOS in a VM on an i7 machine.

[Twanislas]

Hi !
This is still an issue for me as well. I have 1 wake on lan sensor (desktop pc) and 1 ping sensor (internet status, pings 1.0.0.1) :

- platform: ping
  host: 1.0.0.1
  name: Internet
  scan_interval: 15

- platform: wake_on_lan
  name: Twan-PC
  mac: "xx:xx:xx:xx:xx:xx"
  host: 10.0.7.1
  broadcast_address: 10.0.7.255



md5-14f80c72d56e9f643d426658c03cb785



arch: armv7
channel: stable
hassos: "2.12"
homeassistant: 0.103.6
hostname: hassio
logging: info
machine: raspberrypi3
supervisor: "195"
supported_arch:
- armv7
- armhf

Manual pings using ssh work perfect.
Hope this helps :)

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.