Core: Arlo started same symptom of Platforms that do I/O inside properties and battery level update failure

[edif30]

Make sure you are running the latest version of Home Assistant before reporting an issue.

You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:

Home Assistant release (hass --version):
0.55

Python release (python3 --version):
3.6.2

Component/platform:
Arlo

Description of problem:
Update state of arlo camera taking excessively long

Expected:
Quick Arlo cam update

Problem-relevant configuration.yaml entries and steps to reproduce:

1. 
   
   
   2.
   
3.

Traceback (if applicable):

WARNING (MainThread) [homeassistant.helpers.entity] Updating state for camera.front_cam took 19.038 seconds. Please report platform to the developers at https://goo.gl/Nvioub

2017-10-07 17:49:50 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 381, in async_process_entity
    new_entity, self, update_before_add=update_before_add
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 238, in async_add_entity
    yield from entity.async_update_ha_state()
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 237, in async_update_ha_state
    device_attr = self.device_state_attributes
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/camera/arlo.py", line 104, in device_state_attributes
    self._camera.get_battery_level,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/pyarlo/camera.py", line 132, in get_battery_level
    return base.get_camera_battery_level[self.device_id]
TypeError: 'NoneType' object is not subscriptable
2017-10-07 17:49:50 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 381, in async_process_entity
    new_entity, self, update_before_add=update_before_add
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 238, in async_add_entity
    yield from entity.async_update_ha_state()
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 237, in async_update_ha_state
    device_attr = self.device_state_attributes
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/camera/arlo.py", line 104, in device_state_attributes
    self._camera.get_battery_level,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/pyarlo/camera.py", line 132, in get_battery_level
    return base.get_camera_battery_level[self.device_id]
TypeError: 'NoneType' object is not subscriptable
2017-10-07 17:49:50 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 381, in async_process_entity
    new_entity, self, update_before_add=update_before_add
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 238, in async_add_entity
    yield from entity.async_update_ha_state()
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 237, in async_update_ha_state
    device_attr = self.device_state_attributes
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/camera/arlo.py", line 104, in device_state_attributes
    self._camera.get_battery_level,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/pyarlo/camera.py", line 132, in get_battery_level
    return base.get_camera_battery_level[self.device_id]
TypeError: 'NoneType' object is not subscriptable
2017-10-07 17:49:50 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 381, in async_process_entity
    new_entity, self, update_before_add=update_before_add
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 238, in async_add_entity
    yield from entity.async_update_ha_state()
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 237, in async_update_ha_state
    device_attr = self.device_state_attributes
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/camera/arlo.py", line 104, in device_state_attributes
    self._camera.get_battery_level,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/pyarlo/camera.py", line 132, in get_battery_level
    return base.get_camera_battery_level[self.device_id]
TypeError: 'NoneType' object is not subscriptable
2017-10-07 17:49:50 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 381, in async_process_entity
    new_entity, self, update_before_add=update_before_add
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity_component.py", line 238, in async_add_entity
    yield from entity.async_update_ha_state()
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 237, in async_update_ha_state
    device_attr = self.device_state_attributes
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/camera/arlo.py", line 104, in device_state_attributes
    self._camera.get_battery_level,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/pyarlo/camera.py", line 132, in get_battery_level
    return base.get_camera_battery_level[self.device_id]
TypeError: 'NoneType' object is not subscriptable

Additional info:

[bachya]

Wondering if my my PR caused this. Not exactly sure how to address – will dig. @MartinHjelmare, any ideas?

[jwillaz]

I have a PR in the pyarlo library that should help with some of the NoneType errors: https://github.com/tchellomello/python-arlo/pull/48

I don't think the timeouts are specific to the Arlo component, though. I have a dozen other components that are throwing long-running warnings, as well.

[tchellomello]

@jwillaz @bachp I had some issues with the latest 0.55 too.

@jwillaz let's address some points on the PR you sent, then we can test it using the new pyarlo to see if it will fix the issue.

[jwillaz]

Upon further testing, there definitely seems to be an issue with the camera component. I can load the Arlo platform and sensor component just fine - takes less than 30 seconds to restart HASS. However, if I load the camera component, HASS takes an excruciatingly long time to load (doesn't seem like it's coming back up sometimes).

[tchellomello]

@jwillaz agreed. I think the it is related with the device_attributes and it's refresh rate on the camera component.

To test it, I disabled my arlo camera and HASS got back to normal. If I enabled the camera, I barely can load the HASS frontend.

[jwillaz]

Quick fix: Remove battery_level, motion_detection_sensitivity, signal_strength, and unseen_videos from device_state_attributes in the HASS arlo camera component (arlo.py). Also, add STATE_UNKNOWN constant back to import in Arlo sensor component and replace None returns with STATE_UNKNOWN.

[bachya]

Been monkeying around with this, but don't see a good way to fix it. I think it's occurring because the majority of the property calls – get_battery_level, etc. – require going back to the base station with a separate request.

@tchellomello: is there a way to augment PyArlo to return all camera attributes in one shot (perhaps that's what this PR is for)? If not (and unless anyone else has insight into a strategy using HASS' inner workings) we may have to pursue something like what @jwillaz mentions and scrub those outlier attributes from the platform altogether.

[bachya]

Me again. 😎 I _may_ have _a_ solution; not sure if it's the correct one, but it seems to populate the attributes without blocking anything (including loading of the camera imagery): https://github.com/home-assistant/home-assistant/pull/9758

Only noticeable difficulty is that, initially, all the camera properties are null – because I'm adding separate tasks to the executor pool for each camera's attributes (and because, as I mentioned previously, each thread has multiple RESTful calls to Arlo), it takes a bit to update the first time. Nothing earth-shattering.

[jwillaz]

Putting the attribute updates in a separate thread is a good direction. Additionally, there needs to be some logic to selectively query for attribute updates. Realistically, these attributes don't change very often, yet they are constantly queried. This leads to thousands of hits against the Netgear endpoints. Though unlikely, I think the last thing we want is Netgear IP banning because we're DDOS'ing them.

[bachya]

@jwillaz That makes sense. Any suggestions on how to determine whether a re-query is needed? At a very simplistic level, I could just throttle that method to every 10 minutes or so...

[MartinHjelmare]

Update of state including attributes, where I/O is involved, should be done in the update entity method, as far as home assistant goes. There are ways of limiting web api calls in home assistant, eg the Throttle decorator.

The stack trace suggest there's also an issue with the pyarlo library.

[tchellomello]

I think we need to do a better job with caching the attributes on the PyArlo library too.
We currently have a few PRs that will work towards that direction.