Core: Unable to find Vera entitites

[MetalOnes]

The problem

Using Vera Pro with the HASS integration (via YAML).
Occasionally entities are not responsive to turning on/off. Logs say that HASS is unable to find the referenced entities. When I log in to the Vera directly, everything is working normally.
After restarting HASS, they are usually available again, but will stop working again after some time.

Environment

arch | x86_64
dev | false
docker | false
hassio | false
os_name | Linux
os_version | 4.4.0-173-generic
python_version | 3.8.2
timezone | America/New_York
version | 0.107.5
virtualenv | true

- Home Assistant release with the issue: 
- Last working Home Assistant release (if known): 0.106.4
- Operating environment (Hass.io/Docker/Windows/etc.): Ubuntu Server 16.04
- Integration causing this issue: Vera
- Link to integration documentation on our website: https://www.home-assistant.io/integrations/vera/

Problem-relevant configuration.yaml

vera:
  vera_controller_url: http://[VERA IP]

Traceback/Error logs

Log Details (WARNING)
Logger: homeassistant.helpers.service
Source: helpers/service.py:371
First occurred: March 23, 2020, 8:04:45 PM (8 occurrences)
Last logged: 7:27:07 AM

Unable to find referenced entities switch.man_cave_lights_133
Unable to find referenced entities switch.back_spotlight_118, switch.deck_lights_108, switch.front_porch_115, switch.front_spotlight_135
Unable to find referenced entities switch.front_porch_115
Unable to find referenced entities switch.front_spotlight_135
Unable to find referenced entities switch.back_spotlight_118

Additional information

[frenck]

Hi there,

Could you please use the issue template provided when creating an issue? Thanks 👍

[MetalOnes]

Sorry didn't mean to close it, haven't reported many issues before so struggled a bit with Github.

[frenck]

Don't worry m8! We all need to learn and some point, right? Thanks for updating the issue 👍

[Gibby]

I also have been having this issue intermittently the past 2 months, more lately the past 5 days or so. When it happens, I also see that home assistant is trying to add the entity again. Example:

homeassistant.exceptions.HomeAssistantError: Entity id already exists: light.office_light_113. Platform vera does not generate unique IDs

I can still control other vera devices with no issues.

Some times it will add a duplicate entity by appending _1 to it, so in the above example in home assistant I end up 2 entities mapped back to the same device in vera, like
light.office_light_113 light.office_light_113_1

About a month ago I tried removing vera completely, cleaned up all the entities from home assistant and added vera back. Everything was imported correctly and I had no issues for about 2 weeks.

[gottsman]

I am having issues as well... Mine has a slightly different error.

2020-04-01 13:29:15 ERROR (Vera Poll Thread) [pyvera.subscribe] Unhandled exception in callback for device #136 (Half Bath Fan)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pyvera/subscribe.py", line 155, in _event_device
    callback(device)
  File "/usr/src/homeassistant/homeassistant/components/vera/__init__.py", line 162, in _update_callback
    self.schedule_update_ha_state(True)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 430, in schedule_update_ha_state
    self.hass.add_job(self.async_update_ha_state(force_refresh))
AttributeError: 'NoneType' object has no attribute 'add_job'
2020-04-01 13:29:15 ERROR (Vera Poll Thread) [pyvera.subscribe] Unhandled exception in callback for device #137 (Living Room Primary)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pyvera/subscribe.py", line 155, in _event_device
    callback(device)
  File "/usr/src/homeassistant/homeassistant/components/vera/__init__.py", line 162, in _update_callback
    self.schedule_update_ha_state(True)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 430, in schedule_update_ha_state
    self.hass.add_job(self.async_update_ha_state(force_refresh))
AttributeError: 'NoneType' object has no attribute 'add_job'

[Gibby]

I tried for hours the other night to figure out what was going on, I deleted the db, removed the integration, cleaned up the entities. When I added the integration back, everything was fine until the next restart of HA. I tried this like 6 times. In the end I ended up renaming the entities after first import and dropped the vera_id from the end of the name. I still get these errors but everything is still working.

[alanthewild]

(updated to remove the off-topic comment)
For me this actually happened a few versions ago (I think it was 104.3?) but it only happened a couple of times on a restart and a second restart resolved it at that point.

With 107.7 I have rebooted multiple times and different devices are working/not-working every reboot. This makes my dashboard useless.

This is happening with all types of devices - ie sensors, lights, switches.

Happy to be a guinea pig if we can get to the bottom of this.
Any suggestions?

[alanthewild]

@vangorra I just came across your comment in the Pyvera Github repository. (https://github.com/pavoni/pyvera/issues/128#issuecomment-607886064)
Is there any way we can beta test your potential fix for the Vera unique ids earlier than a couple of months away? Or any other suggestions you have that might help?

[MartinHjelmare]

Please open an new issue for any issue that has a different stack trace than in the original post. We don't want to collect all Vera issues in this issue.

[vangorra]

This PR has multiple vera hub support and also fixes the unique entity error (for newly added hubs only). You will have to remove and re-add your old hubs to get the fix applied. There is no migration option possible.

https://github.com/home-assistant/core/pull/33613

[asahani]

Thanks for working on this @vangorra
Do you have an ETA as to when would this be released? 50% of my integrations have been broken since upgrading to 106.6.

[vangorra]

No ETA. The maintainers will get to it when they can.

[alanthewild]

It is in the latest betas of 109.
I tested home-assistant-109.0.dev20200411 and 413 (in docker) but the
option to include devices (switches) as lights didn't work which breaks
about 40% of my devices.
Excluding specific devices didm't work either. I excluded device 382 (which
is the second half of a double switch I'm not using) and it was still
imported and added to the gui.

But adding Vera using the integration is much easier than YAML.Particularly
when you can delete and re-add Vera multiple times WITHOUT restarting Home
Assistant!

Thanks for your work on this though Robert. There are still a few of us who
haven't thrown in the towel with Vera just yet (although I'm getting close
to be fair).
.

On Sun, Apr 12, 2020 at 3:57 PM Robert Van Gorkom [e-mail]
wrote:

No ETA. The maintainers will get to it when they can.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/core/issues/33208#issuecomment-612560560,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG22ZC65NIU5BCGCJ5VLAILRME34DANCNFSM4LST53DA
.

[vangorra]

@alanthewild You you testing from a clean config? If so, then we likely have a bug.

If not, could you search .storage/core.entity_registry for entities with a unique ID that is an integer? The only ones should be vera. I suspect the old config is still getting used for soon reason.

I'll see if I can reproduce each condition.

[vangorra]

@alanthewild I fixed a few things relating to the legacy id import in the PR. Be sure while testing that you use a clean config/.storage (no vera controller was ever setup, even though config file) or a config/.storage from an existing stable release of HA.

[alanthewild]

Thanks Robert.

I used a fresh/blank/vanilla install off a new docker image and the only
thing I added was Vera. To be clear, there was a newly created folder
assigned to /config before I started up the container so there is no way
there was a .storage folder beforehand.
The issue was apparent on the 20200413 version I tried yesterday and the
20200411 version I tried before that. Will wait for the next version of the
beta to come out and try again.

On Tue, Apr 14, 2020 at 3:08 AM Robert Van Gorkom [e-mail]
wrote:

@alanthewild https://github.com/alanthewild I fixed a few things
relating to the legacy id import in the PR. Be sure while testing that you
use a clean config/.storage (no vera controller was ever setup, even though
config file) or a config/.storage from an existing stable release of HA.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/core/issues/33208#issuecomment-612939022,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG22ZCYBWKUW5FSASP5U5ETRMMTFHANCNFSM4LST53DA
.

[alanthewild]

Fully clean config. I tried it initially with an upgraded setup.
Then, when it didn't work the way it should, I created a whole new /config
folder and new docker container from the latest beta.
Let me know if you want me to test anything.

On Tue, Apr 14, 2020 at 1:37 AM Robert Van Gorkom [e-mail]
wrote:

@alanthewild https://github.com/alanthewild You you testing from a
clean config? If so, then we likely have a bug.

If not, could you search .storage/core.config_entries.json for entities
with a unique ID that is an integer? The only ones should be vera. I
suspect the old config is still getting used for soon reason.

I'll see if I can reproduce each condition.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/core/issues/33208#issuecomment-612901604,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG22ZC7ZDN7XMTV3MISJSVLRMMISHANCNFSM4LST53DA
.

[alanthewild]

@vangorra I have tested the latest Vera integration with a fresh install of the latest 109.2 HA.
Same issue as with 109 DEV versions - excluded devices and devices to import as lights are not working as expected. All device IDs set as lights get imported as switches. And excluded devices are still setup and visible in HA.

Can provide screen shots if you'd like.

[vangorra]

Screenshots of what vera says and what home assistant says would be helpful.

[alanthewild]

Legacy Vera config on my normal HA instance: (HA 108.9)

   vera_controller_url: http://192.168.1.18:3480/
  # Optional to exclude devices - this is a list of vera device ids
   exclude: [2,3,4,5,6,7,8,9,10,11,12,13,20,206,116,263,235,237,254,313,382]
  # Optional to import switches as lights - this is a list of vera device ids
   lights: [154,91,84,153,152,155,21,119,22,120,207,208,212,224,225,236,264,265,255,266,390,340]

Showing one of the lights that is a switch in Vera (legacy integration)

And a device that is excluded so doesn't appear: (legacy integration)

Using the new integration in a fresh docker install (config directory was newly created and empty): (using HA 109.2) (Note 91 is included as a light and 382 is excluded)

The switch that should have been added as a light: (new integration)

And the same device that should have been excluded: (new integration)

[vangorra]

@alanthewild The PR has been updated with a fix to the issue you reported.

[alanthewild]

@vangorra Is there any way of knowing when this PR gets added into the dev or release version?

[vangorra]

Probably not until 0.110dev. best case scenario, that's probably a few weeks away.

[vangorra]

Regarding entities showing as unavailable. It looks like this has been reported before for older versions of this integration. #32231
Let's use that original ticket for troubleshooting and updates to keep the communication organized.

[alanthewild]

@vangorra I can confirm your fix is included in the latest dev version - 0.110.0.dev20200506
All works as expected - switch entities that are meant to be imported as lights are - entities that are excluded are non-existent in HA as expected. Thanks again.

[vangorra]

I submitted a PR that I suspect will fix the issue of entities not getting updated. Since I have been unable to reproduce the issue, please test out the PR if you can. https://github.com/home-assistant/core/pull/35703

Based on the responses from many users, this issue appears to be caused by the controller not returning accurate incremental data or a race condition with threads could result in data being squashed. At any rate, this PR ensures that all status data is retrieved from the controller (not only incremental) and all this is done using proper hass event loops.