Core: `Error setting up entry /dev/ttyUSB1 for zha` after upgrading to 0.88

[mkArtak]

Home Assistant release with the issue:

0.88

Last working Home Assistant release (if known):
0.87

Operating environment (Hass.io/Docker/Windows/etc.):
Hass.io running on RaspberryPI 3 B+.
I'm using the HUSBZB-1 USB stick and have a bunch of ZigBee as well as ZWave devices.

Component/platform:

Description of problem:
After upgrading to 0.88 (and now to 0.88.1) all the ZigBee (zha) devices show as Unavailable.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

zha:
  usb_path: /dev/ttyUSB1
  database_path: /config/zigbee.db

Traceback (if applicable):

Error setting up entry /dev/ttyUSB1 for zha
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/homeassistant/config_entries.py", line 260, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/zha/__init__.py", line 176, in async_setup_entry
    await asyncio.gather(*init_tasks)
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/zha/core/gateway.py", line 159, in async_device_initialized
    await zha_device.async_initialize(from_cache=(not is_new_join))
  File "/usr/local/lib/python3.7/site-packages/homeassistant/components/zha/core/device.py", line 192, in async_initialize
    BASIC_CHANNEL).get_power_source()
AttributeError: 'NoneType' object has no attribute '

Additional information:

[dmulcahey]

Few questions:

Can you set zha, bellows and zigpy to debug and restart and put the full log on pastebin?

What devices do you have paired? Are you using any custom lib versions or did you write any quirks on your own?

If the log isn’t enough I may need a copy of your zigbee.db as well.

[prairiesnpr]

I had the same issue after upgrading. Ended up adding a try/except block around self.power_source = self.cluster_channels.get(BASIC_CHANNEL).get_power_source() and then removing the offending entities and re-adding them.

[dmulcahey]

@prairiesnpr that's good to know... still have to find out what is causing basic to be missing but in the mean time I can patch ZHA to be more forgiving.

[mkArtak]

Hi @dmulcahey.
I'm not familiar with the bellows and zigpy components. But I assume this is the configuration to get the logs you asked for:

logger:
  default: critical
  logs:
    homeassistant.components: info
    homeassistant.components.zha: debug
    bellows: debug
    zigpy: debug

Please confirm so I can do the restart/extract-logs and share them with you.
As for the devices, I have a set of Xiaomi Aqara sensors and all those worked just fine for the previous (0.87) version.

[dmulcahey]

That will work. I use a ton of aqara devices during development and they are all working fine. My gut feeling is that you have some that aren’t fully paired.

[mkArtak]

I know I have one which isn't fully paird, and that one was causing an error log in the past, but not impacting the overall zha. I'll get the logs in a moment.

[prairiesnpr]

If the device isn't fully paired, that will cause this error. In my case I had a Aquara Door Sensor (LUMI lumi.sensor_magnet.aq2) and a Aquara Vibration Sensor (LUMI lumi.vibration.aq1) that had not fully paired. Once I removed them, ZHA worked normally.

[mkArtak]

Thanks @prairiesnpr.
I'll try that too as soon as I'll get home.

[mkArtak]

@dmulcahey, I updated the logs but doesn't look like anything interesting came up:

All the TPLink related errors are over WiFi and known things for me. So I'm not concerned about those.

[mkArtak]

@prairiesnpr, I'm wondering how can I remove a zha node? I see no available nodes now. I was thinking there to be a service like zha.remove or something like that, but there are no any zha services as the component didn't load at all. So I'm not sure how to remove the bad node.

[dmulcahey]

@mkArtak please provide the entire log from HA on pastebin or something similar and let me determine if it is useful.

[dmulcahey]

Ok in your case there may be some confusion. Go trip one of the magnets for me... this was in the release notes. The devices will show as unavailable until they communicate for the first time. Xiaomi devices only communicate every 50-60 minutes unless they have a change to report.

[mkArtak]

@dmulcahey, I triggered a door sensor as well as later manually pushed the button on the motion sensor, but I see no changes - all the devices are still unavailable.

[dmulcahey]

They probably fell off the network then, it’s very common for Xiaomi devices. Rejoin them completely.

[mkArtak]

@dmulcahey, how should I go about that? The zha component is completely unavailable, so the Permit calls fails. Should I remove the database manually and restart, so it recreates it again?

[dmulcahey]

The log you sent me is completely clean. What do you mean the “zha component is unavailable”

[mkArtak]

Well, when I click Permit on this screen:

I get the following error:

And when I go into the Services tab, there are no any zha services showing up.

[dmulcahey]

Do you have reserved names for your serial usb devices?

[prairiesnpr]

This is consistent with what I experienced. You need to remove the following device.

2019-02-23 17:51:29 DEBUG (MainThread) [homeassistant.components.zha.core.device] None None: started initialization

This would need to be done manually via SQLite.

[tunatoksoz]

I am seeing the same issue.

The problem i noticed is of the same device types (say motion sensors), some are recognized and have quirks mode enabled, others are recognized as zigbee device only. This might explain why its crashing zha.

[dmulcahey]

@tunatoksoz what devices are you seeing this with?

[tunatoksoz]

I cant tell which one is erroring out. But i have aqara motion sensors, presence sensors, and temperature sensors.

[tunatoksoz]

I only have xiaomi sensors. I saw this quirk difference with motion sensors for sure. Dont remember others.

[dmulcahey]

@mkArtak @prairiesnpr @tunatoksoz can you guys try these changes:

https://github.com/dmulcahey/home-assistant/commit/d4694c2fb230acfb8dea347b0573ff830a517403

and let me know how things work out? They should fix this situation and enhance the power configuration handling in general.

[mkArtak]

@dmulcahey, I actually run into a bunch of issues (probably screwed myself) by trying to reset my zha state completely.
First, following @prairiesnpr's advice, I've removed some devices from the SQLight db but that resulted in some other errors on the UI.
Then I removed the zha config from the configuration.yaml, which somehow didn't have any effect after the restart. So I manually tried to delete the zigbee.db, but it got recreated after the restarted.
I ended up removing the ZHA' integration, and then removing thezigbee.db`.

Finally, I just brought back the zha config in the configuration.yaml and restarted again. The integration is back with no devices, but he Permit call again fails.

So I'm not sure what's going on.

I made a big mistake this time not backing up before taking the 0.88 update, which I usually do.

@dmulcahey, do you think the state I'm in is still valid for trying out the change you've proposed?

[mkArtak]

Finally, after another round of removal of the ZHA integration, zha config and zigbee.db with a following restart cycle things came back to normal and I can successfully call the Permit and pair my devices.

[mkArtak]

I think now I'm seeing another issues somebody else had reported earlier:
After pairing a door/window sensor successfully and seeing it fine on the dashboard, I restarted again.
And it came back like this:

But, this time the sensor actually works just fine and it's node is shown in the Configuration/ZHA just fine:

[prairiesnpr]

@dmulcahey I can confirm that those changes will allow home assistant to start with a device that is missing a basic cluster.

To install that specific branch (Assuming you are using a virtual environment in the suggested location)
Activate your venv.

sudo -u homeassistant -H -s
source /srv/homeassistant/bin/activate

And then run the following command to install the branch with the changes.

pip install git+https://github.com/dmulcahey/home-assistant.git@dm/zha-forgive-missing-basic --upgrade --force-reinstall

And then restart home assistant.

[dmulcahey]

@mkartak there is an issue with manufacturer and model not being ready for HA when joining devices. I have to dig deeper to see where it is. My suspicion is that it’s in Zigpy. The solution is just don’t do anything with the UI until after a restart. They’ll be there with the correct name at that point.

Also, if you haven’t set a static name for the USB stick you’re going to have this issue again at some point. You should look at setting up a symbolic name for it so it survives restarts and removing and reading the usb device.

[tunatoksoz]

Is there anything to make sure correct quirk is applied? Is removing from db/ha and rejoin the only way?

[dmulcahey]

Removing and readding is the ONLY way. The device is not joined correctly. Quirks work by matching cluster definitions to what is in zigbee.db. If they don’t match, they don’t apply and that probably isn’t going to change. The device probably didn’t finish pairing, Xiaomi is notorious for this. I posted instructions on the forum topic that you’ve been posting to that help mitigate this. While joining, press the button every 3-5 seconds until you see all the sensors in the UI

[mkArtak]

@dmulcahey would you please suggest where I can read more about how to set reserved name for a device?

[dmulcahey]

@mkartak http://hintshop.ludvig.co.nz/show/persistent-names-usb-serial-devices/

[tunatoksoz]

@dmulcahey @prairiesnpr Thank you. This seem to have improved a bunch.
Removed zigbee.db and started everythinig from scratch. Quirks are so far all detected nicely - i have about 3 sensors left i think that i need to set up, but everything looks great so far.

[dmulcahey]

@mkartak are you good now?

[glassbase]

I have this issue with a couple Xiaomi temp sensors. I removed the node, restarted HA, add back, and same thing, no quirks. What is the proper process? Do I need to manually remove from zigbee.py also?

Edit - tried to remove node, checked zigbee.db and they were not in the db so I guess that means it removed fine. I noticed when I add back the sensor I show 'None None' node but it has quirks, but as soon as I restart HA, the node has the default quirk again.

[mkArtak]

@dmulcahey yes. Thanks for all the support!

[mkArtak]

I'd leave it up to you to decide whether the issue should be closed or not, given the amount of activity here.

[dmulcahey]

I'll leave it open until the fix is finalized, PR'd and merged.

[dmulcahey]

fixes are merged for this and will be in the beta today. closing

[dennyreiter]

This is consistent with what I experienced. You need to remove the following device.

2019-02-23 17:51:29 DEBUG (MainThread) [homeassistant.components.zha.core.device] None None: started initialization

This would need to be done manually via SQLite.

I just had this happen. I was trying to add a Xiaomi switch which wasn't showing up even though I saw the permit process start. Then everything went unavailable with this error until I went in and deleted it from the tables.