Core: deconz component not update states in 0.73

[donnib]

Home Assistant release with the issue: 0.73

Last working Home Assistant release (if known): 0.72.1

Operating environment (Hass.io/Docker/Windows/etc.): Hass.io

Component/platform: https://www.home-assistant.io/components/deconz/

Description of problem:
The entities does not reflect the state and i can't turn the entities off for example lights.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:
Here is the debug output :

2018-07-05 20:48:48 DEBUG (MainThread) [pydeconz.websocket] Websocket starting
2018-07-05 20:48:48 ERROR (MainThread) [pydeconz.websocket] Unexpected error None
2018-07-05 20:48:48 DEBUG (MainThread) [pydeconz.websocket] Websocket starting
2018-07-05 20:48:48 DEBUG (MainThread) [pydeconz.websocket] Reconnecting to deCONZ in 15.

[Kane610]

We've done some initial testing and for some reason the web socket can't connect to deconz. Donnib have tried this from two different machines with same issue. I've tried it native and in docker on my Mac, without any issue so far.

[mathos77]

I have the same issue with 0.72.1 as well btw, does throw quite a lot of exceptions all over anyway
Sorry, don't have much time currently to dive into it, maybe on Sunday-eve I will find some time.

[donnib]

@mathos77 that's interesting because in my tests the problem goes a way with 0.72.1 and i have tried many times.

[Kane610]

@mathos77 are you running hassio?

[Kane610]

This at least rules out any issue with the change in web socket implementation

[mathos77]

@Kane610 sorry for the late response, running in docker on Intel, Deconz is at 2.5.31, including the git-built rest.

[donnib]

I got the problem both with deConz on 2.05.21 and the latest 2.05.31 so it doesn't seem to be the problem with incompatibility with deConz, it's something in HASS/deconz component that causes this.

[mathos77]

I actually have a different issue.

2018-07-06 07:41:41 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/app/homeassistant/data_entry_flow.py", line 68, in async_init
    return await self._async_handle_step(flow, step, data)
  File "/usr/src/app/homeassistant/data_entry_flow.py", line 100, in _async_handle_step
    result = await getattr(flow, method)(user_input)
  File "/usr/src/app/homeassistant/components/deconz/config_flow.py", line 169, in async_step_import
    title='deCONZ-' + self.deconz_config[CONF_BRIDGEID],
KeyError: 'bridgeid'

Never changed the config, since I updated the container, was there a change in the config section, as it currently simply looks like:

[Kane610]

@mathos77 that is solved in 0.73. It was a small misstake from my part.

[donnib]

@mathos77 i was about to ask you if you were sure it wasn't a different issue, good to know
@Kane610 so back to square one :(

[donnib]

@Kane610 i have some time today until 15 if you need more of my help

[Kane610]

@donnib in your non hassio docker, if you could try to get more information from the error would be great!

It is this if that fails https://github.com/Kane610/deconz/blob/603c1c347e1e1ae24830c97db0fd14036ab7526f/pydeconz/websocket.py#L57

So it travels down to the https://github.com/Kane610/deconz/blob/603c1c347e1e1ae24830c97db0fd14036ab7526f/pydeconz/websocket.py#L64 and breaks

async with self.session.ws_connect(url) as ws:
    self.state = STATE_RUNNING
    async for msg in ws:
        if msg.type == aiohttp.WSMsgType.TEXT:
...
        elif msg.type == aiohttp.WSMsgType.ERROR:

If you find a way to get more information I will also put it in for future errors in the code to support debugging

[ACrazyConcept]

I just created a new issue before I saw this one:
https://github.com/home-assistant/home-assistant/issues/15337
I am running hassbian though. But I suspect the root cause could be the same.

[donnib]

@ACrazyConcept it's the same, doesn't matter what architecture or where you run HASS, have spent quite some time with @Kane610 to try figuring out the problem but were not able too. At the time i was the only one having this issue. @Kane610 could not reproduce this in his setup. I suspect more people will come with this issue. You may close the other issue ?

[Kane610]

@ACrazyConcept @mathos77 are you running deconz on a pi or x86? And are you running it natively or in a docker container?

[Kane610]

My current investigation points to there being a difference between running deCONZ on host and inside of a container. Where it works with the container

[ACrazyConcept]

@Kane610 hi it is me Sand from the discord chat whose pydeconz you fixed ;)

[Kane610]

Great! So I know you're running deconz without docker. My hypothesis still holds up

[mathos77]

Hi @Kane610 , I am running docker natively on the pi. Never got the Raspbee to properly work in a container.
Yet, the API should not be affected right.....perhaps....I know what it is, did you change the WebServer port detection ? I am running it on a different WS port than the default.

[Kane610]

@mathos77 there is no change in management of ws port, it is still retrieved from deconz.

I am pretty certain it actually is a difference in system packages that is the reason behind this issue. But I need help verifying this.

It would be great if you'd try just running deconz in docker with the same config path as your current deconz and see if it works with web sockets. Regardless if it actually connects to raspbee or not

[mathos77]

okay, I will do that, just tomorrow, I am currently not @home

[Kane610]

Absolutely

[mathos77]

@Kane610 I do not have the issue with 0.73.0 (just updated the container, and deCONZ is just running on the pi (native)

[Kane610]

You just broke my hypothesis :)

What os is the pi running?

[mathos77]

Rasbian, just a regular pi install

[aherbjornsen]

I encountered this issue on hass.io upgrading from 0.72.0 to 0.73.0.

Deconz is 2.05.23 running natively on Raspbian (I'm running the SD card image provided by Dresden Elektronik). FWIW I'm using a Raspbee.

Fortunately I have nightly snapshots and rolling back to 0.72.0 resolved the problem.

[Kane610]

@aherbjornsen would you mind going over to using deconz in a docker container?

[aherbjornsen]

@Kane610 That is not an ideal solution. I'm running deconz on the Raspbian image supplied by Dresden Elektronik, and I feel that this is something that should work with hass.io. I'm very comfortable with it and it has been running extremely stable for me since last November.

[xhemp]

I can test whatever is needed and can run on docker, just need to know how I can keep the actual deconz configuration. Im running conbee in a pi standalone

[Kane610]

Allright @xhemp

This is my docker compose, in the volumes section you can see how to reuse your current deconz config and database

version: '2'
services:
  deconz:
    container_name: deconz
    image: marthoc/deconz
    restart: unless-stopped
    network_mode: host
    devices:
      - /dev/ttyUSB0
    environment:
      - DECONZ_WEB_PORT=<webport>
      - DECONZ_WS_PORT=<wsport>
      - DEBUG_INFO=1
      - DEBUG_APS=0
      - DEBUG_ZCL=0
      - DEBUG_ZDP=0
      - DEBUG_OTAU=0
    volumes:
      - /home/<user>/.local/share/dresden-elektronik/deCONZ:/root/.local/share/dresden-elektronik/deCONZ

[xhemp]

Thanks @Kane610 .
It works well running with docker on 0.73:

2018-07-08 16:16:53 DEBUG (MainThread) [pydeconz.websocket] Websocket starting
2018-07-08 16:16:53 DEBUG (MainThread) [pydeconz.websocket] Websocket running

[xhemp]

I am not sure if something changed between 0.73 and 0.73.1 but I was able to use deCONZ natively (no docker) on 0.73.1.

[aronsky]

Weird - currently, 0.73.1 doesn't seem to have any changes at all compared to 0.73, other than the versioning...

[Kane610]

Yes that sounds weird indeed

[xhemp]

Very!
For the sake of it, I tried downgrading to 0.73 (not 0.73.1) to test and it worked as well so I have no idea why it didn't work before on 0.73, though.

[Kane610]

I've created an issue with deCONZ developers.

https://github.com/dresden-elektronik/deconz-rest-plugin/issues/684

[Kane610]

So it seems that old system dependencies is the issue here. You guys need to use a newer QT5, 5.7 to be more precise.

[Try2Fly]

For me Deconz is not working in 0.73.1 (hassio). I'm running Deconz on a separate PI (not docker). Deconz version 2.05.23 and use a Raspbee. Is there already a known fix? I'm downgrading my hassio to 0.72.1 for the moment.

[Kane610]

@try2fly you need to upgrade your raspberry to use debian stretch or similar modern release so you can fill the requirement I posted above

[Try2Fly]

@Kane610 Ok thanks, I'm running jessie on the Pi. I'll first try to update this. If it still fails then I'll update to stretch.

[aherbjornsen]

@Kane610 Thanks for the effort to solve this.
I upgraded my Deconz Pi from Jessie to Stretch and 0.73.1 is running fine now.

[Kane610]

Great to hear @aherbjornsen

[aronsky]

I'm upgrading to stretch at the moment, in the hope that it solves the problem. However, if it does - I suggest to update pydeconz to choose between aiohttp and legacy websocket implementation based on the version of the underlying software, to avoid forcing users to upgrade their distros.

[jkinghed]

I have the same issue - error messages and no updates to sensors:

2018-07-20 10:26:50 ERROR (MainThread) [pydeconz.websocket] Unexpected error None

Running Ubuntu 18.04 (with QT5.9.5) on a NUC with Docker 18.06.0-ce, and latest Home Assistant container (built 2018-07-16 22:58:24), Deconz version 2.05.32, Conbee firmware 261F0500.

When i run pip3 show pydeconz within the container i get version 39:

Name: pydeconz
Version: 39
Summary: A Python library for communicating with deCONZ REST-API from Dresden Elektronik
Home-page: https://github.com/Kane610/deconz
Author: Robert Svensson
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.6/site-packages
Requires: aiohttp
Required-by:

The Dockerfile states version 42 (link), so unsure it there are multiple versions inside the container?

I tried with Ubuntu 16.04 and QT5.5.1 or 5.6.5 (? can't remember now) before upgrading to Ubuntu 18.04 with QT5.9.5.

Do I need to install QT 5.7.1 on the host OS? Get pydeconz up to version 42 somehow?

[Kane610]

Release of pydeconz doesn't matter since there is no changes related to this issue after 39.

QT5.71 should be enough (https://github.com/dresden-elektronik/deconz-rest-plugin/issues/684)

Since you already run home assistant in docker wouldn't it be easier to go over to the deConz container?

[jkinghed]

@Kane610 aha - the qt5 issue is with the deConz installation, not HA/pydeconz?

I have two rpi's running deConz, each with a Conbee stick on the opposite ends of my house both running Strech / QT5.7.1 / 2.05.32.

(Since pydeconz in HA only can handle 1 instance, I'm getting sensor values using REST from the other)

[Kane610]

@jkinghed yes it is system dependencies to deConz that is the issue here.

[jkinghed]

@Kane610 I have too many rpi's, it turns out.. Was looking at the wrong one, but after upgrading the correct rpi to Stretch I can confirm it works OK in HA.
Thanks for your input and assistance!

[Kane610]

Great! Thanks for double checking @jkinghed

[donnib]

@Kane610 I fixed the issue as described, i upgraded by deconz to use the latest rPi image with Stretch.

[aronsky]

While the original issue opener solved it by upgrading to Stretch, the issue still needs a solution. Jessie is a supported platform for both Home Assistant and the Deconz REST plug-in. It's unreasonable to expect everyone who uses this component to suddenly upgrade their whole Raspberry Pi system - especially since it worked perfectly with pydeconz 38 and earlier.

@Kane610, since you're the maintainer of pydeconz, I repeat the suggestion to check the underlying OS version, and fall back to the older websocket implementation if required. It would create an unnecessarily large code in the library, though. Another option (since it doesn't seem the HA component uses new features from the updated version) could be to perform the test on the HA component side, and install an older (v38) version of pydeconz when running on Jessie.

[Kane610]

@aronsky there is no common way to verify OS or QT version from pydeconz since it is deCONZ which is affected by the system version, which can be run on any computer. I've asked for a comment from the developer of deconz on limiting supported QT5 versions. Depending on their response, I will add a legacy option for web sockets in pydeconz manually configured through the components setup.

It is great that you are adamant in this issue, Im happy people are using this component.

[aronsky]

@Kane610 thank you! I appreciate all the work you've done on the component, it does an admirable job in my home.

[Kane610]

@aronsky I've prepared so it is possible to manually configure pydeconz to use my own WebSocket implementation, it will be available with next pydeconz release which will come together with the new PR to add switch platform for power plugs.

It can be enabled by manually editing the .homeassistant/.storage/.core.config_entries file and add "legacy_websocket": true, inside the data structure

[cptpl4net]

@Kane610 thank you for your work!

I had the same issue with non updating states with a newly installed deconz on a separate windows machine. It took a while to figure out: the websocket port was closed.
For anyone with the same issues: the deconz websocket can be configured on any free port in the config (mostly its deconz.local:443). So you have to open that port on the Firewall you are using. If 443 is used by your webserver deconz will choose another port.

PS: to find out what port is used by the websocket of deconz: curl http://deconz.local/api/APIKEY/config/

[Kane610]

@cptpl4net thanks! Though the issue you are mentioning is different from the one in this issue which is about different behavior in deconz depending on system dependency versioning.