Addons: Updated deCONZ addon to 4.0 and 4.1 - missing entities

[VivantSenior]

Hi,

my config is:

• Conbee 2,
• RPi 3B+,
• Hass.io 0.103.2,
• deCONZ add-on 4.1.

I noticed that after upgrading to deCONZ 4.0 my entities are missing. The same issue is present in 4.1.

Logs:

ERROR (MainThread) [homeassistant.components.deconz] Error connecting to deCONZ gateway at core-deconz
WARNING (MainThread) [homeassistant.config_entries] Config entry for deconz not ready yet. Retrying in 80 seconds.

WORKAROUND by @frenck:

The add-on sends a reconfiguration discovery notification to Home Assistant to adjust the settings. However, if in the past a manual configuration flow was used, that probably won't work in that case. The only solution at that point is by removing the integration -> restart Home Assistant -> restart the add-on. After that, a integration should be discovered and visible on the integration screen.

[jan-dewit]

@VivantSenior thanks for opening this, I was just logging in to open this issue myself.

I can confirm this issue.
My config:

• Conbee 1
• RPi 3B+
• Hass.io 0.103.2
• deCONZ add-on 4.1

Logs:
2019-12-20 12:23:51 ERROR (MainThread) [homeassistant.components.deconz] Error connecting to deCONZ gateway at x.x.x.x
2019-12-20 12:24:04 ERROR (MainThread) [homeassistant.core] Error doing job: Unclosed connection

[roaringbits]

Same here. I've only got the VNC port left to configure.

[pboschi]

Same here... Devices are found on Phoscon GW web page, but entities are missing from HA.

How to revert to previous version?

[roaringbits]

Reinstalling the addon did not help, but I fixed it by simple removing the Integration and adding it back in. Solved the issues for me.

[newlund]

Same here...

Conbee 2,
VM running debian buster,
Hass.io 0.103.2,
deCONZ add-on 4.1.

Devices are still in the Phoscon GW but no entities in HA.

Error connecting to deCONZ gateway at 192.168.0.31
14:04 components/deconz/gateway.py (ERROR) - Meddelandet inträffade först 13:56 och har hänt 10 gånger
Config entry for deconz not ready yet. Retrying in 40 seconds.
13:57 config_entries.py (WARNING)

[pboschi]

Changed password and readded integration, entities are now back... but various device names are changed. For some reason, some Xiaomi multi sensors now have temperature and humidity values swapped.

[frenck]

@VivantSenior @jan-dewit The add-on sends a reconfiguration discovery notification to Home Assistant to adjust the settings. However, if in the past a manual configuration flow was used, that probably won't work in that case. The only solution at that point is by removing the integration -> restart Home Assistant -> restart the add-on. After that, a integration should be discovered and visible on the integration screen.

@pboschi @roaringbits, you guys already did the above 👍
@pboschi The naming of devices is something the integration handles and is not in control for the add-on. If that is a reproducible issue, please be sure to raise an issue on the Home Assistant repository, so the maintainer of the integration can look into it.

@hellcry37 The add-on is now available over Ingress and does not use ports anymore. This makes the add-on available from the Home Assistant frontend, no matter how or where you log into it. For example, when using a remote UI, the add-on can be used, without any port forwarding. Most add-ons do this already, this one is now following.

[newlund]

@frenck The add-on sends a reconfiguration discovery notification to Home Assistant to adjust the settings. However, if in the past a manual configuration flow was used, that probably won't work in that case. The only solution at that point is by removing the integration -> restart Home Assistant -> restart the add-on. After that, a integration should be discovered and visible on the integration screen.

Thanks for the workaround!
But I'm a little worried that I would need to rename all entities again if I remove and re-add the integration. Or will the naming be preserved?

[pboschi]

To my understanding, the issue is only with the Xiaomi multi-sensors and is due to the fact that they get named sensor.[name], sensor.[name]_2 and sensor.[name]_3, to report the three values of pressure, humidity and temperature. Every time you re-integrate Deconz (which is necessary after the 4.1 update), the order changes, so their entire history turn into garbage.

[frenck]

@newlund That is, unfortunately, likely to happen indeed. There is a workaround for that (but requires to be a bit of a hero, by editing the storage manually so the integration uses core-deconz as the hostname to connect to instead). But honestly, if you run into this, the integration isn't really set up correctly. Personally I would bite that bullet once, so in future changes, Home Assistant will be able to pick up automatically.

@pboschi The good news is that the add-on now ensures that it uses a mechanism that allows for automatic updates of settings in the future. So re-adding the integration should not be needed anymore.

[newlund]

@frenck Yes, I will do it as soon as I have time. Thanks!

As a temporary solution, should I change the "host" parameter to "core-deconz" in /usr/share/hassio/homeassistant/.storage/core.config_entries and restart HA?

{ "connection_class": "local_push", "data": { "api_key": "xxxxxxxxxx", "bridgeid": "00212EFFFF0406F2", "host": "192.168.0.31", "port": 40850, "uuid": "9ab523fd-7ec8-4d61-a992-9614eb7a6754" }, "domain": "deconz", "entry_id": "b2d089ec8ef94a239551f94a1430005f", "options": { "allow_clip_sensor": false, "allow_deconz_groups": true, "master": true }, "source": "user", "system_options": { "disable_new_entities": false }, "title": "deCONZ-00212EFFFF0406F2", "version": 1 },

[frenck]

@newlund Yes, that might work, however, you've just published your deCONZ api_key (just so you know).

[newlund]

@frenck Ooops... :)
However, my instance is not available from the internet (only via nabu casa remote ui). Should not pose any risk. Right?

[frenck]

I guess you are fine in that case :wink:

[newlund]

Do I have to reinstall the deconz addon in order to change the API key?

[frenck]

Yep, however, with the 4.x add-on, it is no longer possible to reach the API from the outside world. I would not worry to much about it.

[newlund]

@frenck Ok, thank you very much! 🙂

[intelroman]

Hello,

deCONZ 4.1 is missing webport 40850 as docker .
Can't do anymore api's

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2f1cc87615f1 homeassistant/armhf-addon-deconz:4.1 "/run.sh" 11 minutes ago Up 11 minutes addon_core_deconz
37ac9e5d2e51 hassioaddons/mqtt:1.1.0 "/init" 22 hours ago Up 22 hours 0.0.0.0:1883-1884->1883-1884/tcp, 0.0.0.0:4883-4884->4883-4884/tcp addon_a0d7b954_mqtt
928e38084d62 homeassistant/armv7-addon-configurator:4.1 "/run.sh" 30 hours ago Up 30 hours addon_core_configurator
541424ca1f3c homeassistant/armv7-hassio-dns:1 "coredns -conf /conf…" 36 hours ago Up 36 hours hassio_dns
68593b65105e homeassistant/armv7-hassio-supervisor "/bin/entry.sh pytho…" 36 hours ago Up 36 hours hassos_supervisor
87b6f67a0ab6 homeassistant/armv7-addon-ssh:7.1 "/run.sh" 6 days ago Up 6 days 0.0.0.0:22->22/tcp addon_core_ssh
db02b37dc5e7 hassioaddons/grafana-armv7:3.0.2 "/init" 2 weeks ago Up 2 weeks addon_a0d7b954_grafana
64307b482bb5 hassioaddons/influxdb-armv7:3.4.1 "/init" 2 weeks ago Up 2 weeks 0.0.0.0:8086->8086/tcp addon_a0d7b954_influxdb
c6ab762fd625 homeassistant/raspberrypi3-homeassistant:0.101.3 "/bin/entry.sh pytho…" 4 weeks ago Up 18 minutes homeassistant

[intelroman]

Do I have to reinstall the deconz addon in order to change the API key?

Do I have to reinstall the deconz addon in order to change the API key?

You can remove the api :D check the link below
https://dresden-elektronik.github.io/deconz-rest-doc/configuration/#aquireapikey

[frenck]

@intelroman It is not missing, is has been removed. Please read up a bit on this issue, as it has been mentioned before. Ingress support has been added.

[intelroman]

@intelroman It is not missing, is has been removed. Please read up a bit on this issue, as it has been mentioned before. Ingress support has been added.

Yes Frank is ok .. but i need for external app to do API .. do you recommend another way to do API ?

I'm trying to investigate now a Smart plug that the system see as light .. and i need api to get the power meter .

[frenck]

@intelroman Home Assistant reads up the data from deCONZ, you might be able to use the Home Assistant API's for that?

[intelroman]

I'm not sure ..i don't see the data in the HASSIO .. trying to do the manual the DNAT but is still not working :D
... is funny .
The ports are still open in the container:
root@core-deconz:/# ss -l
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
u_seq LISTEN 0 0 /run/udev/control 8628395 * 0
tcp LISTEN 0 128 :8099 *:
tcp LISTEN 0 50 :8081 *:
tcp LISTEN 0 50 :40850 *:
tcp LISTEN 0 128 127.0.0.11:34427 :

i killing me why DNAT is not working
iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 1051 packets, 245K bytes)
pkts bytes target prot opt in out source destination
327K 107M DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 747 packets, 227K bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 2019 packets, 165K bytes)
pkts bytes target prot opt in out source destination
13606 816K DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 2061 packets, 167K bytes)
pkts bytes target prot opt in out source destination
3125 190K MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
146K 8778K MASQUERADE all -- * !hassio 172.30.32.0/23 0.0.0.0/0
0 0 MASQUERADE tcp -- * * 172.30.33.2 172.30.33.2 tcp dpt:8086
0 0 MASQUERADE tcp -- * * 172.30.33.1 172.30.33.1 tcp dpt:22
0 0 MASQUERADE tcp -- * * 172.30.33.4 172.30.33.4 tcp dpt:4884
0 0 MASQUERADE tcp -- * * 172.30.33.4 172.30.33.4 tcp dpt:4883
0 0 MASQUERADE tcp -- * * 172.30.33.4 172.30.33.4 tcp dpt:1884
0 0 MASQUERADE tcp -- * * 172.30.33.4 172.30.33.4 tcp dpt:1883

Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
2 120 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
1631 98220 RETURN all -- hassio * 0.0.0.0/0 0.0.0.0/0
14300 858K DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8086 to:172.30.33.2:8086
2 104 DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 to:172.30.33.1:22
0 0 DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4884 to:172.30.33.4:4884
0 0 DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4883 to:172.30.33.4:4883
2 104 DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1884 to:172.30.33.4:1884
65 3284 DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1883 to:172.30.33.4:1883
0 0 DNAT tcp -- !hassio * 0.0.0.0/0 0.0.0.0/0 tcp dpt:40850 to:172.30.33.5:40850

[thecode]

I'm using Deconz with Node-Red, I have about 70 automations in Node-Red, after updating to 4.1 ports are not exposed anymore and I had to revert to 3.9.
Is it possible to expose the ports on the docker just leave them with no default mapping value (same as for the VNC port) so that it will still be possible to map them?
I only use some of the entities in HA, most of them are only in Node-Red and converting them to use HA entities will be a huge mess.

I love your work, and like the ingress support, Please think of allowing backward compatibility with older systems which use the Deconz via API directly (I use "node-red-contrib-deconz")
Thanks.

[frenck]

@intelroman Not sure what you are trying to achieve, but that is definitely not supported.

@thecode If you use the Hass.io Node-RED add-on, that add-on is still able to use it, you have to change the hostname to connect to. It used to be localhost or 127.0.0.1. If you change that to core-deconz as the host to connect to in Node-RED will be able to connect.
Oh, you could have asked instead of jumping into conclusions 😉

[intelroman]

@intelroman Not sure what you are trying to achieve, but that is definitely not supported.

It is simple i use deCONZ api to gather some data like:
https://github.com/intelroman/hassio_scripts/

[thecode]

Thanks, but right now I'm too afraid to test that, was real mess when I upgraded this morning. All house stopped working. I usually do that verty carefully but I didn't think something will break.
I will try that later, but I still recommend finding a way for having direct access also.
For example I access data from motion Eye on another machine (not from the HA machine) in motionEye there is an option to map the port (with a text written that it is not necessary for ingress support).

I apologize if it looked like I jumped into conclusions, I've asked on few places and no one know the answer....

[frenck]

@intelroman As I already wrote, it is not possible to access the API externally. It is available from within the Hass.io ecosystem on the hostname core-deconz, however, if the code is run inside Hass.io (e.g., using an SSH add-on, or things like AppDaemon). Other than that is not supported at this point and is not something we can help you with.

@thecode Not all add-ons are able to support both. deCONZ can't support it, because it breaks Ingress. There is no option for a choice there. The fact you are scared to try is not something I can do anything about I'm afraid. Besides recommending a snapshot and just try, if it fails for you, you can always fall back onto the snapshot created.

[thecode]

Well I can't do anything about it, but for me having ingress support and losing external access to Deconz is a drawback. If I had to choose I will prefer external access over ingress support.
I have 2 machines for home automation and had plans to move some of the automations to another machine, without having them being able to access one another this is something brakes my plans.

[frenck]

@thecode I'm sorry to hear that, but at the same time, I'm afraid it is an impossible task to make everybody happy. For your case, there is a solution, which I already gave in a previous comment/response directed at you.

[thecode]

Well this only helps for one machine for now. I hope you change your mind in the future and then I'm sure a solution can be found, maybe asking Deconz team to add support for it in some way (like another ports or something) or maybe even have an option to configure the hassio addon to disable/enable ingress or use port access.

Merry christmas and I hope new year will bring new stuff :-)

[repvik]

I keep getting these:

2019-12-20 23:04:18 WARNING (MainThread) [homeassistant.components.websocket_api.http.connection.1803347888] Disconnected: Did not receive auth message within 10 seconds
2019-12-20 23:04:18 WARNING (MainThread) [homeassistant.components.websocket_api.http.connection.1787666608] Disconnected: Did not receive auth message within 10 seconds
2019-12-20 23:04:33 WARNING (MainThread) [homeassistant.components.websocket_api.http.connection.1817278832] Disconnected: Did not receive auth message within 10 seconds
2019-12-20 23:04:33 WARNING (MainThread) [homeassistant.components.websocket_api.http.connection.1791389968] Disconnected: Did not receive auth message within 10 seconds
2019-12-20 23:04:33 WARNING (MainThread) [homeassistant.components.websocket_api.http.connection.1786928336] Disconnected: Did not receive auth message within 10 seconds

I can read the deconz log, and it seems to work perfectly fine, yet I get no devices in the integration.
Also, I cannot open the gateway settings page via ingress.

How do I fix this?

[repvik]

I've removed integration and add-on, rebooted, added add-on, rebooted, added integration. I get the standard daylight sensor and the gateway itself. I still can't access the gateway settings page.
I can't control any lights in the house, and thermostat functions no longer work since all my temp sensors are offline.

An oddity is that from one browser, I get all my lights, switches and sensors in the deconz interface. If I open a different browser, it appears as if the GW has been reset. What is going on? I have more than 50 devices paired, and re-pairing them would be a major pain.

Why can't the API be available from an external port as an option? HomePanel lets me access it from both external port and ingress, so this seems like a completely arbitrary limitation.

[branitelj91]

Ok. To continue from my previous.
Accessing deconz api is now disabled for any given reason. I am, (and I can see I'am not only one), not happy but.., whatever.
The access is possible through other solution, however I cannot find it. It might be good idea to put it as standalone topic and with relevant keywords. It might help. Also, I'd be more than happy if I could have link to this sollutions frenck

[frenck]

@repvik: If I open a different browser, it appears as if the GW has been reset.

That sounds like a browser caching issue? From my experience with deCONZ, is leaving the browser open on the device list page (just before login), it will rediscover and find out by itself things changed.

@repvik: Why can't the API be available from an external port as an option?

This question has now be answered in this issue a couple of times now. I would appreciate reading before asking, it saves us all quite some time.

@branitelj91: Accessing deconz api is now disabled for any given reason

The reasoning for this has been provided a couple of times now.

@branitelj91: The access is possible through other solution, however I cannot find it

You didn't describe your use case, so I'm not sure how to answer or help you with that.

[daanvb]

I have a Pi4 with my ConBee plugged and the DeConz addon running (currently back to 3.9). I then have a VM on the same network that handles all the heavy lifting and automations that links to the DeConz via the integrations page.

With this new update does that mean that this now won’t work?

[frenck]

@daanvb That indeed won't work. dresden electronics offer SD card images for those purposes: https://phoscon.de/en/conbee/sdcard.

[daanvb]

@frenck so I guess I can’t then run Hassio as well as this Phoscon image?

I have a Zwave stick and BT tracking on the Pi as well so I need hassio.

This will completely destroy my system if this can’t be done...

[frenck]

@daanvb It sounds like you are mixing up multiple Hass.io ecosystems. While that works in some cases, it is not something that is supported.

[fantomnotabene]

I have modified (soldered) some of my Xiaomi motion sensors to reset after 5 seconds, and used API to change their configs accordingly

curl -H 'Content-Type: application/json' -X PUT -d '{"duration": 5}' http://hassio_ip:deconz_addon_api_port/api/api_key_here/sensors/motion_sensor_id_here/config

So I fully agree with the question

Why can't the API be available from an external port as an option?

I think user should have an option to use the API

As another example, I have solved the problem when both Aqara cube sensors was with the same identifier - set another name to one of the sensors

This is a really useful feature

[frenck]

ℹ️ I've just done some major updates to our add-on documentation:

https://github.com/home-assistant/hassio-addons/blob/master/deconz/README.md

Added sections:

• Using the deCONZ/Phoscon API with another add-on _(e.g, Node-RED)_
• My gateway shows up in Home Assistant with ID 0000000000000000
• My integration shows now devices after upgrading to 4.x

Also, the Known issues and limitations has been extended to cover all current ones.

[frenck]

@fantomnotabene You can still use curl against the API to change settings. Using:

curl -H 'Content-Type: application/json' -X PUT -d '{"duration": 5}' http://core-deconz:40850/api/api_key_here/sensors/motion_sensor_id_here/config

As long as the command is issued from within the Hass.io ecosystem (e.g., using the SSH add-on). For more information see: Using the deCONZ/Phoscon API with another add-on

I think user should have an option to use the API

That has been explained a couple of times on why it is not an option. It is also listed in the "known issues and limitations" section of the add-on documentation.

[repvik]

This question has now be answered in this issue a couple of times now. I would appreciate reading before asking, it saves us all quite some time.

No, it hasn't been answered. Here's what you've said:

The add-on is now available over Ingress and does not use ports anymore.

Yep, however, with the 4.x add-on, it is no longer possible to reach the API from the outside world. I would not worry to much about it.

As I already wrote, it is not possible to access the API externally. [...] Other than that is not supported at this point and is not something we can help you with.

Not all add-ons are able to support both. deCONZ can't support it, because it breaks Ingress.

"Why can't it be added as an option when other addons support both" isn't answered by the first three. The fourth was added after I asked my question, and doesn't really answer it either.

My question is _why_ does it break ingress for deCONZ while other add-ons have no problems combining them? Is this something that can be fixed, or is this permanent?

[frenck]

@repvik Sorry for that, I have been write answers in many locations yesterday, I was under the impression it was in this location as well.

If the port is opened up, the frontend will discover the same device on multiple addresses, causing it to pick up the last address discovered, which is not accessible for Ingress. Losing access to your device altogether.

This is permanent at this moment.

[repvik]

@frenck Aha. Makes sense.

[branitelj91]

@frenck
Sorry for not providing enough info.
So here is explanation:
I have made HASSIO testing platform on one RPI3 board to check how to migrate to Home Assistant from other platform, (Hubitat elevation, as this is closed platform to the degree and I experienced significant lag in activating certain dimmers and lights).
So, platform consists of three lights, (2 philips, and one sylvania), with three remotes, (Ikea tradfri 5 button puck, osram lightify 4 buttons square remote and lately added Sunricher remote with 8 scenes and 4 groups).
Here is some of my toughts,

• First wanted to check how I can cope with Node-Red,
• Secondly wanted to check sustainability of such ecosystem
• Thirdly, I wanted to check platform interoperability.
  The latter one is one of my most concerns, as I had Domoticz and OpenHab linked to phoscan. The idea behind is to make use of both, three, four ecosystems where each one has it's strong and wek points. I have experienced most good automations with Home Assistant, (and Node-Red), a some of best panel integrations with Domoticz, and some of most versatile device mappings with OpenHab.
  I know, it may sound some "odd"-ish to use three platforms instead only one propperly configured, but give me a credit into this as I am still learning of these systems, as well as I am determining what I will have use of most.
  So far, HA looks most promising, and as panel app I might have either Domoticz, or anything else given.

Regarding to it, I have need of connecting domoticz to Hassio or deCONZ to be able to learn further.

What, I am asking now, is, how to connect those two again.
I know there is at least one workarround, and that is also one solution as well. The example would be a different RPI with Deconz on as standallone, while HA will run in the other box, and Domoticz will be third one. But, that looks bit clumsy, and I am striving to reduce it to two boxes.

And as addition to it, I would be more than happy if I could operate deCONZ entities directly in Node-Red instead their instances in HA.

I hope, I've cleared out little bit background of my idea.

Thank You for any assistance

[frenck]

@branitelj91 I can see how in a learning/orientation phase this is confusing and problematic.

For the Node-RED part, the Hass.io Add-on for Node-RED is able to connect and this is now documented in the documentation of the deCONZ add-on as well.

As stated before, external connections to deCONZ cannot be made anymore. So you will not be able to hook it up with Domoticz or OpenHAB. dresden eletroniks offers an SD card image for these purposes: https://www.phoscon.de/en/conbee2/sdcard

[intelroman]

Hello ,

I didn't try yet .. but is the Nginx ssl proxy .. can be configured as
proxy for deconz 😁😂

On Sat 21 Dec 2019, 12:51 Franck Nijhof, [e-mail] wrote:

@branitelj91 https://github.com/branitelj91 I can see how in a
learning/orientation phase this is confusing and problematic.

For the Node-RED part, the Hass.io Add-on for Node-RED is able to connect
and this is now documented in the documentation of the deCONZ add-on as
well.

As stated before, external connections to deCONZ cannot be made anymore.
So you will not be able to hook it up with Domoticz or OpenHAB. dresden
eletroniks offers an SD card image for these purposes:
https://www.phoscon.de/en/conbee2/sdcard

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/hassio-addons/issues/928?email_source=notifications&email_token=ABDO54KSDQMPKV7R3YCHPMTQZYGNVA5CNFSM4J54PZHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHO3SHA#issuecomment-568178972,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDO54MXRVI6GVAI2LW6INLQZYGNVANCNFSM4J54PZHA
.

[frenck]

@intelroman Yes that might work, but might cause major issues as well. It is definitely not supported.

[intelroman]

I think people it will try to learn . So they need ways to work around it.
Like me ... i'm thinking for solutions to learn more .. and do stuff .
If it will be a payed app i will understand that i need to comply with the
security concerns .

On Sat 21 Dec 2019, 12:56 Franck Nijhof, [e-mail] wrote:

@intelroman https://github.com/intelroman Yes that might work, but
cause major issues as well. It is definitely not supported.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/hassio-addons/issues/928?email_source=notifications&email_token=ABDO54LO4U6ZPLX5MZ35YL3QZYHAVA5CNFSM4J54PZHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHO3UXI#issuecomment-568179293,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDO54PCRLABP7DHX6K2PEDQZYHAVANCNFSM4J54PZHA
.

[branitelj91]

@frenck
small update:

Followed Your last update on walktrohugh, I was able to restore Node-Red usability and have restored all configuration. Which I am grateful for.
As for rest I understand, but, I would more than happy if I could see some port connection to this inside HA ecosystem. For sure, there would be some usability of this idea of mine.

[frenck]

I could see some port connection to this inside HA ecosystem

As I have mentioned directly to you as well a couple of times. That is not supported and not likely to happen because of technical limitations.

[frenck]

@branitelj91: i need to comply with the security concerns

I never stated it was because of security concerns. To avoid misconception around this, let me make this clear one more time.

Direct access has been removed due to technical issues regarding auto-discovery in the Phoscon/deCONZ frontend. This is permanently at this point.

[daanvb]

This sucks so much. My system was really evolving and working great and now its all ruined by this update. I'm really frustrated.

[intelroman]

And like me no backup before update :D

On Sat, Dec 21, 2019 at 2:33 PM daanvb [e-mail] wrote:

This sucks so much. My system was really evolving and working great and
now its all ruined by this update. I'm really frustrated.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/hassio-addons/issues/928?email_source=notifications&email_token=ABDO54LOQLMRSX4DU5K7E2LQZYSLDA5CNFSM4J54PZHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHO5DMY#issuecomment-568185267,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDO54I2NUQ7SPTECKW7IKTQZYSLDANCNFSM4J54PZHA
.

[repvik]

All my paired devices disappeared after upgrade, and deconz appeared to have been reset. Could the add-on take a backup before upgrade?

[BoneheadFraggle]

I experience strange behaviour with the integration after upgrade.
At this point I think I have tried everything, following the instructions.

Upgrading, uninstalling, deleting integrations, restarts, re-installing the new version clean, restoring the setup. And like 25 restarts of HA and a couple of the host just in case.

After a clean install everything seems to work and after a restart HA finds the new integration which is successful. But no devices are found. If I enter the integration all entities are there, but they are not indicated one first integration page.

I can no add another integration (manually) as you did before, which works as excpected. The devices are now indicated on the integration page, but I also have two integrations.

No matter how I try I always end up with an integration that doesn't work.

I can restore the 3.9 add-in, re-create the integration and then everything works again.

Any suggestions?

[thecode]

@frenck Considering all the issues people have, maybe it is better to rollback the changes to 3.9 (with a new version 4.2 and only update Phoscon app to isolate the sensors pairing/missing problems and later on add the ingress?

[PalmMx]

I was on HA 0.102.3 and addon 3.9 and everything worked just fine. Then I updated the addon to 4.1 (big mistake!) and everything died. I have followed the instructions without success. I updated HA to 0.103.3 and tried again without success. I don't know how many times today I have uninstalled, reinstalled, restated the addon, removed and added the integration and not least restarted HA.

I get the correct name on the integration (not the 0000000000 ID). But the integration is empty, no entities. My HA log says the following:
"Error connecting with deCONZ gateway"
and
"Invalid key for deCONZ at core-deconz.local.hass.io"

I have restored a backup in Phoscon so all my entities there seems to be alive, but they won't communicate with HA.
Any ideas?
Or is my best shot right now to restore HA 0.102.3 and addon 3.9?

(Running HA with a ConBee 1 on a RPi 3. The ConBee has had the latest firmware since way back.)

[SiliconAvatar]

@PalmMx

Get the deCONZ addon working (IE restore from backup) and once you can see all lights (etc) in the webui, delete the deconz integration from HA. Then reboot HA. Once HA is back up, restart the addon. Ignore any discovery messages from HA. Once the addon is back up, go to the gateway menu and under advanced, click "Authenticate App". Then in HA go to the integrations page and manually search for and add the deCONZ integration. That should work to add them all back in. (not all of these steps may be needed, but seems to work the most often)

Note, these entities are likely to all show up as unavailable and stop working again if you restart HA, in which case you'll have to do this again.

[mihalski]

I have a Pi4 with my ConBee plugged and the DeConz addon running (currently back to 3.9). I then have a VM on the same network that handles all the heavy lifting and automations that links to the DeConz via the integrations page.

With this new update does that mean that this now won’t work?

Options include trying to use the home assistant eventstream for the zigbee devices or a remote USB adapter (USB over cat5). I'm opting for the latter solution. I also have Cassio with zwave/zigbee located centrally with a Cassio vm on a server in the garage.

[tyk0s]

@PalmMx

Get the deCONZ addon working (IE restore from backup) and once you can see all lights (etc) in the webui, delete the deconz integration from HA. Then reboot HA. Once HA is back up, restart the addon. Ignore any discovery messages from HA. Once the addon is back up, go to the gateway menu and under advanced, click "Authenticate App". Then in HA go to the integrations page and manually search for and add the deCONZ integration. That should work to add them all back in. (not all of these steps may be needed, but seems to work the most often)

Note, these entities are likely to all show up as unavailable and stop working again if you restart HA, in which case you'll have to do this again.

this works for me.. remove integration -> restart HA -> goto deconz and "authenticate app" -> add integration manually in HA, entitys shows up and everything is fine.. 1 restart of HA works fine, the 2nd all entitys is removed under integration and i get "Invalid key for deCONZ at core-deconz" / "Error connecting with deCONZ gateway" logged.. and i'll have to redo the process..

[VivantSenior]

@VivantSenior @jan-dewit The add-on sends a reconfiguration discovery notification to Home Assistant to adjust the settings. However, if in the past a manual configuration flow was used, that probably won't work in that case. The only solution at that point is by removing the integration -> restart Home Assistant -> restart the add-on. After that, a integration should be discovered and visible on the integration screen.

I can confirm that this workaround works but with a small caveat. I had to remove my automations which uses deconz_events (from the automations.yaml because they were invisible from the GUI) and re-add them again.

[mihalski]

I can confirm that this workaround works but with a small caveat. I had to remove my automations which uses deconz_events (from the automations.yaml because they were invisible from the GUI) and re-add them again.

This is also true of device triggers since upon re-integration the device_id is randomly generated. I make backups of the old device registry and after integration search and replace the old device_id for the new device_id in the automations.yaml to avoid having to rebuild everything.

[frenck]

Adding it in manual with the core-deconz will cause issues with updates for sure, I would definitely not recommend to use that workaround.

I currently find the reports on the invalid API token interesting, since that is a new kid on the block, which I will be investigating today.

[branitelj91]

So,
Today I did workaround. Installed CONBEE II on another machine, (phycal, but should work in VM as well). After I restored all in that machine, HA almost instantly informed me on new zigbee entity and propperly informed me about name. Then, it restored all my names properly and, also, all mapped fluently in DOMOTICZ.
Problem solved

[thecode]

So,
Today I did workaround. Installed CONBEE II on another machine, (phycal, but should work in VM as well). After I restored all in that machine, HA almost instantly informed me on new zigbee entity and propperly informed me about name. Then, it restored all my names properly and, also, all mapped fluently in DOMOTICZ.
Problem solved

Can you share more details about your solution?
Which image you installed? What is the machine type (RPI, PC, ?)
Thanks

[frenck]

@thecode He just installed the ConBee software on a different computer, as made available by Dresden directly and is unrelated to the add-on. Please discuss that somewhere else, to try to keep this issue clean (as far as possible at this point).

[intelroman]

I know people will say this is not supported.
However i have found a way to do this.

Install core-ssh

Edit sshd_config file to allow port forwarding.

core-ssh:~# cat /etc/ssh/sshd_config | grep Tcp
AllowTcpForwarding yes

core-ssh:~# ps -ef
PID USER TIME COMMAND
1 root 0:29 /dev/init -- /run.sh
6 root 0:00 /usr/sbin/sshd -D -e
72 root 0:01 sshd: root@pts/0
74 root 0:00 -bash
173 root 0:00 sshd: root@pts/1
175 root 0:00 -bash
225 root 0:00 /bin/bash
240 root 0:00 ps -ef

This way you enable SSH port forwarding.

I'm using for what i do
https://sshtunnel.readthedocs.io/en/latest/

if you need you can use ssh tunneling form your linux box.

My example to access the api.

!/usr/src/Python-3.7.3/python

from sshtunnel import SSHTunnelForwarder
from pprint import pprint as pp
from requests import get
import json, re, os, time
from influxdb import InfluxDBClient

conf = {
'url': 'http://127.0.0.1:40850/api/',
'auth' : '988112a4e198cc1211',
'ifx' : {'host': '192.168.100.244', 'port': 8086, 'usr_': 'admin', 'pass_': 'PaSsWoRd', 'DB_': 'deconz_data'}
}

client = InfluxDBClient(host=conf['ifx']['host'], port=conf['ifx']['port'], username=conf['ifx']['usr_'], password=conf['ifx']['pass_'])
client.switch_database(conf['ifx']['DB_'])
api_ = 'lights'

def url_ (api):
url = conf['url']+conf['auth']+'/'+api
headers = {'content-type': 'application/json'}
response = get(url, headers=headers)
return response

with SSHTunnelForwarder(
('192.168.100.244', 22),
ssh_username="root",
ssh_password="StrongPassword:P",
remote_bind_address=('core-deconz', 40850),
local_bind_address=('0.0.0.0', 40850)) as tunnel:
pp (url_(api_).text)
data = json.loads(url_(api_).text)

or bash:
https://www.ssh.com/ssh/tunneling/example

[frenck]

@intelroman Last warning: please keep things on-topic here. You know it is unsupported, still, you post massively, unrelated stuff that does not belong to this issue.

[frenck]

ℹ️ Small update from my end and day. I've been digging in and setting up all kinds of setups to look further into this issue.

I am making progress in the following area's:

• Limiting the number of random "discover" requests to prevent IP banning: This part is going pretty well, trying to work out some last ones.
• Preventing incorrect discovery data: This is actually pretty much solved at this point. However, I want to make sure it works consistently. In my dev setup, the correct gateway is now available instantly from the web interface.
• I've up until now not been able to reproduce the incorrect API token from the HA integration end.

[branitelj91]

@frenck
It is unrelated to addon, but I gave a workaround since You mentioned that outbond access to deCONZ is disabled in this version. For us who are still learning the system and others that have particular idea of what they want in these with multiple softwares accesing it I had this sollution checked for and published here. Other than this, I understand You and others made effort in creating these pieces of software, (meaning HA and plugins), which we all have to respect as this makes our lifes easier, but as well we have our needs where we find all these apps usable in more or less ammount. Having both of these in mind, I have made this small step. I hope this does not make a issue in this topic.

[frenck]

@branitelj91 Please do so on another topic in e.g., the community forums, as this repo is about the add-on. Secondly, the solution provided the hidden response is a really bad one since it installs things in a non-persistent environment. So again, feel free to discuss non-addon related workaround in a topic on the community.

[frenck]

I'm locking down this conversation for now and will continue to provide status updates.

I believe everything has been said at this point and some workarounds were provided by the community here (which is not an endorsement from my end).

If you feel like you can provide actual solutions, feel free to open up a PR or contact me on Discord.

[frenck]

ℹ️ Status update time. This is what I currently I'm running and testing to death at this moment.

• I've managed to patch out excessive probing that causes IP bans in Home Assistant.
• Fixed an issue on Home Assistant side of things, that caused the integration to be updated with incorrect data (not effected everybody). This is released with version 0.103.4 of Home Assistant.
• Fixed an issue on Home Assistant side of things, where the add-on was unable to communicate all settings to the integration. This is fixed with the 0.103.4 release of Home Assistant as well.
• I've figured out an alternative way on how Phoscon handles discovery, which now instantly makes the login available when opening the app in ingress, and removes the additional weird gateway found.
• Because of the above, it seems like I'm able to open up direct access to both Phoscon, the API and the Hue emulation without issues.
• WebSocket support for Ingress.

I really want to spend some more time tomorrow on making sure everything works as expected.

_So to be clear, for all of the above, it is still unsure if that will be released or works stable._

[hellcry37]

Even if I don't appreciate all the bugs, I do appreciate nice status updates. Thanks

[frenck]

ℹ️ Status update again.

During extensive testing I've found and fixed the following today:

• If the session expires to Phoscon (e.g, clearing cookies) the redirect to the login page will now work.
• Fixed a case where Phoscon will try to fetch a file called developer.js that didn't exists.

I've tortured the add-on today and even went in the running container and destroyed things and it was able to recover.

Even resetting the gateway completely, including de-install/install the add-on, pairing devices again, made HA pick up the changes (on HA 0.103.4) automatically, even keeping my device based automation in a working state.

Things are looking good from my end and I ran out of senario's to test. So personally, I'm 100% happy with the result. I'm going to update to documentation and start working towards opening a pull request to update the add-on.

[maxalbani]

@PalmMx
Fai funzionare l'addon deCONZ (ripristino IE dal backup) e una volta che puoi vedere tutte le luci (ecc.) Nel webui, elimina l'integrazione di deconz da HA. Quindi riavviare HA. Dopo aver eseguito il backup di HA, riavviare il componente aggiuntivo. Ignora eventuali messaggi di individuazione da HA. Una volta eseguito il backup del componente aggiuntivo, vai al menu del gateway e in Avanzate, fai clic su "Autentica app". Quindi in HA vai alla pagina delle integrazioni e cerca manualmente e aggiungi l'integrazione deCONZ. Dovrebbe funzionare per aggiungerli tutti di nuovo. (Non tutti questi passaggi potrebbero essere necessari, ma sembra funzionare più spesso)
Nota, è probabile che tutte queste entità vengano visualizzate come non disponibili e smettano di funzionare di nuovo se riavvii HA, nel qual caso dovrai farlo di nuovo.

questo funziona per me .. rimuovi integrazione -> riavvia HA -> vai a deconzare e "autentica app" -> aggiungi l'integrazione manualmente in HA, le entità si presentano e tutto va bene .. 1 riavvio di HA funziona bene, il 2 ° tutte le entità è rimosso sotto integrazione e ottengo "Chiave non valida per deCONZ al core-deconz" / "Errore di connessione con il gateway deCONZ" registrato .. e dovrò ripetere il processo ..

I have updated to the latest versions but my system still behaves exactly as described above.

Home Assistant: 0.103.4
Deconz addon: 4.1

After configuring the integration, at the first restart of HA everything is ok, at the second no entity available.
And I have to start again!

[jdenver13]

Thanks for the update frenck, the deconz integration has now updated and works well. Still have issues with Safari accessing the deconz Gateway, it just will not display the page, I have to use chrome ATM.

[frenck]

@jdenver13 That issue is tracked in #943

[frenck]

The PR for the 5.0 release is up and running. However, I've found another case that can break the communication between the integration and the add-on (involving backup/restore from deCONZ itself).

Since I have a reproduction on that now, I'm going to see if I can come up with a fix for that as well.

Therefore, I've marked/blocked the 5.0 release at this point.

[jamesking420]

I understand why you would like to wait to resolve all the issues in the next release but for those of us having the problem (especially the one where entities disappear on a reset) it would very useful for you to release 5.0 as soon as possible. If you could consider doing that, it would be great.

[frenck]

@jamesking420 Well actually, the last part you describe is the part that I wanted to improve, so in that case, you should be happy I pulled the brake on that 😉

The good news is though, I've addressed it, the PR is open and ready to go.

[jamesking420]

Haha - OK, thanks. :-) Thanks for all your work on this, looks like it has been a challenging update - but hopefully ends up with us all in a better place after it's working for everyone!

[frenck]

@jamesking420 Yes it was. I've actually set up a lot of testing situations with Home Assistant, and actually hardware to simulate what happened, how upgrade paths go. I even went in via the API and the actually internal database breaking stuff and see if it recovers. TLDR; I've tested tons of real-world cases the past 3 days to ensure it all works flawlessly.

At this point, I'm really convinced this release will work.

[frenck]

:tada: Add-on v5.0 is available right now.

I truly hope it will fix everybody's issues and wishes.

If it does not appear in your Hass.io panel yet, please go to Hass.io add-on store and hit the refresh icon in the top right. Next refresh your browser and the upgrade appears.

Please note: It is advised to use Home Assistant 0.103.4 or newer!

The add-on has now auto-healing capabilities (In terms of the connection between Home Assistant and the add-on). So in case of any issues, restart the add-on, wait at least 1 minute, restart Home Assistant and after that, everything should be normal again.

Sorry about the inconvenience and thanks for the patience.

Merry Christmas 🎄

---

Changelog:

• Fix additional gateway visible on Phoscon login on Ingress
• Fix Phoscon device scanning/probing triggering IP bans
• Fix redirect to login page when Phoscon session expires
• Fix incorrect manifest request from Phoscon frontend
• Fix and improve API key handling with Hass.io discovery
• Change Hass.io discovery to use add-on IP instead of hostname
• Improve Phoscon discovery to work different and faster with Ingress
• Add Websocket support to Ingress for instant and snappy UI updates
• Re-instate direct access capabilities to the Phoscon/deCONZ API

[mihalski]

You're amazing @frenck

Does this mean the API now allows remote integrations from one Hassio to another + Deconz?
If so is remote discovery supported or does the Deconz host have to be manually specified?

I hope you have a wonderful Christmas!

[frenck]

@mihalski

Does this mean the API now allows remote integrations from one Hassio to another + Deconz?

Yes.

If so is remote discovery supported or does the Deconz host have to be manually specified?

If remote, manually (or SSDP might be able to pick it up). But the Hass.io discovery only works within the same ecosystem (which is by design).

[VivantSenior]

Changelog:

• Fix additional gateway visible on Phoscon login on Ingress

@frenck, thank you for the quick fix. Regarding the additional gateway - I can still see it on the login page but now HA doesn't ban my IP :)

[frenck]

@VivantSenior The Phoscon App stores data in cookies that might cause that. Make sure to clean your browser cache. (You can verify by using an incognito mode browser).

[VivantSenior]

@frenck I can confirm that the additional gateway has disappeared.