Deconz-rest-plugin: When switching multiple devices on one-by-one after another some devices stay off - but using unicast works

[githtz]

@morfei1 @tubalainen @MartinTerp @manup
This is the related issue to https://github.com/dresden-elektronik/deconz-rest-plugin/issues/1261#issuecomment-696198606 and suggested by https://github.com/dresden-elektronik/deconz-rest-plugin/issues/1261#issuecomment-696582270

Describe the bug

When I use HA to turn on, 17 bulbs, one-by-one (right after eachother) maybe 3 out of the 17 dosnt turn on, but HA sees them as on.
BUT, if I make a group in deconz. put all the lights in that group, and tell HA to turn on that group, there havnt yet been any issues, all lights turn on.

see https://github.com/dresden-elektronik/deconz-rest-plugin/issues/1261#issuecomment-696198606

Steps to reproduce the behavior

Turn on multiple devices one-by-one.

Expected behavior

All switched devices should be turned on.

Screenshots

I'm working on it. (please be patient)

Environment

deCONZ firmware: 26350500
Phoscon version: 2.05.81

deCONZ Logs

I'll try to get some. (please be patient)

Additional context

The devices that were switched on are shown as "on" in Phoscon and report as "on" via API but actually are not "on".
Sometimes they won't even react if switched off and on again (even multiple times).
The devices that do not react to direct commands, but if a group that the devices are part of is switched on/off, they suddenly obey.
This does not only apply to lights but all kind of switchable devices, for example the OSRAM Smart Plugs.

[morfei1]

Is there a reason you are still on Rpi_0x26370500 but not the latest Rpi_0x26370500 ?

[githtz]

Mostly because I'm lazy. Also, that was the configuration that I am sure the mentioned problems occurred with. Need to verify it's still a problem with 26370500 on 2.05.82 on my end.

[manup]

For that specific case I think the 26350500 firmware doesn't make a difference compared to 26370500. The problem is likely in the plugin for handling the tasks.

[joakimberglund]

Sounds like exactly the same issues Im having (with the same versions)

HA 0.115.2 with ZHA/ConBee II

[shoikan]

Same here. Tried 26350500 and 26350500 firmwares. Sensors also don't seem to work most of the time (yes, swapped batteries out for known good ones). In the phoscon webgui lights appear to be 'offline' (grayed out) in the Lights overview, more or less at random, regardless of whether they are turned on (and even actually lit) or not. Variety of lights are affected: Ikea Tradfri, Innr, Paulmann and Osram, so it also does not appear to be 'brand specific'.

[tubalainen]

Now I am (finally) finished with all OTAU updates of all bulbs to get a baseline and Deconz has been running for two days without restart. The lights just went off here at home (via Home Assistant automation) and all of the (approx 30) turned off as they should. Also the IKEA motion sensors are reporting and working without any issues. Same with my 10+ buttons (switches).

This is WITHOUT any double actions and delays in my automation.

My conf.
All my indoor lights are grouped under "lights.yaml" into two groups window lights and indoor lights. The majority of the lights are deconz lights and like four other lights are esphome ledstrips, they are all triggered via an home assistant automation, here´s what the action part looks like:

  - data:
      entity_id: light.window_lights
    service: light.turn_off
  - data:
      entity_id: light.indoor_lights
    service: light.turn_off

I do no longer see the issue related to this after:

• Version .82 (Marthocs docker)
• Firmware in my conbee 1 stick, 26370500
• Home Assistant Core (in docker) 0.115.2

So for me at this moment of time - it WORKS!

[morfei1]

@tubalainen I've never tested if I have problem turning them off nor I've used any delay for that actuon. I guess I won't have, because most probably it's a single command. The actual problem is, when they need to do 2 commands at a time, for example, change brightness and color. I think its Ikea specific problem, because I don't experience this with my Aqara TS lights doing the same thing. Also, phoscon scenes do have a workaround for this Ikea problem and if one uses them he won't experience this problem.

[tubalainen]

@tubalainen I've never tested if I have problem turning them off nor I've used any delay for that actuon. I guess I won't have, because most probably it's a single command. The actual problem is, when they need to do 2 commands at a time, for example, change brightness and color. I think its Ikea specific problem, because I don't experience this with my Aqara TS lights doing the same thing. Also, phoscon scenes do have a workaround for this Ikea problem and if one uses them he won't experience this problem.

EDIT -- Still having issues, below statement is no longer valid, this statement is withdrawn
Yeap, I hear you. I also have the majority of my deconz lights being IKEA lights. I had the issue you are describing earlier. Before the new version of deconz and the new firmware. At least once a day some light bulb did not react to the commands. But if repeated with a delay it was 99% success. Sometimes even a single command to a single lightbulb failed (before the last updates)

For me it seems to works now, there can be up to 15 seconds delay until some lights reacts but they do turn on/off accordingly since the last updates (and OTAU) of all my lights.,

[MartinTerp]

Still think there is some deconz magic not working correctly.
I have multiple outdoor lights, which i have grouped in different "regions", and if i turn on all groups in the same time, some groups might not turn on

[morfei1]

I experienced the same thing but this time with Aqara motion, Aqara TW and Gledapto dimmable led controller.

The led controller and the Aqara are in separate phoscon groups but in one actual room - my kitchen.

If presence is true and have to switch on my light and my led controller at the same time without delays, the led controller did not turn on the led strip, but the light turned on.

The led controller was working with the motion for about a month without a single fail. Today, I installed the light and decieded to test what gonna happen.

The Aqara lights does not have the Ikea problem not able to switch color and brightness without delay, but it seems sending command to 2 different groups at the same time to do the same thing, fail for one of the group (I think it's random) maybe the next time the light will not turn on, or maybe both groups will turn on. Sooner or later one of them will fail.

I'll join them in one group to avoid this, but note that, if you try to make some complicated automations with different groups and sending commands to them at the same time.

[githtz]

I have now upgraded my firmware and software to

deCONZ firmware: 26370500
Phoscon version: 2.05.82

I haven't had any problems with controlling any devices yet, so currently I cannot provide neither logs nor screenshots. I'll keep you posted. Ah, what logging parameters should I set here?

[tubalainen]

UPDATE
No, yesterday two lights failed to turn off. I withdraw my previous statement above.
I am still experiencing issues when turning off (only off strangely enough) multiple lights from Home Assistant.

[stale[bot]]

As there hasn't been any response in 21 days, this issue has been automatically marked as stale. At OP: Please either close this issue or keep it active It will be closed in 7 days if no further activity occurs.

[githtz]

So yesterday evening I've had this problem again. A few Ikea lights didn't switch to correct color temperature. I've enabled logging but since I don't know what log levels should be used here the logfile is growing too fast. Currently I've set

• DEBUG_INFO=2
• DEBUG_APS=2
• DEBUG_ZCL=
• DEBUG_ZDP=2
• DEBUG_OTAU=2

Any suggestions what values should be used to log relevant messages only?

[Mimiix]

@githtz As per documentation:
https://github.com/dresden-elektronik/deconz-rest-plugin/wiki/deCONZ-debug-flags

[githtz]

So, this morning I've had an osram-switch that didn't turn off. I was able to fetch the log. The device that did not behave was the Flur-Nachtlicht. The light was on at the beginning of the log. At ~08:30 a script turns the light off, but later ~08:32 it's reported as on again. This log was created with the following logging options:

• DEBUG_INFO=2
• DEBUG_APS=2
• DEBUG_ZCL=0
• DEBUG_ZDP=2
• DEBUG_OTAU=0

Flur-Nachtlicht.log

Hope this helps!

[Mimiix]

Hi @githtz

I took a look. The Osram seem to caused a 0xa7 :
Line 1703: {"log":"08:30:17:768 0x7CB03EAA0A06A25F error APSDE-DATA.confirm: 0xA7 on task\n","stream":"stdout","time":"2020-11-03T07:30:18.672397189Z"}

Which is : https://github.com/dresden-elektronik/deconz-rest-plugin/wiki/Zigbee-Error-Codes-in-the-Log#0xa7-no-application-ack-received

It simply didn't reply. So i am really wondering why that occurs.

[githtz]

Hey @Mimiix
Thanks for having a look! I have no idea what's the reason for that, but there are more entries like that in the full (around 100MB) log:

01:20:54:778 APS-DATA.confirm id: 180, status: 0xA7 NO_ACK
01:20:55:038 APS-DATA.confirm id: 179, status: 0xA7 NO_ACK
01:47:25:371 APS-DATA.confirm id: 14, status: 0xA7 NO_ACK
04:33:28:803 APS-DATA.confirm id: 2, status: 0xA7 NO_ACK
08:30:17:768 0x7CB03EAA0A06A25F error APSDE-DATA.confirm: 0xA7 on task
08:30:17:768 APS-DATA.confirm id: 95, status: 0xA7 NO_ACK
12:14:04:935 APS-DATA.confirm id: 175, status: 0xA7 NO_ACK
13:51:49:307 0x90FD9FFFFE895FB4 error APSDE-DATA.confirm: 0xA7 on task
13:51:49:307 APS-DATA.confirm id: 229, status: 0xA7 NO_ACK
13:51:49:879 0x000D6FFFFE52A5C6 error APSDE-DATA.confirm: 0xA7 on task
13:51:49:879 APS-DATA.confirm id: 203, status: 0xA7 NO_ACK
13:51:50:180 0x90FD9FFFFE78BCC8 error APSDE-DATA.confirm: 0xA7 on task
13:51:50:180 APS-DATA.confirm id: 227, status: 0xA7 NO_ACK
14:05:04:736 APS-DATA.confirm id: 156, status: 0xA7 NO_ACK
14:05:04:843 APS-DATA.confirm id: 157, status: 0xA7 NO_ACK
14:05:05:053 APS-DATA.confirm id: 155, status: 0xA7 NO_ACK
14:13:06:580 APS-DATA.confirm id: 252, status: 0xA7 NO_ACK
14:26:59:294 APS-DATA.confirm id: 79, status: 0xA7 NO_ACK
14:27:05:032 APS-DATA.confirm id: 120, status: 0xA7 NO_ACK
15:45:59:839 APS-DATA.confirm id: 205, status: 0xA7 NO_ACK
16:07:31:375 APS-DATA.confirm id: 64, status: 0xA7 NO_ACK
16:34:40:253 0x000D6FFFFE52A85C error APSDE-DATA.confirm: 0xA7 on task
16:34:40:254 APS-DATA.confirm id: 88, status: 0xA7 NO_ACK
16:34:41:466 0x90FD9FFFFE7FE8F3 error APSDE-DATA.confirm: 0xA7 on task
16:34:41:466 APS-DATA.confirm id: 90, status: 0xA7 NO_ACK
16:34:41:520 0x000D6FFFFE52A5C6 error APSDE-DATA.confirm: 0xA7 on task
16:34:41:520 APS-DATA.confirm id: 94, status: 0xA7 NO_ACK
16:34:42:587 0x000B57FFFEEAEDC0 error APSDE-DATA.confirm: 0xA7 on task
16:34:42:587 APS-DATA.confirm id: 92, status: 0xA7 NO_ACK

So in my understanding 0x7A occurs when a command is sent to a device and only the target device does not respond - therefore it's not a routing issue but the target device is unresponsive?

[Mimiix]

Yep!
As far as i know it is.

[MattWestb]

OSRAM devices can corrupting the ack message so other devices is throwing it because its malformed. The NCP is one old slow cc-2530 that is discontinued and the zigbee stack is not getting any updated for bug and security issues.
The most famous is the very nice outdoor plug that is losing children then not acking messages from the NCP.

Is not helping with the problem but explaining some of the behaviour you seeing.
It can also being some interferens that is messing up the mesh.
Did you using the micro at this time ?

PS: Z2M devs have one good sniff of one outdoor plug that was doing the the malformed message then sent ack as reply for one command.

[githtz]

@MattWestb is that problem valid for all kinds of OSRAM devices or the smart plugs only? I also have error 0xA7 for other devices (e.g SPZB0001). Is it still possible this is an issue with the OSRAM devices?

0x7CB03EAA0A06A25F  OSRAM Plug 01
0x90FD9FFFFE895FB4  TRADFRI bulb GU10 WS 400lm
0x000D6FFFFE52A5C6  TRADFRI bulb GU10 WS 400lm
0x90FD9FFFFE78BCC8  TRADFRI bulb GU10 WS 400lm
0x000D6FFFFE52A85C  TRADFRI bulb GU10 WS 400lm
0x90FD9FFFFE7FE8F3  TRADFRI bulb E27 WS opal 980lm
0x000B57FFFEEAEDC0  TRADFRI bulb E27 WS opal 980lm

@Mimiix If the OSRAM devices are the cause here, maybe we could add an option to the GUI to exclude certain nodes from being used as relays? I don't know if you want this "feature" in your software though...

// EDIT

Maybe let's do this another way round. I'll create some fixed routes for all the devices above in the GUI and then take a look the the problem sill occurs for any of them.

[MattWestb]

The outdoor plug is verified corrupting their own packages but also some bulbs having router problems but i dont knowing the models and if its only old ones.

True forcing the router so they doing using the OSRAM plug in the GUI and the osram only talking with the coordinator (if possible). If you is getting it right it can working well but you need getting the end devices not connecting to the OSRAM plug.

[shoikan]

Same here. Tried 26350500 and 26350500 firmwares. Sensors also don't seem to work most of the time (yes, swapped batteries out for known good ones). In the phoscon webgui lights appear to be 'offline' (grayed out) in the Lights overview, more or less at random, regardless of whether they are turned on (and even actually lit) or not. Variety of lights are affected: Ikea Tradfri, Innr, Paulmann and Osram, so it also does not appear to be 'brand specific'.

My issue appears to have cleared up, but will give it a few days to get a final confirmation. While mucking around I noticed that I would get intermittent connections to the raspbee module. The Phoscon webgui would every now and again present me with 'Firmware not connected' messages. Going through this thread I noticed that my dtoverlay was set to pi3-miniuart-bt-overlay instead of pi3-disable-bt. Since everything has worked flawlessly in the past, I can only imagine I edited this myself at some point, or some upgrade may have inserted/changed this... Anyway, changed that, and now everything appears responsive again.

[githtz]

@shoikan Omg, thanks for the info! Just had a look and guess what, it's just the same! Strange though, I cannot remember editing this file, but maybe I changed the setting using the raspi-config command. Eh, stupid me. I've changed it now and will report back!

[tubalainen]

I have just updated to 2.6.0 with the latest Conbee 1 firmware. Lets see if I run into more non responsive lights.
Will keep you guys posted.