Deconz-rest-plugin: IKEA TRADFRI deconz rest api group scene issue

[realwax]

Versions:
TRADFRI bulb E14 WS opal 400lm with ikea firmware 2.3.050 (zigbee 3.0)
Deconz 2.0.5.77 on Conbee I with firmware 26350500

Issue:

After a scene in a group is called and all bulbs were turned on another scene in the same group supposed to turn of certain bulbs does not work. The bulbs stay on, while the still needed one change brin and ct. Furthermore the bulbs that should have gone dark can't be turned off via off (setting on to false) on the first call. It needs a couple ones.

Reproduce:

Create a group of light. e.g 4 bulbs. Create a scene with all 4 on and on certain settings. Create another scene, switch two bulbs off, modify parameters - store.
Recall the first scene illuminating all 4 - recall the second scene that is supposed to illuminate only 2 bulb. In this case as all are turned on (previous scene) - 2 bulbs should be turned off - this is the issue - they don't.

If you call the second scene (with 2 bulbs on and 2 bulbs off) - no problem, but as soon they are turned on due to a scene recall (scene with 4 bulbs on) they do not turn off again if not needed in another scene (scene with 2 on 2 off) in the belonging group. Furthermore they do no turn off if the whole group is turned off - the group needs to be turned off, on, and off again for success. Further test show that only one TRADFRI bulb is enough in a mixed group to reproduce that issue.

Thanks for your solution/fix!

[Mimiix]

Hi!

I recall something , from when i started using zigbee, that i read somewhere that Ikea bulbs couldn't handle more than 1 ocmmand at the time. If your scene does 2 things , that might be the case.

However, we need to wait until someone with knowledge checks in.

[realwax]

@Mimiix Maybe but with this perception please try to explain why it is not an issue with Tradfri firmware 1.x but it is an issue with 2.x. The mitigation is easy and not. Stay on version 1.x but that's a bummer as we need to move on and with future replacements it won't be possible to even have a version 1.x. Maybe deconz needs a workaround. Let aim for solutions...

[Mimiix]

Hey i have no clue , i've acknowledge your issue and i just mentioned it to give another angle 😄. I'll add the To-do label.

[realwax]

Thank you. 🙂

[realwax]

I tested even further and found a "hotfix" that's actually not really good but works. I use iobroker to monitor scene switches. In case of a switch from a scene with more bulbs on to one with less I send an off command to the whole group followed by the desired scene recall. This leds to bulbs being turned off and others to change. I can can catch this in iobroker but i should not as the api or should do it on their own. I pretty confident that something in IKEA fw changed that would be great i deconz could handle it...

[ChrisPrefect]

I have the same issue. Scenes with lights that should be off are not switched off.

Other scenes can't remember the light color (white spectrum). Is this related?
https://github.com/dresden-elektronik/phoscon-app-beta/issues/65

[ebaauw]

The Zigbee standard doesn't support colour temperature in scenes. Different lights have different workarounds for this, but not all. Pretty sure the REST API plugin doesn't handle all workarounds. Safest way to set a scene is to set the light to the desired state, wait some time to make sure any transitions have finished, and issue a _Store Scene_ command (through the _Scenes_ cluster in the GUI, or through the API issue a PUT to /groups/_n_/scenes/_m_/store.

[realwax]

The Zigbee standard doesn't support colour temperature in scenes. Different lights have different workarounds for this, but not all. Pretty sure the REST API plugin doesn't handle all workarounds. Safest way to set a scene is to set the light to the desired state, wait some time to make sure any transitions have finished, and issue a _Store Scene_ command (through the _Scenes_ cluster in the GUI, or through the API issue a PUT to /groups/_n_/scenes/_m_/store.

@ebaauw Thank you. Yes this works. In a group I set the single members to desired state and store the scene. Recall works with desired CT. BUT sadly it doesn't mitigate the on off issue described. So bulbs stay on when they supposed to go off as in the next scene they set to off. They stay on. Something changed whether in the Ikea FW from 1.x to 2.x or/and in the handling of deconz with zigbee 3.x. Maybe on both sides. The only mitigation I know for that is sending a complete off command to the group before recalling the next scene. I used iobroker but in use with Phoscon it's nearly unusable for that. Hope there is a solution soon....

[ebaauw]

So bulbs stay on when they supposed to go off as in the next scene they set to off.

If you have a Zigbee sniffer, it would be good to issue an _Enhanced View Scene_ from the GUI, and double-check whether the _OnOff_ attribute from the _On/Off_ cluster in stored in the scene.

Hope there is a solution soon....

If you created/updated the scene using _Store Scene_ and it still doesn't work, I'm afraid it's an issue in the light firmware. I don't think the REST API plugin can do anything to workaround that. Which Trådfri bulbs and firmware does this concern?

[realwax]

@ebaauw the interesting thing about this issue is that if you recall a scene and after transition (if group was of before) switching the group off works. Doing the same but having a second scene switch, with let's say a group of 4 lights with now only 3 on and the 4th off that was on before, the 4th stays on and even can't be turned off when the group is switched off. In this cases you need to off on and off again to catch the whole group. I think it's is a mix of changed ikea fw and deconz handling this. Who can look into this?
The bottom line is.... Deconz has a low usability on Ikea Tradfri FW >1.x and this will be a present and future issue.

My current knowledge is low on zigbee sniffing but if it's helps I can get into it. Would flash myself a cc2531 and try it on my windows maschine as my raspis are full. Is there any quick guide you could recommend?

In terms of bulbs, fw,... I specified it in the issue first place. As said no issues with 1.x but that's not future proof as newer bulbs don't offer 1.x.

Versions:
TRADFRI bulb E14 WS opal 400lm with ikea firmware 2.3.050 (zigbee 3.0)
Deconz 2.0.5.77 on Conbee I with firmware 26350500

Thank you Erik.

[ebaauw]

I think it's is a mix of changed ikea fw and deconz handling this.

Zigbee scenes are stored on the lights themselves. Each light stores its target state under the group and scene ID. The payload of a Zigbee _Recall Scene_ command only contains these IDs. There's nothing deCONZ can do (or screw up) to influence the state when recalling a scene.

The only place were deCONZ could (and in case of colour temperature often does) screw up is when setting the scene state in the light explicitly (if memory serves through the _Add Scene_ command, which carries the target state in the payload). I plan to refactor that code somewhen in the not too distant future, after I'll be done with the light state. By using _Store Scene_ instead, which also just contains the group and scene ID (and transitiontime, but not the target state), we can rule out deCONZ as culprit, as the light itself stores its current state under the group and scene ID.

[ebaauw]

Is there any quick guide you could recommend?

Sorry, no. I use ZShark with an old ConBee stick for capturing and WireShark for analysis.

[realwax]

I think it's is a mix of changed ikea fw and deconz handling this.

Zigbee scenes are stored on the lights themselves. Each light stores its target state under the group and scene ID. The payload of a Zigbee _Recall Scene_ command only contains these IDs. There's nothing deCONZ can do (or screw up) to influence the state when recalling a scene.

The only place were deCONZ could (and in case of colour temperature often does) screw up is when setting the scene state in the light explicitly (if memory serves through the _Add Scene_ command, which carries the target state in the payload). I plan to refactor that code somewhen in the not too distant future, after I'll be done with the light state. By using _Store Scene_ instead, which also just contains the group and scene ID (and transitiontime, but not the target state), we can rule out deCONZ as culprit, as the light itself stores its current state under the group and scene ID.

@ebaauw
Thanks - well then I understand. This reduces the scope to IKEA messed up their firmware.

Could it be that lights that aren't "on" have been stored in the group/scene setting on the bulb itself with ikea fw 1.x with e.g bri 0 or bri 1 (aiming at bulb is not on) and on ikea fw 2.x it doesn't not store the scene if it is not turned on?

Is there anyone here who could debug this deeper? I doubt I am the only one affected. I am a power user / former sys admin and I need to get into this deeper / lacking of knowledge in terms of zigbee debug.

Do you know how to address ikea with this? Dev FW?

Cheers!

[SwoopX]

My current knowledge is low on zigbee sniffing but if it's helps I can get into it. Would flash myself a cc2531 and try it on my windows maschine as my raspis are full. Is there any quick guide you could recommend?

@realwax Check out this for a start:
https://www.zigbee2mqtt.io/how_tos/how_to_sniff_zigbee_traffic.html

[realwax]

@SwoopX @ebaauw Thank you.

Here is my dump. Used Wireshark with ZBOSS

tradfri dump.pcapng.gz

Now I recorded the following:
Group 14 /0x0e contains of 3 E14s Tradfri.
Recalled scene 4 with one bulb on.
Recalled scene 3 with three bulbs on.
Recalled scene 4 with one bulb on. (other bulbs not going off)
Turned group off (scene 4 bulb went off)
Turned group on & off again (caught the rest of the bulbs off)

Can you help me further with reading the dump. TCP/IP i understand but here it would take me too long to see if its working as supposed. I am confused with plenty messages with same seq number but altering source addresses and so on. Thanks for your help.

In parallel I wrote to IKEA... fingers crossed.

[realwax]

@ebaauw in addition here is the log of store scene. Same group as above.
Turned all 3 bulbs on manually (not group), turned on off, created a new scene, turned bulb on and off again, stored scene again.

tradfri dump.pcapng store.pcapng.gz

[ebaauw]

@realwax, I cannot read the dumps, as the network key isn't present. Can you post that, or perhaps better, PM that to me? You want to pre-configure that in WireShark, under _Settings_ | _Protocols_ | _ZigBee_, to it deciphers the messages for you.

Could you also capture a dump while issuing a _Enhanced View Scene_ commands from the _Cluster Info_ panel, for each of the scenes?

[realwax]

@ebaauw
Here is another file with the E-V-S command for both scenes.
play script: Recalled scene 4 ( 1 bulb one 2 off) recalled scene 3 (all 3 on) recalled scene 4 again
(reproduced issue the other 2 stayed on). I will PM you the keys to your xs4all mail...

tradfri e14 dump scenes 18062020.pcapng.gz

In addition I another debug. I did a recall on one bulb group member. first group 4(not on) then 3 on and 4 again off. it got stuck as well. So a recall directed to the bulb itself while other members were untouched. Afterwards I did the enhanced scene view again on all three bulbs.
tradfri e14 dump scenes more detailed 18062020.pcapng.gz

Thanks a lot

[realwax]

@ebaauw have you received my keys via mail?
@all please elevate this one. Maybe we can connect with IKEA on some level.
https://www.reddit.com/r/tradfri/comments/haqrf5/tradfri_firmware_issue_in_between_major_version/

[ebaauw]

Yes, I have, thanks. By the looks of it, the scenes are stored correctly: scene 0x000E, 0x03 should turn on the three lights (NWK 0x20ce, 0x8498, 0x6797), whereas scene 0x000E, 0x04 should only turn on 0x20ce and turn off the other two lights. They did change the scene logic in their latest firmware, storing colour temperature in _Color X_. I'll dust off my TW light and see it has the same issue.

[realwax]

@ebaauw Thank you for time and work. I see. Glad to hear. I struggle a little bit sorting this out myself as many messages are produced and how to read them correctly. Need to learn deeper. Tcp/IP wasn't a problem. I am very thankful for your support and attention. I tried via reddit and local country store to get hold of the Tradfri team. Fingers crossed...

[ebaauw]

Ok, finally found the time to dust off and test my Trådfri E27 TW. Seeing exactly the same behaviour: the scene seems to have stored ok on the light, but it doesn't turn off when recalling the scene. When the light is already off, recalling the scene won't turn it on.

IEEE 802.15.4 Data, Dst: 0x0000, Src: 0x8997
ZigBee Network Layer Data, Dst: 0x0000, Src: 0x8997
ZigBee Application Support Layer Data, Dst Endpt: 1, Src Endpt: 1
ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x19)
    Sequence Number: 50
    Command: View Scene Response (0x01)
    Payload, String: 
        Scenes Status: Success (0x00)
        Group ID: 0x0100
        Scene ID: 0x10
        Transition Time: 1 seconds
        Length: 0
        String: 
        Extension field set 1
            Cluster: On/Off (0x0006)
            On/Off: 0
        Extension field set 2
            Cluster: Level Control (0x0008)
            Level: 254
        Extension field set 3
            Cluster: Color Control (0x0300)
            Color X: 0.5052

[ebaauw]

I'm afraid this is a bug in the IKEA firmware, for which deCONZ cannot provide a workaround.

I still had the previous (ZLL) firmware file (image 0x2202, version 0x12217572) lying around on one of my systems. After downgrading the bulb firmware and re-creating the scenes, it works. Note that they changed the scenes logic from storing _Current X_ and _Current Y_ to only storing _Current X_ on the latest firmware.

Make sure to delete the latest firmware from ~/otau, or the bulb will be upgraded to the latest firmware automatically, after downgrading.

IEEE 802.15.4 Data, Dst: 0x0000, Src: 0xcad5
ZigBee Network Layer Data, Dst: 0x0000, Src: 0x8997
ZigBee Application Support Layer Data, Dst Endpt: 1, Src Endpt: 1
ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x19)
    Sequence Number: 200
    Command: View Scene Response (0x01)
    Payload, String: 
        Scenes Status: Success (0x00)
        Group ID: 0x0100
        Scene ID: 0x10
        Transition Time: 1 seconds
        Length: 0
        String: 
        Extension field set 1
            Cluster: On/Off (0x0006)
            On/Off: 0
        Extension field set 2
            Cluster: Level Control (0x0008)
            Level: 254
        Extension field set 3
            Cluster: Color Control (0x0300)
            Color X: 0.4599
            Color Y: 0.4106

[ChrisPrefect]

Hmm but even if it is a bug in IKEA, there should be a work around. We can't just install old firmwares, right?
Could deconz send an off-command if the scene requires the light to be off?
Or send an off-command, immediately followed by the load scene command?

[ebaauw]

We can't just install old firmwares, right?

Actually, with IKEA you can (I just did, see above). Not sure if you can still download the old firmware, though.

there should be a work around

Yes, create a group of the lights that are supposed to turn off, and send an _Off_ command to that group, when recalling the scene.

Could deconz send an off-command if the scene requires the light to be off?

Maybe in theory, but practically: no. deCONZ would have to go through each light in the scene, check whether it's currently on, whether the scene would turn it off, and whether the light is on a firmware version with the bug, and then send a unicast _Off_ command to the light. I'm no fan of breaking group commands into unicasts. Even then, it would still be iffy, hoping that deCONZ' cache correctly reflects the scene and the state in each light.

Or send an off-command, immediately followed by the load scene command?

That would cause the lights that aren't turned off by the scene to blink. And IKEA lights would probably ignore the scene command when they're still transitioning to off.

[realwax]

@ebaauw thanks for your efforts and analysis!
Do you see it as bug or does it result out of the changed scene logic? If it is the latter than they won't change much.
There is a downside of downgrading.... Only older bulbs can do that. No older firmware for newer versions and no zigbee 3. If you have to go for replacements you're doomed with that issue. So we should try to get there intension.
I have tried to get in touch with the Tradfri team via the Austrian local support. No feedback until now.
I think the Tradfri team is based in NL and one guy is called Petter. I tried via reddit too. No success yet.
Maybe some of you guys could send them mails reporting that too. They need capitalistic pressure 😅

[ebaauw]

Do you see it as bug or does it result out of the changed scene logic?

This is definitely a bug. I think it was introduced when changing the scenes logic, to support colour temperature as _Current X_ in the scenes. I would hope/expect that it will be solved in future firmware. I think it's limited to the tunable white (colour temperature) lights.

I'm sorry, I don't have any insights into the "Trådfri team". Maybe @manup does. I'm not even sure if their hub would support scenes like this, bringing out this bug, so it would be good to notify them.

[ChrisPrefect]

Ok so I am pretty sure that IKEA won't fix any bugs that are only present when using third party gateways.

And I can't work around this bug, without downgrading all the lights (80 lights, I won't do that).

So we are stuck with this bug forever in Phoscon?