Core: "hue.hue_activate_scene" not working consistently since 0.66

[danrspencer]

Home Assistant release with the issue:
>=0.66.0

Last working Home Assistant release (if known):
0.65.6

Operating environment (Hass.io/Docker/Windows/etc.):
Hass.io running on a Pi3

Component/platform:
https://www.home-assistant.io/components/hue/

Description of problem:
In 0.66.[0-1] I'm having issues with some Hue scenes not working. When testing via the service dev tool I'm able to replicate the same issue I see with my automations.

e.g.

Calling the following doesn't work.

hue.hue_activate_scene

{
"group_name": "Hallways",
"scene_name": "Night"
}

But it does to work when calling the of the default scenes. While the same call failed for most of my rooms it did work for my Kitchen group for unknown reasons.

When I roll my version of Hass back to 0.65.6 everything starts working again as expected.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

hue:
  bridges:
    - host: 192.168.0.2
      allow_unreachable: false
      allow_hue_groups: true

https://github.com/danrspencer/hass-config/blob/master/configuration.yaml

Traceback (if applicable):

Additional information:
I'm happy to dig up additional logging output if it's helpful, just point me in the right direction. I had a quick look through the logging output in About screen but didn't see anything that indicated a problem.

[jimbob1001]

I had this same issue, for me it was scenes named "Full", "Dim" & "Night" that didn't work, but scenes named "White" & "Morning" did work.

I renamed all my native hue scenes from "Night" to "Living Room Night" & they worked again.

[danrspencer]

I've got a lot of rooms so renaming everything would be a huge pain, plus it would mess up using the scenes via Google Assistant, so it's not an ideal solution for me.

The fact everything works as expected when I roll back to 0.65 indicates it's an issue with the new 0.66 release.

Since posting this I've tried a few things such as re-linking the hub and nuking the entity registry but neither has solved the problem (although nuking the entity registry did solve a problem of HA not picking up the new names of some renamed Hue bulbs).

[sj3fk3]

Same issue here. Scene does work for "Woonkamer" but does not work for "Bathroom". I though I was going mental until I saw this bug-report... :-/

[xirixiz]

I have the same issue on multiple hue scenes. "Woonkamer" doesn't work, "Woonkamer Night" does.
Verified that all scenes exist on the HUE bridge.

[bart-roos]

Same issue over here. I guess this has something to do with #13043

[nelbs]

Same issue over here :(

[phairplay]

Just added my new hue to home assistant and found this while googling the problem.

[noodlemctwoodle]

I was able to removed all the default and custom scene from my config in the Philips Hue App and create new ones.

'hue_scene_bedroom_night':
  alias: Hue Bedroom Night
  sequence:
    service: hue.hue_activate_scene
    data:
      group_name: Bedroom
      scene_name: 'Night'

They are now working again in 0.66.0

Not an ideal situation if you have many scenes, but a quick work around

[MarkH170]

Having similar issues here with the following error:

Invalid service data for hue.hue_activate_scene: required key not provided @ data['group_name']. Got None
required key not provided @ data['scene_name']. Got None

When triggering the following script:

script:
  hue_living_room_bright:
    alias: Living room bright
    sequence:
      - service: hue.hue_activate_scene
        data:
          group_name: "Living room"
          scene_name: "Bright"

Edit: Added script

[noodlemctwoodle]

You can bypass the error by using a non-default hue scene.

[xirixiz]

@noodlemctwoodle I also have problems with some non-default hue scenes

[danrspencer]

@noodlemctwoodle I don't have the problem at all with default hue scenes, for me it seems to only effect non-default scenes.

[xirixiz]

I renamed all scenes in the hue app and in home assistant from "name scene a", "name scene b" to just numbers "1" and "2". Now everything is working again.

Update: Not working. Once I added the same scene IDs to other rooms, the problem re-appears

[xirixiz]

What works for me now is to use unique names for scenes in every group. Sucks, but it can be used as a workaround.

[grahamdw]

From what I remember of the Hue API, to activate a scene you need to quote a group ID and a scene ID. The group ID is a number, and the scene ID is a unique "random" string; both are allocated by the bridge. Scene names are much reused, so it's necessary to use a scene ID relevant to the group when calling the API. So a matching process is needed between scenes and groups, in the Hue component.

If you don't do this matching, but your scene name is unique, it will work. If the scene name is not unique it may work, if the component happens to use a scene ID relevant to the group specified. If the scene and the group have no lights in common, then the scene activate "works", but nothing will happen.

[danrspencer]

@grahamdw that would explain the behaviour we're all seeing, sounds like a likely culprit.

[danrspencer]

This is still broken in 0.67, back to 0.65 again I guess.

[sj3fk3]

Still broken.. Also in 0.68 👎

[raymccarthy]

Originally posted to wrong issue. Sorry.

I think I have found the issue. The following code in 0.65 managed the situation where a scene name exists in multiple rooms

        if len(scenes) == 1:
            self.activate_scene(group.group_id, scenes[0].scene_id)
            return
        # otherwise, lets figure out if one of the named scenes uses
        # all the lights of the group
        group_lights = sorted([x.light_id for x in group.lights])
        for scene in scenes:
            if group_lights == scene.lights:
                self.activate_scene(group.group_id, scene.scene_id)
                return

There was also this comment

This provides a convenience way of activating scenes by group
name and scene name. If we find exactly 1 group and 1 scene
with the matching names, we run them.

If we find more than one we run the first scene who has
exactly the same lights defined as the group. This is far from
perfect, but is convenient for setting lights symbolically (and
can be improved later).

The refactored code looks like this

```
group = next(
(group for group in self.api.groups.values()
if group.name == group_name), None)

    scene_id = next(
        (scene.id for scene in self.api.scenes.values()
         if scene.name == scene_name), None)

````
From what I can see it will only find the first scene. I have confirmed this behaviour with TCPDUMP, it always uses the same (first) scene_id it finds.

Will see if I can add back the old scene logic

[Mariusthvdb]

maybe change the OP title in Not working.
Any response from dev side yet? This is quite a setback for such an important piece of integration. We shouldnt create alternatives, but rather get the original back to its correct and reliable proces.

Could this have to do with the other Hue issues? On/unavailabe lights/groups switching all the time, and maybe even preventing Hassio from loading? Host unreachable (while in fact being there prominently....)
this is happening since upgrading to .68 and now .68.1:

[MizterB]

This duplicate scene name issue came across when aiohue was introduced to the Hue component. In order to re-implement the previous logic, the following change needs to be made to the library first.

https://github.com/balloob/aiohue/issues/6

[Mariusthvdb]

thank you for this pointer, i most certainly hope this will be addressed soon, cause it is causing more problems by the day it seems. Horrible timing issues in the logs arise. Glad it is finally identified, always the way to a solution .
Cheers,
Marius

[nelbs]

Unfortunately it is not fixed yet in 0.69.0 ☹️

[Mariusthvdb]

bummer. was hoping desperately...
how can we bump this up on @amelchio 's priority-list ?

[danrspencer]

Ignore the close and reopen. Hit button by accident while scrolling on my mobile.

[amelchio]

@hvbhome Sorry, this is not on my list at all, I am not a Hue maintainer.

[Mariusthvdb]

Sorry, this is not on my list at all, I am not a Hue maintainer.

O I am sorry @amelchio , I thought because of you being mentioned on https://www.home-assistant.io/blog/2018/05/11/release-69/# you'd be the one to talk to ;-)

Seems our issue and error reporting on the new Hue implementation gets a bit lost though, not sure if we could help in solving the issues that are still there, also now on 69.1

[sj3fk3]

Who is the maintainer and does he/she know that this issue basically breaks the whole hue implementation for HA?

[MizterB]

Just submitted a pull request for the change to aiohue.

[MizterB]

Another pull request submitted to aiohue. I have working code that restores the old scene logic, but cannot submit it here until that library is updated.

[MizterB]

If anyone is feeling inpatient and wants to apply a temporary fix, the changes are available here (using a temporary aiohue repository with the required changes):

https://github.com/MizterB/home-assistant/commit/553aec53d081e06d519c85d4684ba5baf787e963

[Mariusthvdb]

impatient here...;-)
how to apply this, ssh into hassio doesnt show these files..?

[MizterB]

Pull Request submitted: https://github.com/home-assistant/home-assistant/pull/14645

[danrspencer]

Has anyone had chance to test this in 0.70? I didn't see anything in the patch notes about it.

[amelchio]

It's scheduled for 0.70.1.

[Sleezly]

Fix doesn't appear to be working in 70.1 unfortunately.

https://github.com/home-assistant/home-assistant/issues/14739

[Mariusthvdb]

do believe it does work again, as reported in https://github.com/home-assistant/home-assistant/issues/14739

[sebk-666]

I can confirm that - at least for me - it's working again. Finally.

[Sleezly]

Thanks for confirming Sebastian.

I have multiple Hue hubs and it appears this may be the result of the inconsistency. I've confirmed sending hue_activate_scene for a group/scene combo works on one hub but fails on the other (my primary).

Can anyone else with multiple hubs confirm hue_activate_scene works for both hubs?

In case it helps, here's what's observed. Invoking hue.hue_activate_scene directly via developer tool services with following JSON payload:
{"group_name":"Downstairs","scene_name":"Dinner"}

Per Hue debug clip API, confirmed there's a Hue Group:
"name": "Downstairs",

and that group has Scene:
"name": "Dinner",

Results in the following error:
Error executing service
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/homeassistant/core.py", line 1007, in _event_to_service_call
await service_handler.func(service_call)
File "/usr/lib/python3.6/site-packages/homeassistant/components/hue/bridge.py", line 137, in hue_activate_scene
group_lights = sorted(group.lights)
AttributeError: 'NoneType' object has no attribute 'lights'

[MizterB]

The error you are getting indicates that the Hue API did not locate the Downstairs group. I don't have multiple hubs, so I don't know how the code handles that - but I suspect that the API is looking for the "Downstairs" group on the wrong hub.

While the code appears to be working for single-hub users (!), this does highlight some cleanup I should do in the logic - which is to move the check for a group of None to execute right after the group search, and not after the scene search. That would gracefully handle the error, but would not address your particular issue.

I'll wait to see if we can get any feedback on multiple hubs before I submit any updates. But if anyone is still experiencing the original issue with a single hub, please let me know!

[Mariusthvdb]

Hi,
First off: thanks for your efforts giving us the activate scenes back!

What I can say is compared to before the asyncio move , the response of the lamps lags significantly . Selecting a scene can take up to a second or 10, while before it was quasi immediate.

Switching a light itself seems alright.

[Sleezly]

Thank you for agreeing with the multiple hub issue speculation, Mitzer.

I have successfully worked around this limitation and possibly confirmed the multiple hub issue by changing the IP address for the primary hub in an attempt to reorder how Home Assistant would look for Hue groups.

Before:
primary - 192.168.1.9
secondary - 192.168.1.15

Now:
primary - 192.168.1.99
secondary - 192.168.1.15

With the IP addresses changed, I'm now able to activate scenes on the primary hub and, as expected, attempting to activate a scene on the secondary hub which previously worked, now does not.

[danrspencer]

Doesn't appear to be working correctly for me on 0.70.1. Some rooms just aren't turning on with the anticipated scenes, as was the behaviour prior to 0.70.1.

Back to 0.65.6 for now.

If I get some free time to investigate further I'll have a dig around. I only have a single hub so that's not the issue for me at least.

Edit: fixed incorrect reference to 0.60.1 instead of 0.70.1

[TheJulianJES]

@danrspencer Are you talking about 0.70.1 or actually about 0.60.1?

[danrspencer]

@TheJulianJES yep I meant 0.70.1, I’ve updated my previous comment to prevent confusion.

I’ve been playing with my lights a bit more and with 0.70.1 running the problem now seems inverted. Previously only my kitchen worked with the duplicate scene names, now every room but the kitchen works. My hypothesis would be that the first room with the duplicate named scene is now being skipped completely and HA is then failing to find the scene / room in the remaining list.

[danrspencer]

@balloob can we get this issue re-opened since the bug does not seem to be fixed fully. Or should we start a new issue?

[balloob]

Should be a new issue.

The fact that this is so fragile and not easy to reason about the correct logic makes me wonder if we shouldn't just remove it. Home Assistant supports scenes, scenes can be pulled in from platforms. This service really is a shortcut/hack.

[danrspencer]

I use Hue scenes specifically so that the rest of the family can tweak them without having to commit code to GitHub and because it makes all of the lights come on at once.

I imagine a lot of people would find HA a lot less useful if it dropped support for Hue scenes.

[MizterB]

OK - something does appear to be wrong with the released version...I'm getting errors and the lights aren't changing. But I just went back to my development branch that was merged and everything is working fine.

Don't give up on this yet - just need some time to debug against the 70.1 release.

[danrspencer]

@MizterB Cheers. I'm going to fork and checkout the HA repo to have a poke around too (although having done no HA dev before it might take me a bit to get up to speed...). If nothing else I might try get some tests written around the bridge component to lock down the scene behaviour.

[MizterB]

@danrspencer - can you post a debug log snippet from when you call the active_scene service? Would be helpful to see what's your system is doing.

[danrspencer]

I get the following:

2018-06-03 15:50:58 DEBUG (MainThread) [homeassistant.components.websocket_api] WS 1919927216: Received {'type': 'call_service', 'domain': 'automation', 'service': 'trigger', 'service_data': {'entity_id': 'automation.kitchen_lights_on'}, 'id': 35}
2018-06-03 15:50:58 INFO (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: domain=automation, service=trigger, service_data=entity_id=automation.kitchen_lights_on, service_call_id=1972850160-100>
2018-06-03 15:50:58 INFO (MainThread) [homeassistant.components.automation] Executing Kitchen Lights On
2018-06-03 15:50:58 INFO (MainThread) [homeassistant.core] Bus:Handling <Event logbook_entry[L]: name=Kitchen Lights On, message=has been triggered, domain=automation, entity_id=automation.kitchen_lights_on>
2018-06-03 15:50:58 INFO (MainThread) [homeassistant.helpers.script] Script Kitchen Lights On: Running script
2018-06-03 15:50:58 INFO (MainThread) [homeassistant.helpers.script] Script Kitchen Lights On: Executing step call service
2018-06-03 15:50:58 INFO (MainThread) [homeassistant.core] Bus:Handling <Event call_service[L]: domain=hue, service=hue_activate_scene, service_data=group_name=Kitchen, scene_name=Day, service_call_id=1972850160-101>
2018-06-03 15:50:58 ERROR (MainThread) [homeassistant.core] Error executing service <ServiceCall hue.hue_activate_scene: group_name=Kitchen, scene_name=Day>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/homeassistant/core.py", line 1007, in _event_to_service_call
    await service_handler.func(service_call)
  File "/usr/lib/python3.6/site-packages/homeassistant/components/hue/bridge.py", line 144, in hue_activate_scene
    if not updated and (group is None or scene_id is None):
UnboundLocalError: local variable 'scene_id' referenced before assignment

Which appears to be the same error as if I try and use a scene that doesn't exist. I've double checked in the Hue API and I can find a room named Kitchen and then a scene named Day with the same lights as kitchen, including the same ordering (as expected since the automation works in 0.65.6).

It look like scene_id should be initialised to None before the if statement on line 130? I'm not convinced that's what's causing the problem but it looks like (my PythonFu is weak) the reason we're getting an exception rather than dropping into the LOGGER messages?

Edit: API extracts incase it helps

{
   "name":"Kitchen",
   "lights":[
      "6",
      "7",
      "10",
      "11",
      "12",
      "13",
      "17",
      "22",
      "23",
      "27",
      "28"
   ],
   "type":"Room",
   "state":{
      "all_on":false,
      "any_on":false
   },
   "recycle":false,
   "class":"Kitchen",
   "action":{
      "on":false,
      "bri":128,
      "hue":39415,
      "sat":13,
      "effect":"none",
      "xy":[
         0.3691,
         0.3719
      ],
      "ct":230,
      "alert":"none",
      "colormode":"xy"
   }
},

   "0ThM80CZEG67XMZ":{
      "name":"Day",
      "lights":[
         "6",
         "7",
         "10",
         "11",
         "12",
         "13",
         "17",
         "22",
         "23",
         "27",
         "28"
      ],
      "owner":"YaxqVTURQoT-xaF39ga8zAmlCUSkYxCTOtq6oqgc",
      "recycle":false,
      "locked":false,
      "appdata":{
         "version":1,
         "data":"PsyL2_r01_d99"
      },
      "picture":"",
      "lastupdated":"2018-04-29T12:16:14",
      "version":2
   },

[danrspencer]

As I said, my PythonFu is weak. But would something like this be a better way of getting the scene and avoiding the uninitialised scene_id?

scene = next(
            (scene for scene in self.api.scenes.values()
             if scene.name == scene_name and scene.lights == group.lights), None)

[balloob]

Please open a new issue.

[balloob]

@danrspencer: I use Hue scenes specifically so that the rest of the family can tweak them without having to commit code to GitHub and because it makes all of the lights come on at once.

Like I wrote, platforms can provide scenes that live on other systems, like Hue. That is the right solution and should replace this hack.