Home Assistant release with the issue: 0.97.2
Last working Home Assistant release (if known): 0.96.5
Operating environment (Hass.io/Docker/Windows/etc.): Raspberry Pi with Raspbian / Debian Linux 9 Stretch
Component/platform: Homekit
Description of problem:
Hi! I have a very basic alarm system, only an Aqara Hub and an Aqara window/door sensor. I installed Home Assistant 0.96.5 and added the Aqara Hub as a HomeKit accesory (the UI detected it automatically, I only had to enter the code in the label). All was working perfectly, when I pressed ARM AWAY or DISSARM button it was almost instantaneously, and when someone opened the door the state changed to TRIGGERD (and I got the notification) very quickly.
Now I have installed from scratch (the SD card died) with latest version, 0.97.2, and I have added the alarm, but I having an tremendous lag. It takes about one minute any change, ARM, DISSARM or TRIGGER.
Any clue?
Lot of thanks
I have downgraded to 0.96.5 and all works instantaneously, so the lag is only present newer versions
It sounds like this is about homekit_controller rather than homekit (the homekit
integration is for using the HomeKit protocol between your iPhone etc and HomeAssistant, homekit_controller
is for using the HomeKit protocol between a device (like the Aqara Hub) and home assistant). Can you confirm that is the case? I can retag to the right area if so.
Also, if its homekit_controller:
The homekit_controller integration is poll only, and it polls once a minute. It has only ever polled once a minute. If it used to seem faster then that was just luck - it must have been triggered near the 1 minute timer running out.
On newer builds there is some times a race on start up where the first poll is missed. But this has no other impact on a running system.
The dev branch i'm working on will replace the blocking homekit client with an asyncio one. The main aim of this work is support for HomeKit events which means that things like this will update near instantaneously, but the current synchronous client is incapable of supporting events. This is all working in my dev environment but relies on a PR getting accepted upstream, then a release happening, then a PR to home assistant to use the new release, then a library, and a good amount of QA needs to be done at each stage. So it will be a few releases probably.
Hello Jc2k !
Wouow ! It looks like wonderful !
Is there any chance you make your dev branch public for the ones who want to work with ? I just can鈥檛 wait to test it ! 馃構
Anyway: a huge Thank you !
Does that mean you agree this ticket is for homekit_controller
and not homekit
? :-)
The code should all be on my GitHub but it involves dev versions of HA and homekit_python so i'm reluctant to describe or support how to try it out just yet! When the homekit_python part is released onto pypi i'll open up a PR on the home-assistant side and ping this ticket.
Hello again.
From my side, I talk about homekit_controller but I am not the first author: your projet sounds like interesting and I wonder if I could try it somewhere.
I understand about support problem. So I will be patient.
Thank you again.
@Jc2k I paired a Honeywell Lyric Controller via HomeKit Controller and noticed the same behavior. The updates would take some time. When I paired them with my phone, the state changes showed instantly.
Just chiming in to add in my experience with homekit_controller
.
Hi everyone,
Having the same issue here.
In my case updates are slow, but also, after a couple of hours (2 to 3) I stop receiving ANY update from any of the devices.
The only solution I've come up with is to restart the container to have it back to normal.
v0.100.2 -- Docker
Xiaomi Aqara Bridge [China Server]
(3 - Temp sensors; 1 - Motion sensor; 1 - Door sensor)
@gargomoma can you please confirm what you mean by slow? As explained, a delay of about a minute is normal and won't be changing for a while until the work to support events is ready. Longer than that is a sign of a problem.
The 2nd part of your comment - stopping after a few hours is definitely not normal, could you please raise a seperate ticket for that though. It's a seperate problem. Please could you check your logs too and post those on the new ticket. It's likely that the connection is timing in a way that we haven't seen before and we'll need to handle that case better than we currently do. Feek free to @Jc2k on the new ticket so I get notified.
@Jc2k Hey mate. Any update on this slow response issue with HomeKit controller with regards to sensors taking up to 1min to respond?
I am also waiting for the fix. Thanks heaps for your efforts
Hey bud. it鈥檚 not really a fix, the code is working as designed. The plan is to moving from a simple synchronous polling model to a fully asynchronous event based model. While I have done a lot of work to prepare for this that is merged and the remaining patch won鈥檛 be that large the effect will be like I rewrote the thing entirely. With that level of change is the risk that every bug I fixed for the old code might be reintroduced.
I鈥檝e run out of spare time and while I have this working for myself more or less it would be irresponsible to drop it on HA and run.
So for now you are kinda stuck, sorry.
Hey bud. Thanks heaps for your efforts for sure. I am not a programmer so I can only imagine the challenges you face with coming up with a solution. Sounds like a huge amount of work you have done. I know you will do the right thing and only release something when you can and when you are confident of success. It would be amazing if other knowledgeable programmers could assist you.
I paired my hub directly to iOS app initially and everything was perfect with instant response from all door sensors and motion sensors, 8 devices operating perfectly and instant. The only issue was getting devices into HA. So unpaired and paired to HA successful with all devices showing in HA, only problem not, Speed of detection, around 1 min to detect sadly.....
I sure hope something can be achieved soon as it is basically unusable in its current state
Also
Tried unpairing from HA to connect directly back to HomeKit iOS and now it won鈥檛 connect or pair back to HomeKit iOS, so now definitely unusable.
I know the community of users having this issue would appreciate a resolution when one ready to deploy. Well done and good luck bud
Yeah, if you are using door and motion sensors homekit_controller is really not for you until the async stuff lands. There is only a motion sensor at all so that I can do dev work on support for HomeKit BLE (I have an Eve bluetooth motion sensor). A workaround is to use the homekit
integration to export switches to your iOS devices and iOS automations to trigger the switches when something is sensed. I use a similar trick so that I can use HomeKit home arrival triggers in HA.
You have to unpair on both sides. On simpler devices this might mean a full reset, but hubs that speak multiple protocols (like Hue) often have a "reset homekit" menu option somewhere. We should be able to tell the devices when HA removes it pairing data, but it's low down the list - I think most users would have e.g. async / events rather than a silightly easier unpair.
Hey bud. Yes I agree most users would want Async/events rather than easier unpair. Async is priority. I cannot work out why I cannot setup new pair with IOS device again, as the little trick of using HA Switches is exactly what I was initially doing to get actions into HA. Worked OK just not ideal.
BLE however is another option. I use Xiaomi Bluetooth Temp and Humidity sensors with BLE Tracker. I do not have any other Bluetooth devices other than these temp/hum and they work flawlessly. They are very solid and reliable using ESP32 board as BLE tracker.
I will try unpair both sides, that is Homekit, and HA and try to setup Aqara Hub directly with IOS Aqara app. Hope it works as using the switches will be fine until and solution is found.
Will look at the Eve Bluetooth Motion Sensors also
To be clear Eve / Bluetooth homekit stuff doesn鈥檛 work with HA natively yet, it鈥檚 next on my list after async.
Firstly, I want to thank you for your work, it is a great contribution to the community. Are you predicting when the update will be available for us to have an instant update?
No, the past 2 months have seen the time I have slow to a near stand still. Async and events is a big chunk of work. Even though it鈥檚 working locally in my dev environment more or less there is still a big gap before it鈥檚 ready to inflict on hundreds of HA users.
Hey there @Jc2k, mind taking a look at this issue as its been labeled with a integration (homekit_controller
) you are listed as a codeowner for? Thanks!
@Jc2k Still following this buddy, Keep up the great work you are doing for the community, it is greatly appreciated for sure.
@Jc2k - please let me know if you need any help with the development and/or testing. Looks like you鈥檝e done most of the work already... I see your pull request to homekit_python is still pending - any blockers there?
@eugr sorry for the slow response. Your comment pushed me to find a 25th hour in one of my days last week and I added a CLI tool to that homekit_python branch. It would be great if you could find time to try it out. I posted examples of using it on the PR. Right now knowing that those commands work on a diverse range of networks and accessories would be a big help.
@Jc2k I have installed both your branch of homeassistant with HomeKit controller events and homekit_python with aioclient and now everything works instantaneously. Thank you!! I am hoping to see this included in the master branch soon
@rodriguezst thanks for testing. Great to hear. Can you share what devices you are testing with? The biggest "risk" is how it handles inevitable TCP connection timeouts. With the old code they could manifest as about 10 different exceptions and we were still finding new ones. So i'm interested to hear how stable it has been and whether you have seen any errors in your logs after running for a few days.
It would also be really great if you could comment on the homekit_python and let the maintainer know it's working for you in HA. If you are also able to try the CLI commands i posted here and confirm that on the PR too that would be great. Post the devices you tested with there too.
To anyone watching this issue, the biggest thing you can do to help right now is to try the commands here and report how it went and with what devices on that PR.
@Jc2k - that's fine, now I'm being slow... I'm going to give it a try and will report back
@Jc2k - is there any way to make it work with existing pairings from my HA instance? Also, I've noticed that it complains about asyncio.all_tasks which is not supported in Python 3.6. Is it expected behavior? It doesn't seem to affect anything though.
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/eugr/src/homekit_python/homekit/aio/__main__.py", line 464, in <module>
run(main())
File "/home/eugr/src/homekit_python/homekit/aio/__main__.py", line 54, in run
_cancel_all_tasks(loop)
File "/home/eugr/src/homekit_python/homekit/aio/__main__.py", line 21, in _cancel_all_tasks
to_cancel = asyncio.all_tasks(loop)
AttributeError: module 'asyncio' has no attribute 'all_tasks'
I only have 3.7 and 3.8 to hand so hadn't noticed this one :( Thanks for pointing it out. It's only used when exiting so not an issue for the testing I need right now. It's also only applicable to the CLI so not a worry for HA either.
As for reusing your pairings - yes. You need to find your configuration/.storage/
folder and in there there should be a file named something like core.config_entries
? In there you should see a bit of the JSON that looks like:
{
"connection_class": "local_poll",
"data": {
"AccessoryIP": "192.168.1.sss",
"AccessoryLTPK": "sdfsdf",
"AccessoryPairingID": "2fsdfsdfC",
"AccessoryPort": 51827,
"Connection": "IP",
"iOSDeviceLTPK": "dddd",
"iOSDeviceLTSK": "dddd",
"iOSPairingId": "sssss"
},
"domain": "homekit_controller",
.... snip ...
You want to take the bit inside data and make a new pairing.json
like this:
{
"myalias": {
"AccessoryIP": "192.168.1.sss",
"AccessoryLTPK": "sdfsdf",
"AccessoryPairingID": "2fsdfsdfC",
"AccessoryPort": 51827,
"Connection": "IP",
"iOSDeviceLTPK": "dddd",
"iOSDeviceLTSK": "dddd",
"iOSPairingId": "sssss"
}
}
You should now be able to use the get_events/get_accessories/get_characteristic/put_characteristic commands.
@Jc2k - thanks! Works well for me. Tested with Insignia Garage Opener - correctly shows "opening", "opened", "closing", "closed" events even if the door operated manually. Can't wait for it to be merged into official HomeAssistant branch! Thanks for your great work!!!
eugr@hassdev:~/src/homekit_python$ python3 -m homekit.aio get_accessories -f ~/pairings.json -a garage
1.1: >accessory-information<
1.2: () >identify< [pw]
1.3: AC-edited-for-security-reasons () >serial-number< [pr]
1.4: Insignia () >manufacturer< [pr]
1.5: NS-CH1XGO8 () >model< [pr]
1.6: GarageOpener-XXXXX () >name< [pr]
1.7: 2.63.51 () >firmware.revision< [pr]
1.10: >garage-door-opener<
1.11: 1 (prop=Curr_DState) >door-state.current< [pr,ev]
1.12: 1 (prop=Target_DState) >door-state.target< [pr,pw,ev]
1.13: 0 (prop=Obs_Detected) >obstruction-detected< [pr,ev]
1.14: GarageOpener () >name< [pr]
eugr@hassdev:~/src/homekit_python$ python3 -m homekit.aio get_events -f ~/pairings.json -a garage -c 1.11
------------ edit: door closed -----------------
{
"1.11": {
"value": 1
}
}
{
"1.11": {
"value": 1
}
}
{
"1.11": {
"value": 1
}
}
{
"1.11": {
"value": 1
}
}
--------- edit: pushed open button (not on this controller) ------------
{
"1.11": {
"value": 2
}
}
--------- edit: opened ---------------
{
"1.11": {
"value": 0
}
}
{
"1.11": {
"value": 0
}
}
{
"1.11": {
"value": 0
}
}
{
"1.11": {
"value": 0
}
}
------------- edit: pushed button again to close ------------------
{
"1.11": {
"value": 3
}
}
{
"1.11": {
"value": 3
}
}
-------------- edit: closed --------------------
{
"1.11": {
"value": 1
}
}
{
"1.11": {
"value": 1
}
}
^C
Hi
I've seen https://github.com/jlusiardi/homekit_python/pull/154 this is now passing all the checks. Do you have an idea when this can be a part of HA?
I eagerly look forward to the release of an event-based homekit_controller
(versus the current polling-based model). I have four ecobee Switch+ devices and each one is presented as five accessories:
The current version of homekit_controller
reveals the first four of the five accessories.
The 1-minute polling cycle is:
Whenever the revised integration becomes available, I am prepared to test it.
Thanks everyone for your continued interest. I have recently rebased the events branch on aiohomekit, rather than homekit_python. This is a new library so that I can control the release cycle and ensure that I can fix any regressions promptly.
The code is largely the same as before, it is just packaged differently. A large chunk of the change is switching every interaction with every device from using sync code to async code. It's a large change! This needs to happen before events can happen. So the next step is to test the migration to native async code (without events):
The bulk of the work is the transition to a native aio library, when this has landed the remaining diff to add events is in the 100-200 lines region. It bolts on to the existing polling framework quite cleanly. So i will push it to both my HA instances and start trying to solicit wider testing here in parallel. We should be able to submit the second PR a week or 2 after the first.
This means that in a world where I work on HA full time it would still be the end of march for GA at the earliest. And i don't. I barely have any time. So you should expect it to be further away than that.
@tdejneka can you open a new ticket for the occupancy sensor please? It would be great if you could attach the data from .storage/homekit_controller-entity-map
(relative to your configuration yaml). This is a fairly small change I think, if we gave the data.
My 2 environments have been on a branch with aiohomekit and no events for about a week now. I've got 2 bridges - an Eve Extend and a Phillips Hue. Between them I am maybe running with about 25 entities through this integration. There were some errors which i've hopefully resolved, but nothing major. So i've gone and got a PR for this into the queue.
It would be good if a few people could test this in their own environment. In an ideal world it will be a no-op, but it will help make sure there are no regressions on the release with it in. Ideally you'd run with this change for a good few days to look for problems with the connection timing out. In my main environment I have a 2nd checkout of my HA fork:
git clone git://github.com/Jc2k/home-assistant homekit_controller
cd homekit_controller
git checkout homekit_controller_aio
And then because i am using docker-compose I bind mount ~/homekit_controller_aio/homeassistant/components/homekit_controller
into my HA container:
version: '2.3'
volumes:
config: {}
services:
home-assistant:
container_name: home-assistant
build: .
volumes:
- <snip>
- /home/john/homekit_controller_aio/homeassistant/components/homekit_controller:/config/custom_components/homekit_controller
But you could just cp -r
your homekit_controller_aio/homeassistant/components/homekit_controller
into your HA instances custom_components folder.
And then restart your HA. You should see something like this in your logs:
2020-02-23 16:22:35 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for homekit_controller which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
That means you are running my branch.
Obviously the more testing we get done now, the sooner we can roll out events - so thanks for your help in advance!
The PR was merged into dev
, so you can test dev
directly if you are more comfortable with that.
This also means that the events branch is now rebased off dev
as well.
PR for events branch is now up in WIP form here.
The branch was merged into dev
! It's working wonderfully with my Phillips Hue so far.
If i understand right, this update was already released. I am running HA 0.106.5. I've reset and reconnected Aqara Hub through HomeKit controller integration but i see state changes for various motion sensors connected to Aqara with a big delay, much like before. Any ideas?
@sarevok1 - I don't see this commit merged into either master or rc branch, so it's not released to the public yet (unless you run the dev branch)
i'am not. is this the right place to check if it was released: https://github.com/home-assistant/core/releases, PR#32213, then? i am new to this... thanks.
yep, or in the official release notes. It likely won't appear in the official release until at least 0.107.0
@eugr is correct. It was merged into dev
shortly before 0.106.x
was released, but rc
had already been updated for 0.106.x
so it "missed the boat". I'm running dev
homekit_controller on my main home-assistant and motion sensors are just working for me with my Hue bridge and hue motion sensors, there is no need to re-pair when this is released. If your device supports it, it should just start working.
I'm not expecting it to go out before 0.107.0 now.
If you aren't a dev the best bet is the release notes here. I did a bunch of homekit stuff in 0.104.0 as an example - and you can see it by searching for homekit_controller on that post. So the async PR should show up in the same format in the release notes for 0.107.0.
I am expecting #32141 #32158 #32175 #32177 #32188 #32212 #32213 #32228 #32404 #32526 and #32610 to debut in 0.107.0. I'm not sure if stateless remotes will make it into the next release at this point.
awesome; will be eagerly waiting, thank u for your help.
hello @Jc2k , I installed the pre-release and stopped getting any updates from my devices, it this normal?
@Mohannadshariefhome is it a Tado device of some kind?
@Jc2k no it is Xiaomi v3 hub and yes it si confirmed no updates from my devices
@Jc2k no updates from contact sensors, temp sensors or motions sensors, it only gets the latest status before the home assistant update
@Mohannadshariefhome are you able to ping me on discord and try some debug steps with me?
@Jc2k yes sure, what is your ID
@Jc2k I am ready whenever I get your tag
Jc2k#6068
Added @Jc2k
Hey @Jc2k - I'm running 0.107.0 with your asyncio implementation and my HomeKit garage opener (Insignia) doesn't respond to commands or respond to events anymore. I can ping it and I can connect to it using your aiolclient branch of homekit_python library (at least the version I was testing a couple of months back), but I'm getting connection timeouts in HA:
2020-03-18 18:39:44 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 90, in _ensure_connected
await asyncio.wait_for(self.connection.ensure_connection(), 10)
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/core.py", line 1257, in _execute_service
await handler.func(service_call)
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 198, in handle_service
self._platforms.values(), func, call, required_features
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 402, in entity_service_call
future.result() # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 599, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 433, in _handle_entity_call
await result
File "/usr/src/homeassistant/homeassistant/components/homekit_controller/cover.py", line 105, in async_open_cover
await self.set_door_state(STATE_OPEN)
File "/usr/src/homeassistant/homeassistant/components/homekit_controller/cover.py", line 114, in set_door_state
{CharacteristicsTypes.DOOR_STATE_TARGET: TARGET_GARAGE_STATE_MAP[state]}
File "/usr/src/homeassistant/homeassistant/components/homekit_controller/__init__.py", line 98, in async_put_characteristics
return await self._accessory.put_characteristics(payload)
File "/usr/src/homeassistant/homeassistant/components/homekit_controller/connection.py", line 342, in put_characteristics
results = await self.pairing.put_characteristics(characteristics)
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 246, in put_characteristics
await self._ensure_connected()
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 93, in _ensure_connected
"Timeout while waiting for connection to device"
aiohomekit.exceptions.AccessoryDisconnectedError: Timeout while waiting for connection to device
Tried to restart HA, but nothing changed. Hasn't tried to re-pair yet as it's in hard to reach area, but can try tomorrow if needed. Any advise on how to debug it?
Don't try repairing, none of the changes should need that and especially if we had it working before it's probably going to be a small change to get it working. Might be best to spin up a new ticket though.
The timeout is added here:
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/pairing.py", line 90, in _ensure_connected
await asyncio.wait_for(self.connection.ensure_connection(), 10)
So bump that 10 to 30 and see if that helps, and tell me either in a new ticket or on discord.
@Jc2k - thanks, pinged you on Discord
Hi @Jc2k
I'm getting
2020-03-22 12:48:26 ERROR (MainThread) [homeassistant.components.homekit_controller.config_flow] Pairing attempt failed with an unhandled exception
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/homekit_controller/config_flow.py", line 318, in async_step_pair
self.finish_pairing = await discovery.start_pairing(self.hkid)
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/discovery.py", line 61, in start_pairing
await self._ensure_connected()
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/discovery.py", line 47, in _ensure_connected
await self.connection.ensure_connection()
File "/usr/local/lib/python3.7/site-packages/aiohomekit/controller/ip/connection.py", line 230, in ensure_connection
await asyncio.shield(self._connector)
concurrent.futures._base.CancelledError
Any idea how to tackle this?
@stoprocent any chance you could open a new ticket and include all the details listed there. Also try increasing your log level for homeassistant.components.homekit_controller
and aiohomekit
and include the log before the exception not just the exception. And most importantly what kind of device so i know if its something i've already triaged or something different. Also did it work previously? If you are re-pairing something that was previously working that might be a useful diagnostic. Feel free to @ me on new ticket.
@Jc2k I'm not sure if I should :) I already found a solution. The problem was on the homebridge side related to this issue: https://github.com/homebridge/homebridge/issues/2098
When homebridge is dockerized it is advertising wrong ip address. Applying thefix from workaround on this issue made aiohomekit working perfect.
Ah nice! We've always struggled with homebridge support! Any chance you could detail what plugins you are using with homebridge? I'm trying to keep track of known working devices and setups.
I'm using just 2 plugins:
It's a modified version of blinds plugin: https://github.com/stoprocent/homebridge-gpio-blinds
It's something that can easily be done in python for hass but the only nice implementation is done so far only for homebridge: https://github.com/mgcrea/homebridge-tydom
My modified blinds plugin works just great. There is a 0.5 sec delay comparing to using it straight from homebridge.
The second plugin is losing states sometimes, but for what I use it it's functional.
Nice. Thanks for that.
Was this released to the public yet?
homekit_controller has supported events since about 0.107, with some bug fixes in 0.108.
I mean the 1 minute polling delay others mentioned. It seems to still be the case for me in my aqara -> Homekit integration to HA
Everything discussed in this old closed ticket is released to the public and I think has been since 0.107.0. I'd normally say you need to open a new ticket but as it happens, you are probably hitting issue #33511. This is an Aqara specific bug and sporadic. For some users its working fine. For some it rarely works. Please move this discussion to #33511, or if you don't think its the same issue please open a new ticket.
Thanks @Jc2k, will do that!
Most helpful comment
The branch was merged into
dev
! It's working wonderfully with my Phillips Hue so far.