Shelly-homekit: Shelly disconnecting

[CHCHcc]

After successfully pairing shelly 1 with HomeKit, the device disconnects from WiFi and won't reappear. Only cutting power helps to make a new connection. After some minutes up to a couple of hours it disconnects again.

[andyblac]

I would say check your wifi strength in that room.

[CHCHcc]

I have two of them running (rather sleeping). One in the room with the WiFi router and one in the next room. So strength should not be a problem. Since it drops the WiFi connection I can’t go to their IP addresses either.

[tsdrummer]

Do you have running 2,4 and 5 GHZ with the same name / SSID?

[CHCHcc]

Yes, 2,5 and 4 GHz on the same name. But all other devices in my network work flawlessly.

[thp1105]

I have exactly the same issue here. Two shelly 1 connected to WIFI. Working flawlessly with the stock firmware, but when flashing the homekit firmware both start disconnecting at random, only eventually reconnecting. Access to the devices ist possible through browser, even if homekit sees them offline.

[tsdrummer]

Does switching off 5GHz signal solve issue? (Even OEM Firmware works with both)

[kzyapkov]

Activity on the 5GHz band would be irrelevant here. ESP8266 doesn't have a 5.8G transciever.

[thp1105]

I just monitored both Shellys, they in fact stay on 2.4 GHz all the time. Right now one is reactive on homekit, the other is not. Both can be contacted in a browser through their address and acting normally.

[CHCHcc]

I have no connection to the shellys at all. They simply drop out of the network and cannot be reached by browser.

[rojer]

somehow we will need to retrieve console logs from one of the devices experiencing this.
can anyone with this problem connect serial console to one of the switches?

[CHCHcc]

If you give me a detailed guide how to do it, I can try. I just received a USB to TTL adapter today.

[rojer]

http://www.thesmarthomehookup.com/wp-content/uploads/2018/08/diagram.jpg

Do not connect 3V3 and GPIO0 though, you only need RX, TX and GND.
After that, use any terminal program to connect at 115200 baud, 8-N-1.
Log output from the device to file and capture the moment when it becomes unreachable.

[CHCHcc]

I solved the problem by flashing HAA onto my shellys. wasn't as convenient to install as this firmware but it works flawlessly in my network environment.

[rojer]

ok, it may no longer be affecting you but the issue is still there.
can somebody else who is affected please help by providing console logs?

[tsdrummer]

I have no issues, but router is only 8 feet away.

[andyblac]

I have the issue, but it imo it related to WiFi strength, I have my shelly 1 in a Shelly Button, if I put in on the floor next to lamp, it drops off AP, but if I have in on table next to lamp it seems fine.

[Supereg]

Got said issues. Definitely not a connectivity router caused by my network.

This is a serial capture AFTER it was already disconnected and continuously tries to reconnect.
I try to capture know when it actually encounters the disconnect.

mgos_wifi.c:345         WiFi STA: Connect timeout
mgos_wifi.c:88          WiFi STA: Using config 0 (Tasty Door)
mgos_wifi.c:270         WiFi STA: Connecting to Tasty Door
mgos_net.c:86           WiFi STA: connecting
shelly_main.c:254       Uptime: 2011.61, RAM: 32936, 23692 free
shelly_main.c:254       Uptime: 2012.61, RAM: 32936, 23692 free
shelly_main.c:254       Uptime: 2013.61, RAM: 32936, 23692 free
scandone
mgos_wifi.c:119         WiFi STA: Disconnected, reason: 201
mgos_net.c:82           WiFi STA: disconnected

[Supereg]

Additionally I get some errors like the following:

HAPPlatformLog.c:65     IPAccessoryServer Failed to allocate session. (Number of supported accessory server sessions should be consistent with the maximum number of concurrent streams supported by TCP stream manager.)

Seems not related and don't know if that error is somehow a problem

[Supereg]

Did the serial output help?

Okay some theory. Is there the possibility, that this is some issue which is only present when booting up the first time after a firmware upgrade (coming from stock firmware)? 😅
Couldn’t reproduce it yet. Shelly1 is running rock solid (with some reduced response times sometimes) for three days straight 😅

[rojer]

code 201 means it cannot find the wifi network, as in - doesn't see any APs to connect to. this is weird.
unfortunately, there isn't much i can do about it - the wifi machinery is in the libraries provided by Espressif, if it says "not found", i can't do much here...
one thing i have in mind is update to latest version of the SDK and libraries they provide, we haven't done an update for a while.

[rojer]

the sessions problem suggests that you have more than 3 clients trying to conenct to the device. we currently have a limit of 3 simultaneous sessions due to shortage or RAM.

[Supereg]

the sessions problem suggests that you have more than 3 clients trying to conenct to the device. we currently have a limit of 3 simultaneous sessions due to shortage or RAM.

Yeah I know, have seen your Issue at the HomeKitADK repo.

When looking at the HAPIPAccessoryServer inside HandlePedningTCPStream the HAPPlatformTCPStreamManagerAcceptTCPStream is called first. This function (in the default POSIX PAL) returns a kHAPError_OutOfResources if no more TCP session can be opened. And thus fail directly. The Shellies however fail some lines below when the AccessoryServer tries to allocate a new Session. Looking at your init code you correctly set both maxTCPsessions and maxSession (to 3). Haven't looked at your TCPStreamManager PAL however. But seems like there is some mismatch? @rojer

[Supereg]

code 201 means it cannot find the wifi network, as in - doesn't see any APs to connect to. this is weird.
unfortunately, there isn't much i can do about it - the wifi machinery is in the libraries provided by Espressif, if it says "not found", i can't do much here...
one thing i have in mind is update to latest version of the SDK and libraries they provide, we haven't done an update for a while.

That's weird 😅 I can tell you that the network was definitely up and running 😂
But haven't experienced a dropout since I restarted after that dropout. Anyway I can help you debug this further? Or are we just trying to upgrade the wifi stack and hope it's gone?

[ursusHU]

I have the same issue here, after a few days of installation of the Homekit firmware the unit just timed out, won't pair again. If I reinstall the shelly firmware works flawless. For testing I have reinstalled now the firmware 1.8 only on one Shelly-1 the same result now, and I'm not able to connect anymore. MAC address of the unit not found in the network, so it is not a case of a sudden IP change.

[aidanboyce]

Hi, same issue here. I have 2 Shelly-1 devices. 1 connects without issue (and is further away) the 1 I have the problem with (is really close to the wifi access point) connects to an alternative network but not the network I want it to connect to.

Here's the output from the console

[jurban85]

I have the same issue. Have now installed two shelly 1 and both of them randomly disappear from my network. The strange thing is, bothe start to work when I restart my router. It seems that my router is doing some weird channel changes randomly and bother Shellys have trouble to reconnect to my network. Does anybody have the same experience?

[andyblac]

Do you have auto channel setup on your router ? Where the router auto will choose the channel what it thinks it is the best unused Chanel, I’d try disabling as I doubt the Shelly will support roaming that is required on the client for the feature to work correctly.

It sounds like the router is changing Chanel and Shelly is stuck on old channel.

[jurban85]

Do you have auto channel setup on your router ? Where the router auto will choose the channel what it thinks it is the best unused Chanel, I’d try disabling as I doubt the Shelly will support roaming that is required on the client for the feature to work correctly.

It sounds like the router is changing Chanel and Shelly is stuck on old channel.

That’s what I thought as well. I gave my network a fix channel (recommended was 11) and after 30 minutes, one shelly disconnected the other one was still in the network. 🙁 the strange thing is that one might work flawlessly while the other one is offline. And it’s not always the same shelly. Both have the same issue.

[jurban85]

One additional thought, could this issue be caused by my mesh repeaters? I was checking the connection and it seems that they are not trying to access one of those repeaters because they are closer to the router itself. It doesn’t seem that they have trouble with changing access points. Does anybody have similar experiences

[ursusHU]

I think it is somehow related to the network we use(I have a simple router from my ISP, next week I will build my mesh, so I will see if anything changes), but definitely this firmware handles it differently than the original Shellies. With them I have no issues at all independently how I set my network. Currently I have 20 shellies around and 1 has bad coverage, time to time it gets unreachable in HomeKit, but always manages to get back without further actions.

[andyblac]

@jurban85 can you lock device to a particular AP, in Unifi you control minimum RSI of a client that is request for a device to use the AP help stops a client jumping around AP's

[jurban85]

@jurban85 can you lock device to a particular AP, in Unifi you control minimum RSI of a client that is request for a device to use the AP help stops a client jumping around AP's

That's the thing: I can't. My router and its mesh repeater doesn't have this option. I will try to disable the mesh network for a few days, maybe this will help.

[jurban85]

Newsflash. Right now, one of two shellys was offline again. Instead of restarting my speed port pro router, I have switched the WiFi channel from 9 to 1. After I have done this, shelly was online again! This kinda ruins my mesh ap theory but my issue seems to be related to different WiFi channels.

I have turned off automatic channel selection but my problem still happens quite often.

Does anybody have an idea?

[rojer]

@jurban85 not yet. i am listening to all the conversation here, keep the information coming.

[jorgerray]

I have shellys 1 and 2.5 (no blind) with your firmware for homekit that disconnect from my wifi network after restarting the internet network. The only way for them to work again is to disconnect them from the electrical network and reconnect them.
Would it be possible to add a scheduled daily auto reset button at a specific time? Thnaks

[ursusHU]

I have installed the mesh network, one unit has the HK firmware since yesterday. Fingers crossed as currently the new Shelly Homebridge plugin gives only headaches ;)

[felixmueller]

@rojer Since I also have issues with repeatedly disconnecting Shelly 1 devices, what about an optional auto reboot feature as suggested by @jorgerray ?
The implementation could offer a time setting for automatic reboots, e.g. every 24 hours.
This could solve big issues with disconnecting and not reconnecting Shellys which I experience quite often. I have to regularly disconnect my whole place from power to solve this.
Thank you so much!

[rojer]

we can add reboot if wifi connection is not re-established for some time

[felixmueller]

@rojer It depends if wifi disconnect is the only reason for becoming unresponsive. And if the device is always able to detect it and handle accordingly. A timed reboot count solve all other problems, if the device has other issues.

[ursusHU]

Let me share my experience after installing a tplink mesh system. All the units are working now fine since a week. I have the connection problem only with one Shelly-1 PM Which is in the basement (with original firmware no problems were occurring). The funny thing that after firmware change the Shelly is disappearing from the tplink app. It is visible with the original firmware.

[felixmueller]

@rojer After experiencing some more permanent wifi disconnect issues over the weekend, I came to the conclusion that an optional auto reboot feature in case of the wifi not reconnecting (as suggested by you) would be very very helpful in my case!
Is there any chance of implementing it in the near future? Thank you so much for your work and the firmware!

[andyblac]

I had a strange thing happen yesterday, I have 9 Shelly devices a mix of 1's / 2.5's on this firmware and Dimmers / RGBW2 on official via HomeBridge. anyway, yesterday I had to cut power as I am getting work done in the house, when I powered back up, half of the shelly's would not show in homekit nor could I see the WEBUI, but they all had IP address shown on router, I cut the power again hoping it would force them to reboot up properly, but now a different lot would not come online. The devices that had official firmware on then had no issue s both times, they booted up five and both WEBUI and in HomeKit via homebridge.

I tried doing a force 'reconnect' in Unifi software manger on each client, and eventually they all came back online, it almost like HomeKit whilst it is doing it connection thing, is locking up the device, as I have also noticed if you hammer HomeKit, by opening / closing / switching rooms etc, that ask for the device to update it status, it causes the device to drop off too (no connection in HomeKit).

too me it almost if the device can't handle multiple requests to and from homekit in quick succession, and when you have a load of devices all asking to sync with HomeKit, if it does not respond in time the device gets hung up.

so @rojer is there something you can put in the code that if the device has had no response from HomeKit after a request was sent, try to reconnect the Wifi or reboot the device.

I never had any issues when I only had a couple of Shelly's, but as I add more and more, this is what seems to causing the issue on this firmware.

[jurban85]

Guys I wanted to leave some feedback. I have had the same issue, my shellys kept disconnecting after a few hours, the only thing helped was rebooting my router (speedport pro). Meanwhile I have up and went back to original FW. I have also installed the homebridge plugin but it didn’t go well. I decided to rename my 5ghz network. Since the HB plugin did also not work, I gave this custom fw a chance and for whatever reasons my shellys are online for 2 days now. I don’t know if rojer did some magic to the latest release or changing ssids names was the key but it works for me now! Thanks for your input guys I am a happy man now!

[andyblac]

one of my shelly2.5 has dropped off this morning no ip or listed in router, was thinking... we have a reset action of 10 x on/off, could we have a reboot action with 5 x long on/off (1 sec hold each press)

[andyblac]

seem Official Firmware is having same issues ????

Device WI-Fi disconnect - BETA program version 2. 
We make some improvement. 
Please do not use this thread for another issues or anything. I need short and clean answers: Work or NOT
Because tehre is some reports for occasional disconnects from Wi-Fi network, we make firmwares for Shelly1 and Shelly2.5 with workaround fix based on latest firmware version. This is happens for less than 1% of the costumers and it's related to specific router settings or network infrastructure or another Wi-Fi devices connected to the same network. 
Simptoms: Devices lost Wi-Fi connection and need power cycle to connect again. If your devices do not has this simptoms you not need to install the fix. 

If you have such a issue, you can test this fix.
How to install it: 
You need to know device IP. Then execute this command from any browser: 
http://deviceIP/ota?url=URL
where URL is: 

Shelly2.5: 
http://repo.shelly.cloud/firmware/dimo/SHSW-25.zip

You can revert devices any time to standart version. If BETA fix is sucessfull we will implement it in next frimware version for all deveces. 

Please report in this thread. 

[felixmueller]

Just a quick update of my experience with disconnecting Shellys: I have a Mesh WiFi in my home using a repeater in the upper floor. The disconnecting Shellys are more often the ones closer to the Mesh repeater than to the Mesh router. Maybe it is a Mesh issue? I have zero problems with other devices.
I still hope for an auto reboot feature after a specific time or event to solve these issues. Thank you so much @rojer !

[andyblac]

here the link to the FaceBook support group discussing this issue on Shelly 1/2.5
https://www.facebook.com/groups/ShellyIoTCommunitySupport/permalink/2875875842511761/

[felixmueller]

Another update from my side: I have now reverted my most problematic Shellys to the stock firmware. Since then, I have had no more wifi disconnects. I now use the shelly-homebridge plugin. Automations that trigger on switching operations now work reliably, with the HomeKit firmware they unfortunately only did it in half the time.

[75TiagoMatias]

Hi,

Same issues on a Plug-S. From time to time during the day it disconnects and is only available about 5/10min after disconnection.

Already changed Automatic channels to 11 Fixed Channel on 2.4Ghz network... No luck...

When I go to the IP back-end of the device, when it's not available, I can't connect. But when it reconnects, the uptime is running 2/3 days ago... I mean... When disconnects it does not turns off or reboots, only doesn't have access to the network...

[ursusHU]

Another update from my side: I have now reverted my most problematic Shellys to the stock firmware. Since then, I have had no more wifi disconnects. I now use the shelly-homebridge plugin. Automations that trigger on switching operations now work reliably, with the HomeKit firmware they unfortunately only did it in half the time.

I had the same issue, since I updated the wifi network to a mesh network (tplink deco m5), everything works smoothly with the HomeKit firmware as well. I set static ip on the shellies (if that’s matter)

[rojer]

again, apologies for lack of concrete action on this but i am reading all the comments.
i think there are two issues at play here:
1) disconnections due to some sort of bug/issue
2) disconnections due to expensive crypoto + aggressive APs
we definitely need to fix (1), which i think is the most prevalent issue people experience, fixing (2) is more complicated due to non-preemptive sdk we use.
my next step is to try updating wifi libraries to address (1). i'll get on it soon.

[juankibcn]

Same issue here. I’ll wait for your update.

[tdXFS]

I have the same issue, the only way to restore the connection is to reboot the shelly or restart the access point

[meraki-master]

Same issue here. I see a lot of failed authentications in the log when this happens. Rebooting the access point helps from time to time. Power cycling the shelly helps every time.

[rojer]

ok, i updated SDK and toolchain and built Shelly 1 and Shelly 2.5 firmware. please test attached builds and let me know.

(edit: please test 2.0.3 instead, see below)

[andyblac]

loaded it on 2 of my shelly 1's see how it goes, if no issues I'll do the rest of my Shelly's :)

[juankibcn]

I've loaded it on 1 shelly 1 and 1 shelly 2.5
Now i have:

• 1 x Shelly 1 with 2.0.1
• 1 x Shelly 1 with 2.0.2
• 1 x Shelly 2.5 with 2.0.1
• 1 x Shelly 2.5 with 2.0.2
• 1 x Shelly 2.5 with original firmware

Starting the countdown....

[andyblac]

here is official response from Shelly for Shelly 1 / 2.5 they had disconnecting issues in 1.7.x firmware, they fixed it in 1.7.4 here is what they did, @rojer maybe help us ?

June:25 UPDATE: Dear All - this fix looks successfull workaround for costumers which have experiance with disconnecting with their devices. Now we will implemented this fix in all The Shelly devices and next week will release firmware 1.7.4 for all of them. About the fix: We added additional check (ARP Check). If devices lost connection for 1 min we reboot Wi-Fi stack, even that SDKs do not report us nothing. This is workaround for the problem. We working to implement newest Espressive SDK in firmware 1.8 which should fix the real problem, not just to workaround it.

so it seems the latest Espressive SDK has the real fix. @rojer is this what you used in the latest beta for us to test ?

[rojer]

yes, more or less. i mean - not the arp check, just the sdk update.

[juankibcn]

At the moment, I’ve lost the WiFi connection of one of my Shelly with 2.0.1 firmware.
2.0.2 online during 16 hours...
TBC

[billcalle]

I upgraded three Shelly 1s to 2.0.2 firmware. Two of them are still up after 9 hours, but the third one stopped responding sometime after about 6 hours. The one that stopped responding is sitting on my desk powered straight off a cord plugged into the wall and only switching my DMM for test purposes. I checked the web interface and it is also down, so it isn't just a HomeKit non-response thing.

[ursusHU]

After around 30 days connected to my new mesh network, all the units which weren’t restarted since that time suddenly not responding. Firmware 2.0.1., after power cycle everything back to normal. Units are reachable through the web interface.
Just tried the reboot function over the web interface, after reboot the unit is up again.

[ursusHU]

Uptimes of the units which were suddenly dropped (in the morning I forgot to check the others.

[Ryf81]

Good news, I installed 23 Shelly 1 two weeks ago with Firmeare 2.0.1. Most time, the majority of the devices have worked well. But i had still daily need to reboot the wifi since one ore two of them lost connection. And not always the same ones. It changed daily and all of them were affected. I have now updatead all of them to the new Firmware 2.02. Let's see if the new Firmware will solve the issue.

[rojer]

@Ryf81 very good, please keep us posted.

[andyblac]

I have also update all of my Shelly 1/2.5's 8 of them, see how they got but I did not really have the issue before, only once or twice.

[Ryf81]

@rojer , It seems that the issue is still occur. After 3 hours the first 3 Shelly's are not responding.

[meraki-master]

I have updated my two Shelly 1 to 2.0.2. They are up and running for 24 hours now. No issues so far.

[rojer]

@rojer , It seems that the issue is still occur. After 3 hours the first 3 Shelly's are not responding.

ok, good to know.

what kind of wireless etup do you have? what model of router? any repeaters?

[Ryf81]

@rojer , It seems that the issue is still occur. After 3 hours the first 3 Shelly's are not responding.

ok, good to know.

what kind of wireless etup do you have? what model of router? any repeaters?

I use the setup (Router and Range extender) that was provided from my Provider "Swisscom" in Switzerland, and additional a TP-Link Range extender. But i don't know what brand is behind the Swisscom devices. It works with 5 and 2.4 GHz

Here the Links if you're interested in the technical details:

Router (in the ground floor): https://www.swisscom.ch/en/residential/help/device/internet-router/internetbox2.html
Range extender (in the first floor): https://www.swisscom.ch/en/residential/help/device/home-network/wlan-box.html
Range extender (in the basement): https://www.tp-link.com/en/home-networking/range-extender/tl-wa850re/

Edit: Even when i disconnect the Range extenders, some Shelly near the Router lost still connection after a while. That means a Range extender has probably nothing to do with the problem.

[billcalle]

For my three Shelly 1 experiment at 2.0.2 firmware, one dropped off after 6 hours (as noted above) and a second one dropped off after about 12 hours. The third one is still going after 23 hours. Neither of the ones that dropped off are available via their web interface.

From a networking perspective, I have three eero 2 Pros and an eero Beacon. I can provide any other requested network configuration information. As I have three spare Shelly 1s that I haven't installed yet, I'm also happy to perform any relevant troubleshooting, captures of logs, etc. The problem is very repeatable and I haven't seen any version of the firmware go for more than a day without dropping out.

[ursusHU]

From a networking perspective, I have three eero 2 Pros and an eero Beacon. I can provide any other requested network configuration information.

Did you setup static ip on the shellies? Did you have connection problems with the original firmware?

[billcalle]

I haven't setup static ip on the shellies, but I'm happy to do that if we believe it can provide some troubleshooting information. When you say original firmware, are you asking about the native Shelly firmware? If so, then I do not know as I bought these with the intention of using them with HomeKit, so I upgraded to this firmware immediately. In general, I have a very stable network with no connection issues with any other devices.

[juankibcn]

mine with 2.0.2 are still working fine. I have them with static IP

[Ryf81]

I'm also using static IP on all my Shelly's, but i have the issue. So static IP is not the solution for solving the problem.

[meraki-master]

I am using Cisco Meraki MR53, MR72 und MR74 as Wireless Access Points in my setup. Dedicated 2.4 GHz SSID (just for the Schelly´s). The Schelly´s using DHCP, however i reserved them a dedicated IP Address. Up and running for 46 hours now with 2.0.2.

[andyblac]

I am using Ubiquity AP's (various types AC-LITE, AC-LongRange, AC-IW-PRO and 3 x AC-IW) with separate 2.4Ghz SSID for IoT, managed RSSI to keep shelly's connected to the closest AP. the only time I have issues is when I have to power of lights via Consumer Unit, and power back up so times not all shelly's connect so do a force reconnect in the Unifi GUI.

I have been running 2.0.2 for 3 days and shelly's seem solid, I will be powering off my lights again Thursday / Friday to replace a light fitting, so time will tell how they all reconnect.

[Supereg]

Just wanted to throw in a quick feedback. No more disconnects since I installed the 2.0.2 firmware 4 days ago. Running a Fritz.Box mesh system here (quite common here in Germany). Had some disconnects here and there with any previous firmware (though they all reconnected sometime later).
Just running a Test-Shelly here, but with 2.0.2 I could consider actually flashing all the other Shellys laying around (currently flashed with Tasmota).

[AnMiPe]

Maybe the network‘s multicast settings are an issue here? The developers of the homebridge-Shelly plugin mentioned, that a working multicast is very important for the shellies: https://github.com/alexryd/homebridge-shelly/issues/109#issuecomment-630787208

[juankibcn]

I’ve lost all my shellys with 2.0.2. I had some with 2.0.1, 2.0.2 and original firmare. This one is the only is still connected after 4 days update to 2.0.2.
It’s an ubiquity USG3 with UniFi AP.
2,4Ghz only

[rojer]

ok, i have a new build for you to try.

http://rojer.me/files/shelly/2.0.3/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.3/shelly-homekit-Shelly25.zip

this one contains latest 2.2.x SDK from Espressif and also pulls in a more modern LwIP version 2.1.2 (up from 1.4.0).
please test and let me know.

[rojer]

@juankibcn @meraki-master @Ryf81 @billcalle i'm especially interested if 2.0.3 fares any better in your environment.

[juankibcn]

Done

[Ryf81]

@rojer The firmware 2.0.2 is now running well in my network. Since 2 days no problems. After i read this article in the internet i chang the channel from Auto to a fix channel.

https://jlian.co/post/reliable-homekit/

The problem was still there after that, bit much better. Then i replaced the TP link repeater with an additional one from my provider and now it works fine. Now everything in my wifi netwok is in one „wifi cloud“ and appers under the device list in the router settings. Before my router was not showing the devices that were connected to the TP link repeater as my raspberry and several shellys althought they worked and had connections.

It seems that the issue is more related to the network and not to the firmware 👌🏻

But once i‘m back at home next week, i will test as well the new firmware for you.

Send from my iphone

[billcalle]

Okay, I updated all 3 switches and will report back. As for @Ryf81 commenting about the network vs. firmware, I would say that it is likely that network configuration is exposing a problem in the firmware. Obviously these disconnections do not happen for everyone, so there is something situation specific. I do, however, have dozens of other devices that work flawlessly on the same network. I appreciate @rojer not giving up on this and working hard to make this robust for all users.

[rojer]

@billcalle that's right, devices shouldn't go off the network completely. if need be, we will take evasive measures such as restarting wifi subsystem or even rebooting. but i want to make sure we're at least up to date on the sdk and libraries we use before exploring that. 2.0.3 gives us that, if it's not enough we'll move on to implementing workarounds.
that said, the article linked by @Ryf81 made one good point: i have personally seen routers that weren't doing a good job at bridging between 5 ghz and 2.4 ghz networks, that may explain some cases where device is online but unavailable in the home app.

[75TiagoMatias]

Thanks @rojer for the good work here! I really appreciate that!

btw, I think the disconnecting problems are everywhere... I mean... look at last changelog of RavenSystems HAA:

"Networking: Added ping gateway watchdog to detect connection issues. When enabled, device will do an ICPM ping to the gateway each second. If ping fails more that allowed errors in a row, device will start reconnection process."

[billcalle]

Oh oh, that's not good. One of the 3 Shellies rebooted itself and turned off the connected accessory. I noticed because it turned off and went unresponsive in HomeKit. Before doing anything else I went to the Shelly's webpage and it only showed a minute of uptime. It was still unresponsive in HomeKit, so I rebooted it again and then it came back to life in HomeKit. @rojer is there any logging on the device that would be helpful in troubleshooting should this happen again?

[rojer]

there is, but it's all on the serial console. i'll add a small amount of local filesystem logging.

[billcalle]

Darn it, just did it again. I wondered if it was doing it all day long and I just hadn’t noticed because the connected lights were turned off, but on checking the other two Shellies, they are showing a long uptime so I don’t believe they have rebooted. Also, one of them has been on all evening (connected to a fan) and I never saw it go off.

[rojer]

so it's just one of the three, always the same?

[billcalle]

No, in general the disconnect problem has occurred with all 3 of them. Right now the reboot problem is happening with 1 of them. And BTW, it just happened again twice , so it is fairly frequent. I wonder if it is being triggered by a connectivity issue.

[rojer]

yeah, i understand all 3 were experiencing disconnections. but with 2.0.3, only one has been rebooting so far, correct?

[billcalle]

As far as I can tell, yes. The other ones are showing 6+ hours of uptime each.

[mallito1307]

Hello,
2.0.3 is working better for me in a Eero Pro.
20h running
Sometimes when i open home app, shelly is disconnected but a few seconds after it connect by itself

Regards

[juankibcn]

My 6 Shellys are connected and working with 2.0.3

[billcalle]

2 of my 3 Shellies have been up for 42 hours straight. Not a single reboot. The 3rd Shelly reboot many times initially but has now been up for 21 hours. Unfortunately it is up in that it has a reachable web interface, but it is not consistently reachable via HomeKit. I don’t have logs to confirm this, but I believe it is only up because HomeKit gave up on the accessory and stopped communicating with it. When I tried accessing it in HomeKit, it would come up after a few minutes and then reboot sometime later. I have know idea why only 1 of 3 is acting this way. It happens to be immediately adjacent to one that is working just fine.

[rojer]

@billcalle yeah, that's a riddle. can you try rolling back to 2.0.2 or 2.0.1 and then forward to 2.0.3? basically, can you confirm that it's a problem with 2.0.3 and not 2.0.2.
meanwhile, i'm working on an additional fix for mDNS listener and will have 2.0.4 to test soon.

[billcalle]

@rojer, per your request I have downgraded to 2.0.1 on the one unit and will continue to check status. For what it is worth, it has been up for 28 hours on 2.0.3. I agree that this is definitely a riddle. To make sure we get clean data from the 2.0.1 reinstall, I reset the HomeKit settings and deleted and re-added the device. I'll let it run until it either drops out of HomeKit or reboots.

[consoleman]

I see the latest firmware is 2.01?!
You are talking about 2.03 ? Official?

[rojer]

I see the latest firmware is 2.01?!
You are talking about 2.03 ? Official?

i am testing improvements and cutting versions that are not yet official, we are testing them here.
https://github.com/mongoose-os-apps/shelly-homekit/issues/30#issuecomment-653831205

[rojer]

ok, here's a build of 2.0.4:

http://rojer.me/files/shelly/2.0.4/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.4/shelly-homekit-Shelly25.zip

it includes https://github.com/mongoose-os-libs/dns-sd/commit/2b06e716e3c8d5f47df399a537662a000ee7cd42 which i believe could be responsible for some cases of devices disappearing from the Home app while responding to web ui by ip address.

[mallito1307]

Hello,

For my shelly 1 with Eero Pro the behaviour is the same with 2.0.3 and 2.0.4

Sometimes is connected, other times is disconnected.

In most cases it appear as disconnected and after 10-15 second it appear as connected

Regards

[rojer]

In most cases it appear as disconnected and after 10-15 second it appear as connected

do you mean that after a period of inactivity when you open the home app, shelly device is unavailable for the first 10-15 seconds and then becomes available?

[mallito1307]

In most cases it appear as disconnected and after 10-15 second it appear as connected

do you mean that after a period of inactivity when you open the home app, shelly device is unavailable for the first 10-15 seconds and then becomes available?

Exactly

[rojer]

that's new...

ok, can we agree that 2.0.4 is an improvement over 2.0.1? ok to push?

[mallito1307]

For my side is working much better than 2.0.1, so i think you can publish it

[rojer]

published.

[billcalle]

@rojer, I downgraded the one unit to 2.0.1 and never experienced another reboot. Ironically, it also never experienced a problem with HomeKit. The other 2 units that that I upgraded to 2.0.3 never experienced a reboot, and never lost connection with HomeKit. Both of those were running for over 3 days straight. I just upgraded all 3 units to 2.0.4. In general, I agree that you are making improvements. Maybe the one rebooting unit had corruption of the downloaded image? Or maybe there is something else strange going on with the WiFi network that only happens occasionally.

[billcalle]

@rojer, after two days of operation, I can report that 2 of the units are working flawlessly (no reboots and no disconnection from HomeKit). The 3rd unit is generally working well (available in HomeKit every time I've checked) but looking at the uptime statistics every time I have checked indicate that it has rebooted several times during the two day period. For the record, this is NOT the same unit that was rebooting previously when we I was trying the 2.0.3 firmware. In case there is a firmware corruption issue during initial load, I will reload 2.0.4 onto the troubled unit and keep monitoring.

[billcalle]

@rojer, as mentioned above, I re-downloaded 2.0.4 onto the troubled unit. It stayed up all weekend without a reboot. Unfortunately one of the other units started rebooting yesterday. I don't think this is a firmware corruption during download issue. If there are any other suggestions on what I can try or if there is any logging information you want me to pull off of the units, let me know.

[billcalle]

@rojer, not trying to pester you, just providing information in case it helps you think through the issue. I've been closely monitoring uptime reports from all 3 units. At this point, all 3 have experienced random reboots. I went back to 2.0.1, and made it through 24 hours with no reboots on any of the 3 units. Of course it is impossible to know if after 25 or 26 or some number of hours after bringing them up it would have reboot on 2.0.1, but with both 2.0.3 and 2.0.4, the reboots are more common.

[rojer]

@billcalle i remember about your case, it's a weird one for sure. i need to add local logging and sending local logs to a server to get any kind of debugging info... i'll get back to you once it's in place.

[billcalle]

@rojer, don't worry I haven't lost faith, just trying to provide data. To that end, I've been monitoring uptimes and have noticed that there is a statistically significant anomaly. More frequently than can be attributed to random chance, 2 of the 3 units have rebooted within a few seconds of each other. It isn't like the devices reboot many times per minute which would increase the odds of concurrent reboots. Instead, the devices will run for hours (sometimes 2 or 3, sometimes up to 16 to 18) and then reboot. As I said, several times two units have rebooted within a few seconds of each other after one of these really long runs. For clarification, these units are all on different circuits, so it isn't a power-line glitch or the like. I'm speculating that some sort of network traffic or request is being handled in a way that causes a reboot.

I'll wait for your local logging before I bug you further, unless I notice anything else curious.

[rojer]

@billcalle ok, thanks for the info. it's not unprecendented - there have been cases of crashes on unusual network traffic before.
you are positive this wasn't happening with 2.0.1 and started happening with 2.0.2, right?

[75TiagoMatias]

Hi @rojer, I have a slightly different situation.
I have 2.0.4 firmware and its up and running for almost 4 days in the web interface. Sometimes I have a "Not responding" shelly in Homekit only (but Ok with shelly IP and web interface). When that happens sometimes automation just doesn't work because its in "No response" status.
If I wait a little bit, close the Home app and open again, the shelly is connected again. So I think the problem in my case is bad communication with Home app, not about wifi disconnection...

[andyblac]

@75TiagoMatias @rojer I get this too with 2.0.4, when I open the Home app, some shelly briefly show "No Response", and then update ok after a few seconds, others I have too force close the Home app, and re-launch and all come back online, I never had an issue with 1.9.1. I thinking of reverting back to 1.9.1, as most of my automations are not working.

webUI is always reachable.

[Ryf81]

After my vacation, i updated all my shelly from 2.0.2 to 2.0.4. althought the 2.0.2 was running 9 days without any disconnections. I was so happy with that firmware.
Now 2.0.4 is installed, and i have several "no response" in my HomeKit.When i close the app and restart agin, they are response again. But with 2.0.2 i don't had that issue. So i will downgrade again all shelly to 2.0.2

[75TiagoMatias]

@andyblac Exactly that!! Time to time I have COLD water because my water heater doesn't turn on with the automation... I get sooooo mad sometimes... lolol

It's not the wifi connection, it's just the home app connection....

[andyblac]

@75TiagoMatias

@andyblac Exactly that!! Time to time I have COLD water because my water heater doesn't turn on with the automation... I get sooooo mad sometimes... lolol

It's not the wifi connection, it's just the home app connection....

last night I was sitting in the dark watching Netflix, I thought It's dark in here tonight, then realised that my lamps had not turned on. lol, had to open phone and sure enough thad the "No Response" for about 4 secs, then came online, I had to manually turn on my lamps. just goes to show how lazy git's we are all turning into 😄

[rojer]

ok, so sounds like we had a regression in 2.0.3 or 2.0.4.
can you guys narrow it down to specific version that started behaving like this?

[andyblac]

for me after I think it was when I flashed 2.0.4, do not seem to recall having this issue on 2.0.3. but I'm not 100% sure. it defiantly did not have the issue in 2.0.1

[rojer]

so, a reminder (mostly for myself):
2.0.2 - ESP SDK update to more recent, toolchain update
2.0.3 - ESP SDK update to latest + LWIP update
2.0.4 - dns-sd socket code refactor.

[75TiagoMatias]

I only installed 2.0.4, but before at 2.0.1 I was having the issues as well...not the disconnecting from the network but some problem with Home App not responding device...

[billcalle]

@rojer, since there are different problems that we are discussing, I'll try and be detailed about what I've seen:

• 2.0.1: Never observed a reboot of the system. Definitely saw "Not Responding" in the Home app. When "Not Responding", the system was not available via its web page and required power cycling before it would become responsive again. The frequency of "Not Responding" was very high with it being almost guaranteed that at any given time at least 1 of my 3 units was offline.

• 2.0.2: Same as 2.0.1 with the exception that the "Not Responding" happened less frequently.

• 2.0.3: System randomly rebooted. Very rarely saw "Not Responding" or very long "Updating" in the Home App. In either case, the system was still available via its web page and the issue in the Home App self-resolved without any intervention on my part. The frequency of rebooting varied from every few minutes up to dozens of hours apart and might be dependent on network traffic.

• 2.0.4: Same as 2.0.3 with the exception that the rebooting happens less frequently.

Scanning through my supplies, I noticed I have two more, brand new Shellies sitting in their original boxes. I'll go ahead and get these two configured and run them on my desk side by side with my other test Shelly-1. If you would like, I'm happy to deploy different firmware versions to the units and let them sit side by side on the same network if you need comparison testing.

[AnMiPe]

Guys, check your multicast / IPMG Network config! I had Multicast related errors in my network and my shellies disconnected or were not available. After fixing that, I have no problems!

[andyblac]

@AnMiPe just enabled "port isolation" on my switch ports that my AP's are connected too see if it helps. Do you have "Block LAN to WLAN Multicast and Broadcast Data" disable or enabled or your SSID ?

[andyblac]

well that made it worse now all shelly devices that are not on the AP's that my phone is connected too show as "no response"

[andyblac]

ok atm I have IoT VLAN and all my IoT is on that VLAN, I might try to setup a HomeKit VLAN and only have HomeKit clients on that VLAN, it should help reducing broadcast data. anyone else doing this ?

[AnMiPe]

My WiFi Access points (Apple AirPort Extreme) have IGMPv3 Snooping disables, but my Switches „behind“ have all IGMPv3 Snooping enabled.

[andyblac]

My WiFi Access points (Apple AirPort Extreme) have IGMPv3 Snooping disables, but my Switches „behind“ have all IGMPv3 Snooping enabled.

mine is controlled on the VLAN level so have it enabled. thats why I was considering creating a new HomeKit only VLAN and disabling it.

[ursusHU]

To be honest I think your wifi network is too complex. ;)
I have a simple Deco M5 mesh (almost nothing can be set) with network backhaul through ethernet , no separate wifi for anything. Around 20 wifi devices + 20 Shellies. And it works.

[billcalle]

I’m with @ursusHU on this one. I have an eero Pro mesh network which is not configurable with the level of granularity that @andyblac can configure his network. In any case, I have 95 clients on my network, all of which work well with the exception of these Shellies.

[AnMiPe]

But maybe your 95 client don’t rely heavily on Multicast to report back their status…

[billcalle]

I’m sure that is true of many of them, but a lot of them are HomeKit devices that seem to be working well.

[rojer]

ok, here's a build with more debug enabled:

http://rojer.me/files/shelly/2.0.5/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.5/shelly-homekit-Shelly25.zip

NB: this build will send debug logs to my server and allows remote RPC access (to collect logs and crash dumps), please only flash to devices that are not critical.
@billcalle i am especially interested in getting one on your network, i hope to capture a crash dump (assuming reboots are crashes, which they most likely are).

[rojer]

also, does anyone experiencing the issue have more than 3 devices that the home is shared with? currently firmware cannot handle more than 3 connections, i wonder if that could be the reason.

[flying-stefan]

also, does anyone experiencing the issue have more than 3 devices that the home is shared with? currently firmware cannot handle more than 3 connections, i wonder if that could be the reason.

unfortunately yes. the system with 2.0.4 works pretty fine when I'm alone (2 iPad, 1 iPhone, 1 Mac). As soon as my girlfriends joins the system with her iPad and iPhone on top, the system totally becomes unreliable. It works or not. Some Shellys get lost totally, some continue working properly. A restart of my wifi (FritzBox) has always solved the problem yet and all Shellys get connected again. cheers.

[andyblac]

@rojer the only issue I have atm, is that for example if I have automation action linked to a switch, the automation will not run until I open the home app.

ie.
I have a shelly2.5 in bedroom, SW1 is hardwired to the Light, and SW2 is not hard wired to anything, but instead is linked to an automation where IF SW2 is ON, open bedroom smart shades, IF SW2 is OFF close bedroom shades. But, the automation will not run unless Home App is open and on screen.

BUT I have another automation on Bathroom Shelly2.5 where SW1 is hard wired to light, and SW2 is hardwired to FAN, and I have an automation, where if the humidity increases it turns ON the SW2, and turns OFF SW2 when humidity decreases, this automation works NP.

so it seems automations that use a Shelly as the TRIGGER device don't seem to work, but do work when the Shelly is the ACTION device.

[rojer]

@andyblac please deploy 2.0.5 on a problematic device, it has connection logging patch, as well as dns-sd debug logging that should give some clarity. please let me know your device id (shown on the web ui) so i can trace it in the logs.

[andyblac]

@andyblac please deploy 2.0.5 on a problematic device, it has connection logging patch, as well as dns-sd debug logging that should give some clarity. please let me know your device id (shown on the web ui) so i can trace it in the logs.

ok, will do, I'll report back.

edit:
I have loaded it on, I'll wait a while before testing automation, as it seems to have to timeout, ie it looses connection to home app.
ID = 8CAAB505B323

[rojer]

@andyblac yes, you are definitely hitting the connection limit:

[Jul 20 15:13:24.037] shellyswitch25-8CAAB505B323 254 1595258004.421 2|HAPPlatformTCPStream:43 0x3fff7f0c MG_EV_ACCEPT HAP connection from 192.168.3.9:50858
[Jul 20 15:13:24.037] shellyswitch25-8CAAB505B323 255 1595258004.429 2|HAPPlatformTCPStrea:163 0x3fff7f0c Accepted HAP connection from 192.168.3.9:50858
[Jul 20 15:13:24.037] shellyswitch25-8CAAB505B323 256 1595258004.449 2|HAPPlatformLog.c:65     IPAccessoryServer Failed to allocate session. (Number of supported accessory server sessions should be consistent with the maximum number of concurrent streams supported by TCP stream manager.)

[rojer]

i see 7 distinct clients trying to connect while we can only support 3 sessions. ok, this raises the priority of item 3 on this list - https://github.com/mongoose-os-libs/homekit-adk#todo.

[andyblac]

@rojer just pressed the switches no automation

regarding limit, no Home app are open on any devices, only thing I can think of is the Hubs, I have 2 HomePods, 2 AppleTV (hub disabled on those) 2 iPhones (no Home App open), 1 AppleWatch, 1 iPad, (no Home App open), MacBookPro (Home App not open)

so it seems HomeKit is using all those devices even though none was in use at the time.

[billcalle]

@rojer, I loaded the new update on the following devices:

shelly1-BA257B
shelly1-B9F4CD
shelly1-C48509

I have HomeKit Accessory Security enabled, but I set the Allterco Switches to No Restriction. Please let me know if you have problems connecting to the switches and I can just disable HomeKit Accessory Security altogether.

As for devices that my Home is shared with, I have it shared with all 4 family members. That translates into the following devices that can access HomeKit:

• 4 iPhones
• 4 Apple Watches
• 1 iPad Pro
• 1 iMac
• 2 Macbook Pros
• 1 Macbook
• 5 HomePods
• 3 AppleTVs

Like @andyblac, I would say that almost none of those devices actually have the Home App open and running. If the HomeApp is performing background updates, however, than the device limit mentioned by rojer would be a problem that I'm surprised more people haven't run into.

[andyblac]

@rojer in the short term, is there a way we could have a priority list so that Hubs, master iPhones etc get priority ?

[rojer]

@billcalle thank you, i can see logs from the devices and can reach them via RPC. there's an absolute storm of activity, i'll try to sift through it later. the total number of devices in your setup is 21. i see connection attempts from 14 unique addresses.

so, HAP spec requires device to support at least 16 connections, doesn't say anything about maxuimum. with the memory budget we have, supporting that kind of concurrency will be tricky...
the way HAP works is that connections are basically never closed once established, even if sitting idle.

in the short term, is there a way we could have a priority list so that Hubs, master iPhones etc get priority ?

the problem is that we don't have a way to tell "important" connections from less important either.

to give you an idea: at the moment we have around 19K free when idle and we need about 8 to be able to OTA and do other things. this will be a tough one to solve. but i like that we seem to be zeroing in on the problem, thank you for your patience!

[andyblac]

@rojer if you need anything else, or wanted something testing just ask, I have spare Shelly1 & 2.5.

[billcalle]

@rojer, I added the other two test Shellies to the network and installed 2.0.5:

shelly1-C41C8B
shelly1-B8F5CD

These two, along with shelly1-C48509 are just sitting on my desk connected to switches and a DVM. Feel free to reset them, try new code on them, etc. shelly1-BA257B and shelly1-B9F4CD are connected to real devices. Snoop around all you want, but one is currently controlling a couple of hundred cafe lights in my back yard and the other is an outdoor fan.

As for the HAP spec requiring a device to support at least 16 connections, I assume you mean 16 pairings. They do give you an out, by allowing you to return kTLVType_State kTLVType_Error kTLVError_MaxPeers. While having support for less than 16 pairings would prevent the device from getting certified as HomeKit compatible, I think the system would still work, although it might be a bit slower to respond when brining up the Home App.

[petergPRG]

Even though it seems to make things a little more complicated. I have five Shelly 1. All were configured with the HomeKit FW and had different software versions. I kept getting "no answer" on all of them. With SW 1.3 this jam stayed longer, from SW 2.01 it got better and mostly disappeared when I switched to the room in the Home App.
Since five days I've been using the original Shelly FW (1.7) on the Shellys and running it over the Shelly Binding in OpenHab. Then I access them via the OpenHab HomeKit plugin.
I now have no more failures. All shellsys react immediately and are always available.
I also suspected the Apple Bridges. I have 1 x Apple TV and three HomePods running.
Maybe this is helpful

[rojer]

@billcalle thank you, this was very helpful. I believe I have identified and fixed a bug that was causing bad behavior when connection limit is reached. With this change, at least it should behave sensibly. We should still increase the number of simultaneous connections we support (the spec requires at least 8 for IP accessories, that should be doable) but with the above fix there should already be an imporovement. I've deployed it to shelly1-C41C8B and shelly1-B8F5CD already, here are non-debug builds (no logging / phone home), please test:

http://rojer.me/files/shelly/2.0.6/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.6/shelly-homekit-Shelly25.zip

debug builds:

http://rojer.me/files/shelly/2.0.6-dbg/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.6-dbg/shelly-homekit-Shelly25.zip

[billcalle]

@rojer, okay I've deployed 2.0.6 (build 20200722-195911/2.0.6-g8373630-master) to:

shelly1-C48509 (Test1)
shelly1-BA257B (Back Yard Cafe Lights)
shelly1-B9F4CD (Back Yard Fan)

I'll monitor and let you know if I have any other issues.

I'm very excited that these are getting closer and closer to working flawlessly. If these work well I'm going to deploy about 50 of them. I've tried several commercial switches (Leviton, WeMo, etc.) and they all have issues of one sort or another. The Leviton switches are the absolute worst (completely unreliable and periodically stop working as actual switches if they receive switching noise from adjacent switches). The WeMo switches are absolutely rock solid and work flawlessly, but lack in configurability. They have a very bright LED on them that is always on and is just too disruptive. In any case, thank you for your hard work on this project. I believe it is the optimal solution once the reliability is nailed down.

[Marfre888]

@rojer Something I've come across to substantiate what you are saying, on the Shellies is if I ping them while connected to Homekit they time out every 30sec to 1min. However, if I remove them from Homekit they never timeout (with the same firmware)

Strange, shouldn't only the hub just be communicating with it if one is present?

[andyblac]

@rojer ok just loaded 2.0.6 debug version of fw, one thing I noticed, after device rebooted I got Connections: 0/0/3 pending/active/max so I rebooted my master HomePod hub, hoping it would be the 1st to connect, but after it rebooted, the shelly still reports Connections: 0/0/3 pending/active/max why is the HomePod hub not connecting ?

edit:
after I launch Home app on my iPhone, I get Connections: 0/1/3 pending/active/max so, it seems the HomePod is not trying to connect, and by the time it dose there are no connections left ???

edit2:
ok, I have managed to force the HomePod to connect to the shelly by, opening an automation that uses the shelly, and press Test this automation and I now have Connections: 0/2/3 pending/active/max so it seems the HomePod is now connected, I will keep an eye on the connections to see if it drops.

edit3:
even with these 2 connections, when I turn on the SW it still dose not run the automation, unless that Home app is open.

could you add to the gui what host names or IP's that have a connection it would help us debugging.

[rojer]

@andyblac i see the following clients regularly connecting to shellyswitch25-8CAAB505B323:
192.168.3.8, 192.168.3.9, 192.168.3.14, 192.168.3.35
conenction management logic seems to work properly - existing connections get evicted when necessary.
192.168.3.14 behaves strangely: sometimes it will try to open a second connection when one already exists. what device is 192.168.3.14?

[andyblac]

@rojer .14 is a family members iPhone, they where out at the time I flashed the device, they rarely open the Home app, but is is involved in some automations, (if it leave's the home homekit turn's off power sockets, heating to that room etc, and reverse when it arrive home)

.8 = my MacBook Pro
.9 = the main HomePod hub.
35 = my iPhone.

it is strange that automations still do not work whilst a shelly is the TRIGGER device, unless the Home app is open.

I have even tried opening the webUI and turn on the SW2 and the HomeKit automation dose not run.

[andyblac]

@rojer can you try setting up an automation using a Shelly as a TRIGGER, see if it works for you.

1: setup the automation using a SHELLY SW as the TRIGGER.
2: close Home App on iPhone / iPad
3: wait 20 mins
4: try switching the manual switch or in the webUI try turning on the SW, see the the automation runs.

here is my automation

I have a 2nd automation for the closing using the Shelly OFF as TRIGGER, whilst I wait for detached mode to be implemented.

[rojer]

it is strange that automations still do not work whilst a shelly is the TRIGGER device, unless the Home app is open.

it's not, actually. the only way to communicate from device to controller is via an existing open connection. controller is responsible for keeping a connection open, if there is no connection event notifications from device won't work.
at the moment we cannot support all the clients connected at the same time, we can only support 3 (and actually separate bit of logic in homekit library will actively try to keep one available slot, so it will get down to 2 after 60 seconds of inactivity, i'll disable that bit).
i kind of hoped that there will be a slow rotation of connections will ensure all controllers will get a chance to communicate, but it doesn't seem to be what's happening. what i see in the logs are bursts of activity rather than a slow churn of connections.
we need to support more simultaneous connections, that's for sure.

[andyblac]

@rojer so is there no way we could manually add a IP in webUI homekit part to keep poling a chosen IP aka the master Hub, that way it is always connected, so polling it once every 50 seconds should keep that connection open ?

[andyblac]

it's not, actually. the only way to communicate from device to controller is via an existing open connection. controller is responsible for keeping a connection open, if there is no connection event notifications from device won't work.
at the moment we cannot support all the clients connected at the same time, we can only support 3 (and actually separate bit of logic in homekit library will actively try to keep one available slot, so it will get down to 2 after 60 seconds of inactivity, i'll disable that bit).

so if this is the case how would detached mode work once you have implemented it, if there is no connection and we turn on the switch it will not tell HomeKit to do the action ?

[rojer]

keep poling a chosen IP aka the master Hub, that ways it is always connected

the peripheral (device) has no choice in the matter, it can only accept incoming connections but no way to initiate them.

so if this is the case how would detached mode work once you have implemented it, if there is no connection and we turn on the switch it will not tell HomeKit to do the action ?

when there's no way to support all the connections at the same time, i guess the answer is - pretty badly...

P.S. i took the liberty to update shellyswitch25-8CAAB505B323 with a new firmware, it should support 3 persistent connections at the same time as it's supposed to and not have the 3rd evicted after it's been idle for 60 seconds.

[andyblac]

the peripheral (device) has no choice in the matter, it can only accept incoming connections but no way to initiate them.

hmm, so we have no way, to make sure the HUB is always connected. I wonder how official peripherals manage to keep connections alive.

[andyblac]

@rojer just tried my SW again, and is dose seem to be working, if I keep devices that open Home app to a minimum. so you are correct what your saying, we just need to support more connections, but it would be handy if we could have a white list, so any IP in the white list the shelly would try to keep those connections open.

how dose the shelly decide what connection to drop ?, dose it just drop the oldest connection 1st ?

[rojer]

[0/2/3]

there have only been connection requests from only 2 controllers since reboot, 192.168.3.9 and 192.168.3.35:

[Jul 23 15:42:55.848] shellyswitch25-8CAAB505B323 163 1595518974.862 2|HAPPlatformTCPStrea:119 0x3fff7354 192.168.3.35:53876 Incoming HAP connection, ns 0/0/3
[Jul 23 15:42:55.848] shellyswitch25-8CAAB505B323 164 1595518974.872 2|HAPPlatformTCPStrea:270 0x3fff7354 192.168.3.35:53876 Accepted HAP connection, ns 0/1/3 ts 0x3fff7034
[Jul 23 15:45:55.088] shellyswitch25-8CAAB505B323 217 1595519154.092 2|HAPPlatformTCPStrea:119 0x3fff749c 192.168.3.9:51869 Incoming HAP connection, ns 0/1/3
[Jul 23 15:45:55.088] shellyswitch25-8CAAB505B323 218 1595519154.103 2|HAPPlatformTCPStrea:270 0x3fff749c 192.168.3.9:51869 Accepted HAP connection, ns 0/2/3 ts 0x3fff72dc
[Jul 23 15:56:24.633] shellyswitch25-8CAAB505B323 412 1595519784.616 2|HAPPlatformTCPStrea:104 0x3fff7354 192.168.3.35:53876 HAP connection closed, ns 0/2/3
[Jul 23 15:56:26.169] shellyswitch25-8CAAB505B323 414 1595519785.131 2|HAPPlatformTCPStrea:119 0x3fff7bbc 192.168.3.35:53952 Incoming HAP connection, ns 0/1/3
[Jul 23 15:56:26.169] shellyswitch25-8CAAB505B323 415 1595519785.141 2|HAPPlatformTCPStrea:270 0x3fff7bbc 192.168.3.35:53952 Accepted HAP connection, ns 0/2/3 ts 0x3fff7034

I wonder how official peripherals manage to keep connections alive.

by, like, not dropping them? having actually enough memory to keep them alive?

how dose the shelly decide what connection to drop ?, dose it just drop the oldest connection 1st ?

yes. we keep track of the time of the last request from the device and when choosing which one to drop will choose the one that has the longest time since the last request.

[andyblac]

@rojer so can we have a box to add a Hub IP, so once it gets a connection it dose NOT get dropped, it will drop other connection 1st.

[rojer]

that's a possibility. still, it's fragile - ip addresses can change, and then suddenly things will break.
before going that route, let me see if i can squeeze the library to just support more connections.

[juankibcn]

Hi
@rojer I have 5 shellys with 2.0.6
Two of them doesn’t work

[rojer]

@juankibcn how exactly? is web ui accessible?

[andyblac]

that's a possibility. still, it's fragile - ip addresses can change, and then suddenly things will break.
before going that route, let me see if i can squeeze the library to just support more connections.

my HomePod hub, and all HomeKit devices are always on reserved IP's.

[juankibcn]

@juankibcn how exactly? is web ui accessible?

Uppss
Don’t worry. It was switching on with 0% light
😜

[billcalle]

@andyblack, I'm with @rojer on this one. He's looking for a generic solution that will work for everyone. Your solution will solve your specific problem but doesn't help others as not everybody has static IP addresses (nothing wrong with that, just not something that people do unless they are trying to solve a very specific problem). The change you are looking for is not difficult to implement. Perhaps you should fork the code and put in the fix you are interested in until rojer is able to resolve the connection count issue. The current implementation is not HomeKit compliant, so trying to get there first is the best course of action.

[andyblac]

@billcalle ye, I understand that, it would just be short term fix, and it would be a quick fix (if I knew C++), and it should not effect others if the field in webUI was left blank.

[rojer]

Squeezed input buffers (1, 2), got 3 additional sessions out of it - we're up to 6 max now.
deployed to shelly1-B8F5CD shelly1-C41C8B and shellyswitch25-8CAAB505B323.
both maxed out the connections very quickly... @billcalle and @andyblac please let me know if it's any better.

non-debug builds:
http://rojer.me/files/shelly/2.0.7/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.7/shelly-homekit-Shelly25.zip

debug builds:
http://rojer.me/files/shelly/2.0.7-dbg/shelly-homekit-Shelly1.zip
http://rojer.me/files/shelly/2.0.7-dbg/shelly-homekit-Shelly25.zip

[billcalle]

@rojer, I deployed the non-debug builds on the other 3 switches so that I can track those as well. Everything seems to be working, but I concur that it almost instantaneously maxed out the connections. Responsiveness doesn't seem to be suffering, but I'll let you know how we hold out.

[andyblac]

@rojer thanks for your work on this issue I think I can say for all of us that we all really appreciate it, and yes, that update has made things better, I must say I find it strange on Apples part that if a 'hub' is being used, all control devices (iPhone, iPads, Apple Watches, etc) still connect to the HomeKit devices direct, I would have thought that IF a home had a 'HUB' that it would be better that it became the master controller and all other controllers connected to it rather than direct, but we are where we are I guess.

with the new FW my wall switch open my shades np for the 1st time since installing it.so thanks, and as @billcalle it dose not seem to be suffering any ill effects, but, I'll keep an eye on it for a day or 2, then I'll flash the rest.

[Marfre888]

@andyblac Yeah, it's very strange that apple opted for each device to independently connect to the device separately, but I'd guess that gives you a level of redundancy should the hub fail

@rojer Would it be possible to get a version for the '1PM'. I would love to test this out, as it's currently quite annoying having devices not turn on or remain on after an off trigger.

[rojer]

@Marfre888 here you go - http://rojer.me/files/shelly/2.0.7/shelly-homekit-Shelly1PM.zip

[Marfre888]

Thank you

EDIT:
@rojer Happy to report that I've been running this for a day, and I have not had a single 'no response' or automation failure.

I've removed some of my family members from the Home to be sure we don't go over the 6 connections currently allowed.

It would be awesome if in the future you could figure out how to squeeze more connections out of the shelly or add a priority list, but at least for now it works well enough.

Thank you so much..

[juankibcn]

Just now, no device has worked. I have had to open the home app and since then the automations have started again.

[rojer]

Just now, no device has worked. I have had to open the home app and since then the automations have started again.

how many active connections do you see in the web ui?

[andyblac]

@juankibcn yeah, increasing the max connections has helped massively, but I have noticed that a connection is not limited to a single device, ie if you use 3 different HomeKit app (Eve, Home, myHome, Home+, Controller, etc, etc) there all use a connection to a device.

so, try force quitting ALL app's on your iPhone, and only use them as required.

@rojer I really do think having a White list, let's say with a MAX of 2 IP's, that should NEVER gets closed, so for example Master Home Hub and the master iPhone (one connection only) in the Home. this would help with automations, even if you could increase to 10 max connections, this limit could still easily be reached.

[billcalle]

@rojer, when you have some time, you should check the logs on my devices. Judging from the uptime discrepancies across all 5 devices I can tell we’re still getting reboots. I have also seen some not responding messages in the home app. I checked the web ui one time that this happened and sure enough the count was 5/6/6.

[juankibcn]

Just now, no device has worked. I have had to open the home app and since then the automations have started again.

how many active connections do you see in the web ui?

Sorry, witch web ui? my router? Each Shelly IP?

[juankibcn]

yeah, increasing the max connections has helped massively, but I have noticed that a connection is not limited to a single device, ie if you use 3 different HomeKit app (Eve, Home, myHome, Home+, Controller, etc, etc) there all use a connection to a device.

so, try force quitting ALL app's on your iPhone, and only use them as required

I did some automated rules on Home App for Shellys, and I want to use it without the App, with switch. Some people (parents...) don’t use app.

[andyblac]

@juankibcn 1st make sure you have 2.0.7 fw version (see links above in this thread) then go to the webUI of the shelly in the HomeKit section you should see something like:

you see in this above screen shot I have maxed out all connections, even though NO controller device (iPhone, etc) has an HomeKit app open, it seems ALL HomeKit capable devices try to connect to each shelly device so it is ready to control as / when required by the user, it dose not just seek a connection open app open, the OS seeks a connection itself, BUT all any additional APP on a controller seek an additional connection, so if you open say Home App on an iPhone that seems to require 2 connections, each Home Hub requires a connection too, unless you have it disabled in GUI (aka AppleTV), but as you can't disable Hub support on HomePod's that will mean every HomePod you own seeks a connection. But I have also noticed that even if the slave Hubs have a connection, but IF the master dose not have a connection automations will fail to run.

[andyblac]

@rojer could you update the FW to list IP's so we can we try to workout what devices are hogging connections ?

[75TiagoMatias]

@rojer could you update the FW to list IP's so we can we try to workout what devices are hogging connections ?

Good idea, if possible!

@rojer Do you have a PlugS 2.0.7 as well?

Thanks

[juankibcn]

@juankibcn 1st make sure you have 2.0.7 fw version (see links above in this thread) then go to the webUI of the shelly in the HomeKit section you should see something like:

you see in this above screen shot I have maxed out all connections, even though NO controller device (iPhone, etc) has an HomeKit app open, it seems ALL HomeKit capable devices try to connect to each shelly device so it is ready to control as / when required by the user, it dose not just seek a connection open app open, the OS seeks a connection itself, BUT all any additional APP on a controller seek an additional connection, so if you open say Home App on an iPhone that seems to require 2 connections, each Home Hub requires a connection too, unless you have it disabled in GUI (aka AppleTV), but as you can't disable Hub support on HomePod's that will mean every HomePod you own seeks a connection. But I have also noticed that even if the slave Hubs have a connection, but IF the master dose not have a connection automations will fail to run.

Hi,
I had 2.0.6
now with 2.0.7
Could you tell me about what 0/3/6 pending/active/max means?
I have 5 Shellys
4 of them: 0/3/6 pending/active/max
1 of them: 0/1/6 pending/active/max

[andyblac]

Hi,
I had 2.0.6
now with 2.0.7
Could you tell me about what 0/3/6 pending/active/max means?
I have 5 Shellys
4 of them: 0/3/6 pending/active/max
1 of them: 0/1/6 pending/active/max

basically means you're not maxing out the shelly available connections, so everything should be working.

[rojer]

pending are incoming HAP connections that have not been accepted yet, active are accepted connections, max is maxim,um number of active connections.
0/N/N means a maxed out server, M/N/N means maxed out server with even more clients trying to connect.

here's a new builds:

https://rojer.me/files/shelly/2.0.8/
https://rojer.me/files/shelly/2.0.8-dbg/

i added a simple debug page, it can be found at /debug/ (http://A.B.C.D/debug/, A.B.C.D is shelly ip) and will list HAP connections.

@rojer Do you have a PlugS 2.0.7 as well?

yes, i have full set of builds, find one at the links above.

[andyblac]

@rojer just getting "not found" in page, loaded debug ver on that shelly.

[rojer]

@rojer just getting "not found" in page

make sure you add that slash at the end, it's important: /debug/, not just /debug.

[andyblac]

@rojer just getting "not found" in page

make sure you add that slash at the end, it's important: /debug/, not just /debug.

ahh got it, Safari removed the trailing slash on 1st load.

[andyblac]

so IMO, if you have more than 6 HomeKit controllers, they will be fighting for a connection, so I do really think we need a way to tell the shelly what the IP of the master Home Hub is, so that connection never gets dropped and thus automation stop working.

@rojer is this something that you could add ?

[billcalle]

@andyblac, you keep asking for a very specific solution that won’t benefit all users. For example, I have 5 HomePods and 3 AppleTVs. Even if I disable all 3 AppleTVs, I’m left with 5 devices capable of being “Master Home Hub”. From watching the Home App, I can tell that this responsibility is not fixed to a specific HomePod, but rather rotated between devices (I don’t know what the logic HomeKit uses for assigning this responsibility is). Mind you, I also don’t have fixed IP addresses configured, so that would also conflict with your proposal. I think it would be best if we try to understand how the connection handling mechanism is intended to work on a device with limited time sources. I have plenty of other devices in this same HomeKit environment that work just fine. I am confident that we can as well without needing to support an unlimited number of connections.

[andyblac]

@andyblac, you keep asking for a very specific solution that won’t benefit all users. For example, I have 5 HomePods and 3 AppleTVs. Even if I disable all 3 AppleTVs, I’m left with 5 devices capable of being “Master Home Hub”. From watching the Home App, I can tell that this responsibility is not fixed to a specific HomePod, but rather rotated between devices (I don’t know what the logic HomeKit uses for assigning this responsibility is). Mind you, I also don’t have fixed IP addresses configured, so that would also conflict with your proposal. I think it would be best if we try to understand how the connection handling mechanism is intended to work on a device with limited time sources. I have plenty of other devices in this same HomeKit environment that work just fine. I am confident that we can as well without needing to support an unlimited number of connections.

I get that I have 2 Home Pods myself and iPads etc, they all can be 'Hub' but only one is a 'Master', and TBH with the limited resources the shelly has there no way it can handle the amount of 'controllers' that HomeKit is designed to run on, that's why if not officially supported, so no matter how much work @rojer puts in IMO it will never handle ALL connections all at once, so you will always have controllers fighting for a connection, and with that the possibility 'Master Hub' gets dropped, and from what I understand about HAP if the Hub is seen on the network it will NOT pass automations to the 2nd Hub that may have a connection to shelly.

so IMO the resource management is always going to have to be handled by user.

[andyblac]

From watching the Home App, I can tell that this responsibility is not fixed to a specific HomePod, but rather rotated between devices (I don’t know what the logic HomeKit uses for assigning this responsibility is).

the only time the 'master hub' get replaced with another is if it goes offline. ie no power, or wifi connection gets dropped. or software update, thats why after a power cut or software update I checked to make sure my preferred 'Master' is indeed the master.

as I said IMO it all about resource management. lets you have a large family each with a iPhone , or you have lots and lots of 'controller' devices what then ? how can you tell shelly what to give priory too ?

I was just saying that we need a way to tell the shelly "this device(s) gets priory over everything else".

[andyblac]

from what I can see the master hub dose try to keep the connection alive (once it has one)

Config number: 8
HAP connections:
  192.168.3.10:60420 last_io 4472
  192.168.3.14:49265 last_io 5214
  192.168.3.18:56743 last_io 5224
  192.168.3.7:50222 last_io 5222
  192.168.3.9:52883 last_io 350
 Total: 5

192.168.3.9 is my master hub, 192.168.3.7 is a Standby Hub,

[billcalle]

@rojer, I think there is a bug in the connection management. On at least two devices, I’m seeing 0/6/6, but only 5 IPs are being listed. Since the two stats are being listed on different pages, I thought it could be a timing issue, so I sat and refreshed each page in two browser tabs as quickly as possible. While I was seeing the last_io value update, there was no change in either the connection statistics or the IP address list. I have the debug build installed on shelly1-C41C8B which is exhibiting the problem if you would like to connect yourself and investigate.

@andyblac, in checking all 5 devices, I have never seen a single IP listed twice despite intentionally running several HomeKit apps from my iPhone. How did you determine that multiple apps on a single device each open their own unique connection? Wouldn’t each app access HomeKit devices through the framework on a given device thus utilizing the existing open connection? Did you see this documented somewhere or does your router show a different view into connections that might be useful resolving this issue?

[andyblac]

@andyblac, in checking all 5 devices, I have never seen a single IP listed twice despite intentionally running several HomeKit apps from my iPhone. How did you determine that multiple apps on a single device each open their own unique connection? Wouldn’t each app access HomeKit devices through the framework on a given device thus utilizing the existing open connection? Did you see this documented somewhere or does your router show a different view into connections that might be useful resolving this issue?

it was an assumption, that why I asked for the debug page, as I like you saw that 0/6/6 was shown but I had only 3 devices powered up at the time (master hub, my iPhone, and a family member iPhone).

maybe the debug page only showing a single line per IP's but it could be that multiple connection are on a single IP ?

[75TiagoMatias]

Welllll..... I would say that, in my case with 5 devices and 5 active connections, this FW version works for me. The problem of No Answer in homekit was indeed the 3 limited number of connections!! I will try one day or two more.. but for now its all good :)))) thank you all!! ehehe :)

[rojer]

@billcalle thanks for letting me know! you are right that number of HAP sessions should match the connection listed on the debug page. i have a tentative idea of how the discrepancy might happen, i've built 2.0.9 with some additional debug information that should help me pinpoint the reason and deployed it to shelly1-C41C8B.

[rojer]

2.0.10: extended debug info, fixed a bunch more issues with connection management.

https://rojer.me/files/shelly/2.0.10/
https://rojer.me/files/shelly/2.0.10-dbg/

@billcalle again, access to your network has been immensely useful. it's like a natural DDoS test :)
i think 2.0.10 should fix the connection inconsistency and occasional reboots that we were seeing. i deployed to the two test devices, will monitor.

[billcalle]

@rojer, thanks for your efforts. Good luck debugging and let me know if I can do anything to help.

In the meantime, I took some time this weekend to read through the entirety of the HomeKit Accessory Protocol Specification (Non-Commercial Version, Release R2). My next step is to download the code and start working my way through it. I'll try and keep my questions to a minimum so that you can focus on coding and not answering questions. I will point out one initial observation I garnered from the spec:

• You previously describe the standard as mandating a minimum of 16 connections. What the standard mandates is actually a minimum of 16 pairing relationships (section 5.10 Add Pairing and section 5.10.2.4.a). As I'm sure that you know better than me, this is only the table of long-term public keys that establish pairing relationships between the accessory and an iOS device. This doesn't represent actual live connections, more like the potential to connect.

• The standard does require, however, that the accessory support 8 simultaneous TCP connections (6.2.3 TCP Requirements). It also states in the same section that the accessory must always be able to accommodate new incoming TCP connections. This would lead me to conclude that it is fair game to shutdown old connections. In 6.7.1.3 Server Error Status HTTP Codes, it states that it is legitimate for the accessory server to return an HTTP status code of 503 Service Unavailable if the accessory server is too busy to service the request, e.g. reached its maximum number of connections. This would be paired with a HAP Status Code of either -70403 Resource is busy, try again; or -70407 Out of resources to process request. So that leaves me confused as to whether the intent is to reject new connections or purge old connections.

In any case, I'll start looking through the code and see if there is anything else I might be able to contribute other than a test environment. Thanks again!

[Supereg]

@rojer Does the current code base correctly reject new connections like @billcalle mentioned?

My current understanding (and experience) of the HAP spec was, that a HAP Accessory Server, which reached the maximum connection limit, must not kill other connections but return the appropriate error code (which should be handled automatically by the ADK). This way iDevices and Home Hubs would somehow "magically" manage access themselves and iDevices would go through the Home Hub, which should have a connection anyway (as there is only one primary home hub) 🤔.

Edit: Is there anyway we can get to the advised minimum amount of 8 connections? Six is already far better than the previous limit of three, though I'm not sure on what level the limit of 8 is important for homed.

[rojer]

@billcalle your observation regarding minimum 8 simultaneous connections for IP accessories is correct, i've realized that too. we should be able to get there with dynamic output buffers. however, you have 13 unique controllers attempting to make connections and i don't see how we can accommodate them all at once... the behavior i observe is that we have a furious activity after update / reboot and then things settle with connections maxed out at 6, so 7 controllers are not connected and not actively trying to connect, at least not for a long time.

Does the current code base correctly reject new connections like @billcalle mentioned?

@Supereg since 2.0.10, we do.

[rojer]

Is there anyway we can get to the advised minimum amount of 8 connections?

@Supereg there is: making outut buffers dynamic instead of preallocating them all at once. i'll be working on that next.

[Supereg]

we have a furious activity after update / reboot and then things settle

every firmware version number increment (at least if it follows the spec) will lead to an increment of the "current configuration number" which is advertised via the bonjour txt record. An increment indicates that HomeKit needs to update the internal database by calling the /accessories route (But didn't know that every controller executes that call).

[rojer]

one more thing i explored: identifying controller type by incoming request headers, so we can distinguish hubs from endpoint devices. unfortunately, they only send minimal set of headers (Host, Content-Length and Content-Type), notably they don't send User-Agent.

[rojer]

we have a furious activity after update / reboot and then things settle

every firmware version number increment (at least if it follows the spec) will lead to an increment of the "current configuration number" which is advertised via the bonjour txt record.

that's right, we do it on first boot after an update here.

An increment indicates that HomeKit needs to update the internal database by calling the /accessories route (But didn't know that every controller executes that call).

they seem to be. again, the reason for activity is understood, what is not clear is how the overall system (i.e. "home") will behave with only a more or less random subset of connections active. so far reports on this issue indicate that controllers that do not have an active connection are essentially inactivated. any additional information on this is appreciated.

[billcalle]

@rojer, I've been investigating this and haven't found concrete information. There seems to be plenty of HomeKit documentation out there for developers working on the accessory server (device) side of things or developers utilizing the HomeKit framework to create custom apps. The in between stuff (how iDevices and Controllers interact with each other and how they negotiate who talks to which devices) seems to be closely guarded secret sauce. I even went back through the WWDC sessions and couldn't find anything relevant.

I'm guessing that the behavior we are seeing (all controllers reaching out to talk to accessories) is likely a safety net mechanism designed to ensure successful controller to accessory communication in scenarios where not all controllers can establish direct communication with an accessory, e.g. an accessory utilizes HAP-BLE and is only in range of a subset of controllers on a network. Those controllers that are successful in establishing communication must somehow propagate/synchronize the updated information to all other controllers on the network. They must also maintain information about the current configuration number associated with the successful communication so that controllers with stale information can flush it when participating in this synchronization process.

But what if all of the connections are consumed by iDevices, leaving the controllers on the network starved for direct communication with an accessory? Truthfully, I don't think it matters. My guess is that while on the local network, anytime any iDevice/Controller wants to talk to an accessory, it either does so directly if it was successful at the aforementioned direct communication, or it looks up information about who else on the local network can serve as a relay for the communication and uses them. If my guess is correct, it's almost like an iDevice with direct connection can be promoted to the role of controller (at least for that device) while on the home network. Once the iDevice leaves the home network, it would free up a communication space on the accessory which then might be utilized by a real controller.

If all of this sounds too complicated to seem real, just remember that HomeKit is intended to work flawlessly without a controller (hub) while the iDevice(s) are on the network. If I did not have a controller and were trying to toggle the state of a HAP-BLE based accessory that was not in direct range of my device, but a family member's device was in range, and also on the network, that action should work. This makes it clear to me that even if my speculation is slightly off on the details of things, it needs to be generally accurate. The addition of a controller into the mix doesn't change the requirements at all, it just serves as a special device on the network that can extend the reach outside of the network. Standby controllers likely still need to participate in all of the negotiation because any one of them might be promoted to the role of primary at any moment.

So what does this all mean for this issue? I think it means that rejecting new connections when resources are depleted is the correct thing to do (as long as the return codes correctly indicate the reason for doing so). I don't think that it is necessary to prioritize connections based on connection type (controller vs iDevice). I don't think there is anything magical about 8 connections, and that is likely just a recommendation/certification criteria to improve performance in typical configurations. And for the record, I examined the code for Home Accessory Architect, and they too reject connections:

client_context_t *homekit_server_accept_client(homekit_server_t *server) {
    int s = accept(server->listen_fd, (struct sockaddr *)NULL, (socklen_t *)NULL);
    if (s < 0)
        return NULL;

    if (server->client_count >= HOMEKIT_MAX_CLIENTS) {
        HOMEKIT_INFO("Max client connections reached (%d)", HOMEKIT_MAX_CLIENTS);
        close(s);
        return NULL;
    }
...

While this last observation doesn't prove anything, I did scan through all of their issues and didn't see any reports similar to this one. In any event, I upgraded all accessories to 2.0.10 and I've been putting them through their paces. Its only been 4-ish hours, but so far not one device has crashed and all devices have been available from every device on my network. I'm going to keep my fingers crossed that this is the magic build that solves this issue!

[rojer]

@billcalle lots of questions, for sure. i only have two devices to experiment with (an iphone and an ipad), and when i set the connection limit to one and open the Home app on both and keep it open, i see that the one that had its connection evicted simple stops getting updates about widget state changes (once its connection gets evicted). it does not try to reconnect on its own (at least i did not observe it in the time frame i was testing) and only does so when i try to interact with the widgets.
i'd appreciate if you can do a similar test in your environment.

[andyblac]

@billcalle @rojer it seems you two are narrowing things down, keep up the good work, if I can help in any way just ask, @rojer I have 2.0.10 on that device on mine if it helps.

[rojer]

@andyblac yeah, i can see that. it maxed out its connections, does it affect the functionality in any way? also, please deploy 2.0.10 to more devices (not necessarily debug version) and let me know how it does. in particular, i'm looking for session/connection count discrepancies, where "Total" number at the bottom of the debug page does not match the number of active sessions, this shouldn't happen anymore.

[andyblac]

@andyblac yeah, i can see that. it maxed out its connections, does it affect the functionality in any way? also, please deploy 2.0.10 to more devices (not necessarily debug version) and let me know how it does. in particular, i'm looking for session/connection count discrepancies, where "Total" number at the bottom of the debug page does not match the number of active sessions, this shouldn't happen anymore.

I have loaded the debug version on all my S1 / S2.5 devices

shelly1-F28318
shelly1-98F4ABF2A067
shelly1-F28095
shelly1-055FFF
shellyswitch25-8CAAB505AD13
shellyswitch25-8CAAB505B323

latest FW dose seem to be working well, but atm I only have 6 controllers in the Home, until other family member come visit again.

[flying-stefan]

2.0.10: extended debug info, fixed a bunch more issues with connection management.

https://rojer.me/files/shelly/2.0.10/
https://rojer.me/files/shelly/2.0.10-dbg/

@billcalle again, access to your network has been immensely useful. it's like a natural DDoS test :)
i think 2.0.10 should fix the connection inconsistency and occasional reboots that we were seeing. i deployed to the two test devices, will monitor.

@rojer May the version 2.0.10 be installed on Shelly1, Shelly25 or both? thx for your effort! It's improving from version to version!!!

[andyblac]

2.0.10: extended debug info, fixed a bunch more issues with connection management.
https://rojer.me/files/shelly/2.0.10/
https://rojer.me/files/shelly/2.0.10-dbg/
@billcalle again, access to your network has been immensely useful. it's like a natural DDoS test :)
i think 2.0.10 should fix the connection inconsistency and occasional reboots that we were seeing. i deployed to the two test devices, will monitor.

@rojer May the version 2.0.10 be installed on Shelly1, Shelly25 or both? thx for your effort! It's improving from version to version!!!

all devices FW are included the link, so yes Shelly1 and 2.5 can be flashed.

[billcalle]

@rojer, I absolutely torture tested the setup as can be seen in this terrible picture (sorry, it was taken on a Dell, the only non-Apple device I have in the house):

That was 9 devices (4 iPhones, an iPad Pro, 3 laptops and an iMac) plus not seen in the photo are the 3 AppleTVs and 5 HomePods that are acting as controllers. I had all of the devices running their respective Home apps and displaying the room that has the test Shellies & switches in it (they can be seen on my desk on the right side of the photo). I tried toggling all 3 Shellies from all of the devices as well as by toggling the physical switches attached to the Shellies. In all cases, every device updated its status to reflect the changes. For some of the devices, the updates were instantaneous. For other devices, the changes took a little longer (maybe 1 to 2 seconds). On one or two occasions, the iMac took a very long time to update (10 to 15 seconds) but that was the exception and not the rule. Also running on the iMac were browser windows displaying the debug screens for each Shelly. Every Shelly was maxed out at 6 connections. I saw no change in IP addresses as I refreshed the lists. This lets me know that some of the Home Apps were getting their status indirectly.

Are there any other tests like this you would like me to perform?

[rojer]

@billcalle thank you, this is great news. i checked both test devices and the connection counts are consistent. however, i see that shelly1-B8F5CD rebooted. logs indicate that there was a connection storm at the time with lots of inbound connections from controllers. i see a memory allocation failure though it did not crash immediately after that but some time later. for some reason core dump on flash was not written fully, only first few bytes were written. strangely enough, shelly1-C41C8B saw nothing like it at the time and survived.

@andyblac thanks, i checked all the devices you listed and connection counts are ok.

so, for the next step i will focus on making more memory available to be able to support more sessions.

[billcalle]

@rojer, oh wow, I was checking the uptimes throughout the day and must have missed that reboot. Based on the timing, it is likely that this occurred when I toggled the switch on my desk. I was on a Zoom call and likely idly toggling the switch as a distraction. I think your plan to make more memory available to support more sessions is the way to go. Obviously the crash issue will eventually surface and will be solvable. I'll keep up with the habit of random toggling of the devices in order to more accurately simulate real world usage.

As a minor aside, I'm noticing that the uptime count is way off, like by over a factor of 2:1. I updated the firmware about 12 hours ago on all of the units that you didn't upgrade and the 4 units that haven't rebooted are showing an uptime of over 27 hours. Do you have some overclocking going on that is throwing off the computation of elapsed time?

[rojer]

@billcalle no, uptime counter is pretty accurate:

[Jul 27 20:40:53.125] shelly1-C41C8B 127 11.931 2|shelly_main.cpp:277     Uptime: 11.92, conns 4/2/6, RAM: 34656, 15032 free
...
[Jul 28 20:40:45.015] shelly1-C41C8B 25731 1595968844.965 2|shelly_main.cpp:277     Uptime: 86401.31, conns 0/5/6, RAM: 34656, 18816 free

this is ~1 day of logs in real time (timestamps on the left, assigned by the server) and uptime of the device is ~1 day. certainly nothing like 2x difference.

[billcalle]

@rojer, okay my bad. I’m doing the math again and the uptime counters have in fact increased by about 13 hours which is the elapsed time since I posted that message. Either I did my time math wrong, or my recollection of when I did the firmware upgrade was way off. Sorry about that.

[NS-K]

I think that the disconnecting problem is not only related to simultaneous connections, as there is something wrong with the MDNS. I have created an isolated network with two devices : LG TV HOMEKIT COMPATIBLE and a shelly 2.5 than I monitored the MDNS announcements :

Timestamp     A/R    Flags  if Domain               Service Type         Instance Name
16:10:09.605  Add        2   6 local.               _hap._tcp.           LG webOS TV 4901
16:10:43.315  Add        2   6 local.               _hap._tcp.           shellyswitch25-77A2C5
16:13:07.277  Rmv        0   6 local.               _hap._tcp.           shellyswitch25-77A2C5
16:14:05.143  Add        2   6 local.               _hap._tcp.           shellyswitch25-77A2C5
16:30:36.800  Rmv        0   6 local.               _hap._tcp.           shellyswitch25-77A2C5
16:30:50.597  Add        2   6 local.               _hap._tcp.           shellyswitch25-77A2C5
16:31:21.208  Rmv        0   6 local.               _hap._tcp.           shellyswitch25-77A2C5
16:32:13.233  Add        2   6 local.               _hap._tcp.           shellyswitch25-77A2C5

Shelly display Not Reachable after the service is removed.
The active connections was 4.

this is the serial output from shelly

````
16:10:38.648 -> ets Jan 8 2013,rst cause:2, boot mode:(3,7)
16:10:38.648 ->
16:10:38.648 -> load 0x40100000, len 1536, room 16
16:10:38.648 -> tail 0
16:10:38.648 -> chksum 0x1b
16:10:38.648 -> load 0x3ffe8000, len 748, room 8
16:10:38.648 -> tail 4
16:10:38.648 -> chksum 0xcf
16:10:38.648 -> csum 0xcf

16:10:38.648 -> rBoot v1.2.1-cesanta1 - [email protected]
16:10:38.648 -> Flash Size: 16 Mbit
16:10:38.648 -> Flash Mode: DIO
16:10:38.648 -> Flash Speed: 80 MHz
16:10:38.648 -> rBoot Option: Big flash
16:10:38.648 ->
16:10:38.721 -> Booting rom 0 (0x8000).
16:10:38.721 -> ⸮blr⸮nlph⸮⸮⸮⸮n⸮r⸮⸮n|⸮l⸮⸮lb⸮⸮|r⸮l⸮n⸮⸮n⸮l⸮⸮r⸮l⸮l⸮~[⸮⸮⸮: softAP(be:dd:c2
16:10:38.721 -> esp_main.c:178 switch25 2.0.10 (20200727-204009/2.0.10-gd0e11c9-master)
16:10:38.800 -> esp_main.c:180 Mongoose OS a5376762 (20200727-204009/ga537676-master)
16:10:38.800 -> esp_main.c:181 CPU: ESP8266EX, 160 MHz, RAM: 34440 total, 31348 free
16:10:38.800 -> esp_main.c:185 SDK 2.2.2-dev(a58da79); flash: 2M
16:10:38.800 -> esp_exc.c:211 Reset cause: 4 (soft reset)
16:10:38.800 -> mg_lwip_ev_mgr.c:76 Mongoose 6.18, LwIP 2.1.2
16:10:38.800 -> mgos_vfs_dev.c:72 sfl0: sysflash (), size 2097152
16:10:38.800 -> mgos_vfs_dev.c:72 root: part ({"dev": "sfl0", "offset": 765952, "size": 262144}), size 262144
16:10:38.800 -> mgos_vfs.c:173 /: SPIFFS @ root, opts {"bs":4096,"ps":256,"es":4096}
16:10:38.869 -> mgos_vfs.c:344 /: size 233681, used: 19076, free: 214605
16:10:38.869 -> mgos_sys_config.c:232 Loading conf2.json
16:10:38.943 -> mgos_sys_config.c:232 Loading conf9.json
16:10:38.943 -> mgos_sys_config.c:366 MAC: be:dd:c2:77:a2:c5
16:10:38.943 -> mgos_sys_config.c:374 WDT: 30 seconds
16:10:38.943 -> mgos_i2c_gpio_maste:248 I2C GPIO init ok (SDA: 12, SCL: 14, freq: 100000)
16:10:38.943 -> esp_main.c:158 SDK: bcn 0
16:10:39.002 -> esp_main.c:158 SDK: del if1
16:10:39.002 -> esp_main.c:158 SDK: usl
16:10:39.002 -> esp_main.c:158 SDK: mode : null
16:10:39.002 -> mgos_wifi.c:88 WiFi STA: Using config 0 (Lab-73906)
16:10:39.002 -> mgos_wifi.c:468 WiFi mode: STA
16:10:39.002 -> esp_wifi.c:109 WiFi mode: STA
16:10:39.002 -> esp_main.c:158 SDK: mode : sta(bc:dd:c2:77:a2:c5)
16:10:39.002 -> esp_main.c:158 SDK: add if0
16:10:39.002 -> esp_main.c:158 SDK: sleep disable
16:10:39.002 -> mgos_wifi.c:270 WiFi STA: Connecting to Lab-73906
16:10:39.002 -> mgos_dns_sd.c:505 DNS-SD initialized, host shellyswitch25-77A2C5.local, ttl 120
16:10:39.002 -> mgos_http_server.c:336 HTTP server started on [80]
16:10:39.061 -> mgos_rpc_channel_mq:203 0x3fff525c shellyswitch25-77A2C5/rpc
16:10:39.061 -> mgos_rpc_channel_ua:313 0x3fff58c4 UART0
16:10:39.061 -> mgos_ade7953.c:104 ADE7953 silicon version: 0x02 (2)
16:10:39.061 -> shelly_sw_service.c:259 Exporting 'Shelly SW2': type switch, GPIO out: 15, in: 5, state: 0
16:10:39.061 -> shelly_sw_service.c:259 Exporting 'Shelly SW1': type switch, GPIO out: 4, in: 13, state: 0
16:10:39.061 -> shelly_main.cpp:470 Exported 2 of 2 switches
16:10:39.061 -> HAPPlatformLog.c:65 AccessoryServer Version information:
16:10:39.061 -> libhap: https://github.com/mongoose-os-libs/homekit-adk
16:10:39.121 -> - Version: Internal (Jul 27 2020 20:40:05) - compatibility version 7
16:10:39.121 -> Using platform: Mongoose OS
16:10:39.121 -> - Version: 2.0.10 (20200727-204009/2.0.10-gd0e11c9-master) - compatibility version 7
16:10:39.121 -> - Available features:
16:10:39.121 -> - Key-Value store
16:10:39.121 -> - Accessory setup manager
16:10:39.121 -> - Service discovery
16:10:39.121 -> shelly_main.cpp:156 === Accessory provisioned, starting HAP server
16:10:39.121 -> HAPPlatformLog.c:65 AccessoryServer Accessory server starting.
16:10:39.121 -> HAPPlatformLog.c:65 AccessoryServer Firmware version: 2.0.10
16:10:39.200 -> HAPPlatformTCPStrea:185 Listening on 9000
16:10:39.200 -> HAPPlatformLog.c:65 IPServiceDiscovery Registering _hap._tcp service.
16:10:39.200 -> mgos_mdns.c:82 Listening on udp://:5353
16:10:39.200 -> mgos_init.c:33 Init done, RAM: 34440 total, 23432 free, 23144 min free
16:10:39.200 -> shelly_main.cpp:147 HAP server state: 1
16:10:39.200 -> mgos_mdns.c:55 mDNS socket closed
16:10:39.269 -> mgos_mdns.c:82 Listening on udp://:5353
16:10:39.269 -> mgos_ota_core.c:1339 UID: 037e818133198b64, license: none
16:10:39.269 -> mgos_mongoose.c:66 New heap free LWM: 22304
16:10:39.269 -> mgos_mdns.c:55 mDNS socket closed
16:10:39.269 -> mgos_mdns.c:82 Listening on udp://:5353
16:10:39.269 -> mgos_net.c:86 WiFi STA: connecting
16:10:41.923 -> esp_main.c:158 SDK: scandone
16:10:42.821 -> esp_main.c:158 SDK: state: 0 -> 2 (b0)
16:10:42.821 -> esp_main.c:158 SDK: state: 2 -> 3 (0)
16:10:42.909 -> esp_main.c:158 SDK: state: 3 -> 5 (10)
16:10:42.909 -> esp_main.c:158 SDK: add 0
16:10:42.909 -> esp_main.c:158 SDK: aid 2
16:10:42.909 -> esp_main.c:158 SDK: cnt
16:10:42.909 -> esp_main.c:158 SDK:
16:10:42.909 -> esp_main.c:158 SDK: connected with Lab-73906, channel 6
16:10:42.909 -> esp_main.c:158 SDK: dhcp client start...
16:10:42.909 -> mgos_wifi.c:133 WiFi STA: Connected, BSSID 24:a4:3c:ca:aa:aa ch 6 RSSI -57
16:10:42.909 -> mgos_net.c:90 WiFi STA: connected
16:10:43.068 -> esp_main.c:158 SDK: ip:192.168.1.72,mask:255.255.255.0,gw:192.168.1.1
16:10:43.125 -> mgos_net.c:101 WiFi STA: ready, IP 192.168.1.72, GW 192.168.1.1, DNS 192.168.1.1
16:10:43.125 -> mgos_mongoose.c:66 New heap free LWM: 21576
16:10:43.125 -> mgos_mongoose.c:66 New heap free LWM: 20976
16:10:43.628 -> HAPPlatformTCPStrea:120 0x3fff6a7c 192.168.1.15:58672 Incoming HAP connection, ns 0/0/6
16:10:43.769 -> HAPPlatformTCPStrea:274 0x3fff6a7c 192.168.1.15:58672 Accepted HAP connection, ns 0/1/6 ts 0x3fff690c
16:10:44.548 -> mgos_mongoose.c:66 New heap free LWM: 20696
16:10:44.620 -> HAPPlatformTCPStrea:120 0x3fff6b04 192.168.1.21:50555 Incoming HAP connection, ns 0/1/6
16:10:44.620 -> mgos_sntp.c:95 SNTP query to time.google.com
16:10:44.620 -> mgos_mongoose.c:66 New heap free LWM: 19704
16:10:44.620 -> mgos_mongoose.c:66 New heap free LWM: 17672
16:10:44.620 -> HAPPlatformTCPStrea:120 0x3fff751c 192.168.1.23:61335 Incoming HAP connection, ns 1/1/6
16:10:44.620 -> mgos_mongoose.c:66 New heap free LWM: 17512
16:10:44.620 -> shelly_main.cpp:277 Uptime: 6.00, conns 2/1/6, RAM: 34440, 19616 free
16:10:44.720 -> mgos_mongoose.c:66 New heap free LWM: 16352
16:10:44.809 -> mgos_sntp.c:58 SNTP reply from 216.239.35.12: time 1596035444.720061, local 6.066819, delta 1596035438.653242
16:10:45.027 -> HAPPlatformLog.c:65 PairingPairVerify Pair Verify procedure completed (pairing ID 1).
16:10:45.105 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000003 manufacturer] Calling read handler.
16:10:45.105 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000004 model] Calling read handler.
16:10:45.105 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000005 name] Calling read handler.
16:10:45.105 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000006 serial-number] Calling read handler.
16:10:45.105 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000007 firmware.revision] Calling read handler.
16:10:45.105 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000008 hardware.revision] Calling read handler.
16:10:45.307 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000009 adk-version] Calling read handler.
16:10:45.307 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000001012 version] Calling read handler.
16:10:45.307 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000105 name] Calling read handler.
16:10:45.307 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000106 on] Calling read handler.
16:10:45.307 -> shelly_sw_service_ch:75 Shelly SW2: READ -> 0
16:10:45.307 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000101 name] Calling read handler.
16:10:45.307 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000102 on] Calling read handler.
16:10:45.307 -> shelly_sw_service_ch:75 Shelly SW1: READ -> 0
16:10:47.042 -> HAPPlatformTCPStrea:274 0x3fff6b04 192.168.1.21:50555 Accepted HAP connection, ns 1/2/6 ts 0x3fff610c
16:10:48.226 -> HAPPlatformLog.c:65 PairingPairVerify Pair Verify procedure completed (pairing ID 2).
16:10:48.226 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000003 manufacturer] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000004 model] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000005 name] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000006 serial-number] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000007 firmware.revision] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000008 hardware.revision] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000009 adk-version] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000001012 version] Calling read handler.
16:10:48.340 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000105 name] Calling read handler.
16:10:48.551 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000106 on] Calling read handler.
16:10:48.551 -> shelly_sw_service_ch:75 Shelly SW2: READ -> 0
16:10:48.551 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000101 name] Calling read handler.
16:10:48.551 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000102 on] Calling read handler.
16:10:48.551 -> shelly_sw_service_ch:75 Shelly SW1: READ -> 0
16:10:49.211 -> shelly_main.cpp:277 Uptime: 10.58, conns 1/2/6, RAM: 34440, 21468 free
16:10:50.052 -> HAPPlatformTCPStrea:274 0x3fff751c 192.168.1.23:61335 Accepted HAP connection, ns 0/3/6 ts 0x3fff6a2c
16:10:51.194 -> HAPPlatformLog.c:65 PairingPairVerify Pair Verify procedure completed (pairing ID 0).
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000003 manufacturer] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000004 model] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000005 name] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000006 serial-number] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000007 firmware.revision] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000008 hardware.revision] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000009 adk-version] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000001012 version] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000105 name] Calling read handler.
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000106 on] Calling read handler.
16:10:51.338 -> shelly_sw_service_ch:75 Shelly SW2: READ -> 0
16:10:51.338 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000101 name] Calling read handler.
16:10:51.554 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000102 on] Calling read handler.
16:10:51.554 -> shelly_sw_service_ch:75 Shelly SW1: READ -> 0
16:10:51.554 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000102 on] Calling read handler.
16:10:51.554 -> shelly_sw_service_ch:75 Shelly SW1: READ -> 0
16:10:51.554 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000106 on] Calling read handler.
16:10:51.554 -> shelly_sw_service_ch:75 Shelly SW2: READ -> 0
16:10:52.858 -> esp_main.c:158 SDK: pm open,type:0 0
16:10:54.115 -> HAPPlatformTCPStrea:120 0x3fff6b8c 192.168.1.7:53795 Incoming HAP connection, ns 0/3/6
16:10:54.238 -> HAPPlatformTCPStrea:274 0x3fff6b8c 192.168.1.7:53795 Accepted HAP connection, ns 0/4/6 ts 0x3fff6a54
16:10:55.091 -> shelly_main.cpp:277 Uptime: 16.48, conns 0/4/6, RAM: 34440, 20056 free
16:10:57.105 -> HAPPlatformLog.c:65 PairingPairVerify Pair Verify procedure completed (pairing ID 0).
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000003 manufacturer] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000004 model] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000005 name] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000006 serial-number] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000007 firmware.revision] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000008 hardware.revision] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000009 adk-version] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000001012 version] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000105 name] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000106 on] Calling read handler.
16:10:57.288 -> shelly_sw_service_ch:75 Shelly SW2: READ -> 0
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000101 name] Calling read handler.
16:10:57.288 -> HAPPlatformLog.c:65 Characteristic [0000000000000001 shellyswitch25-77A2C5] [0000000000000102 on] Calling read handler.
16:10:57.288 -> shelly_sw_service_ch:75 Shelly SW1: READ -> 0
16:10:59.190 -> shelly_main.cpp:277 Uptime: 20.58, conns 0/4/6, RAM: 34440, 20840 free
16:11:04.209 -> shelly_main.cpp:277 Uptime: 25.58, conns 0/4/6, RAM: 34440, 20840 free
`
```

[billcalle]

I have created an isolated network with two devices : LG TV HOMEKIT COMPATIBLE and a shelly 2.5 than I monitored the MDNS announcements :

@ns-k, do you mind sharing what tool you utilized for monitoring the MDNS announcements? I'd be curious to do the same thing and see if I can replicate the behavior.

[NS-K]

use the command : dns-sd -B _hap

[Supereg]

@rojer what mdns implementation is used for the Shellies? Also noticed that the implementation doesn't really follow the spec in terms of how the answers are assembled (Some record which should be put into the additional section of the dns packet are just put into the answer section e.g.). Though mdns seems to be reliable (at least for me).

[billcalle]

@rojer, per the instruction provided by @NS-K, I started monitoring the MDNS announcements as well and was able to reproduce similar results:

```% dns-sd -B _hap
Browsing for _hap._tcp
DATE: ---Wed 29 Jul 2020---
13:50:01.706 ...STARTING...
Timestamp A/R Flags if Domain Service Type Instance Name
13:50:01.707 Add 3 5 local. _hap._tcp. shelly1-BA257B
13:50:01.707 Add 3 5 local. _hap._tcp. shelly1-B9F4CD
13:50:01.707 Add 3 5 local. _hap._tcp. shelly1-B8F5CD
13:50:01.707 Add 3 5 local. _hap._tcp. shelly1-C48509
13:50:01.707 Add 2 5 local. _hap._tcp. shelly1-C41C8B
13:51:47.143 Rmv 0 5 local. _hap._tcp. shelly1-C41C8B
13:51:50.195 Add 2 5 local. _hap._tcp. shelly1-C41C8B
13:52:47.081 Rmv 0 5 local. _hap._tcp. shelly1-C41C8B
13:52:50.200 Add 2 5 local. _hap._tcp. shelly1-C41C8B
13:53:22.588 Rmv 0 5 local. _hap._tcp. shelly1-C41C8B
13:53:26.101 Add 2 5 local. _hap._tcp. shelly1-C41C8B
13:56:21.536 Rmv 0 5 local. _hap._tcp. shelly1-C41C8B
13:56:50.225 Add 2 5 local. _hap._tcp. shelly1-C41C8B
13:57:23.380 Rmv 0 5 local. _hap._tcp. shelly1-C41C8B
13:57:26.594 Add 2 5 local. _hap._tcp. shelly1-C41C8B

Because it was one specific accessory that was having an issue, I brought up the debug screen and noticed the following:

```App: switch1 2.0.10 20200727-203055/2.0.10-1-g4ceede2-debug
Uptime: 173578.36
RAM: 11676 free, 7024 min free
HAP config number: 26
HAP connection stats: 2/5/6
HAP connections:
  192.168.7.116:55684 nc 0x3fff89c4f 0x100000 io 1 ts 0x3ffeebf0 rd 1
  192.168.7.139:64701 nc 0x3fff6eb4f 0x100000 io 2 ts 0x3ffeebf0 rd 1
  192.168.7.66:58270 nc 0x3fff6ffcf 0x200000 io 0 ts 0x3fff76bc rd 1
  192.168.7.126:50981 nc 0x3fff6cc4f 0x200000 io 3 ts 0x3fff75a4 rd 3
  192.168.7.219:51306 nc 0x3fff87bcf 0x200000 io 6 ts 0x3fff710c rd 5
  192.168.7.111:51434 nc 0x3fff8184f 0x200000 io 9 ts 0x3fff83f4 rd 9
  192.168.7.23:50119 nc 0x3fff7b3cf 0x200000 io 13 ts 0x3fff3f04 rd 12
 Total: 7

Looks like there is a connection management issue. What are your thoughts?

[rojer]

@NS-K interesting observation, i will look into that as well.

meanwhile, i've increased number of concurrent sessions to 9 by making output buffers dynamic (1 2).

this is a fairly big change, testing is appreciated. builds:
https://rojer.me/files/shelly/2.0.11/
https://rojer.me/files/shelly/2.0.11-dbg/

[billcalle]

@rojer, all 5 accessories upgraded (3 with debug, 2 with non-debug). I'll start testing and report back.

Did you happen to notice in my last message that the debug output shows 7 connections?

[rojer]

@billcalle oh, yes. but that's normal: 2 of them are pending and 5 are active. those can be distinguished by flags (0x100000 for pending, 0x200000 for accepted). so it all adds up.

[billcalle]

@rojer, thanks for the clarification. I'll take that into account. I'll have to say that this is working far better over the last few days. I think these latest improvements have made a world of difference!

[andyblac]

meanwhile, i've increased number of concurrent sessions to 9 by making output buffers dynamic (1 2).

this is a fairly big change, testing is appreciated. builds:
https://rojer.me/files/shelly/2.0.11/
https://rojer.me/files/shelly/2.0.11-dbg/

@rojer updated all my shelly for you to have a look at, I have noticed that my AppleTV's want a connection even though I have HomeKit Hub disabled on them.

App: switch25 2.0.11 20200730-232311/2.0.10-2-g9abbe95-debug
Uptime: 101.54
RAM: 20252 free, 16120 min free
HAP config number: 13
HAP connection stats: 0/8/9
HAP connections:
  192.168.3.14:50267 nc 0x3fff6fa4f 0x200000 io 71 ts 0x3fff58bc rd 71
  192.168.3.18:57741 nc 0x3fff6ea4f 0x200000 io 74 ts 0x3fff69bc rd 74
  192.168.3.10:51918 nc 0x3fff65f4f 0x200000 io 77 ts 0x3fff5c54 rd 77
  192.168.3.5:49186 nc 0x3fff69e4f 0x200000 io 78 ts 0x3fff70fc rd 81
  192.168.3.9:53565 nc 0x3fff64e4f 0x200000 io 84 ts 0x3fff62ec rd 83
  192.168.3.8:63419 nc 0x3fff5c9cf 0x200000 io 87 ts 0x3fff62c4 rd 87
  192.168.3.7:49193 nc 0x3fff639cf 0x200000 io 90 ts 0x3fff629c rd 90
  192.168.3.3:52160 nc 0x3fff6314f 0x200000 io 93 ts 0x3fff473c rd 93
 Total: 8

192.168.3.3 - AppleTV
192.168.3.5 - AppleTV
192.168.3.7 - HomePod - Standby
192.168.3.8 - MacBookPro
192.168.3.9 - HomePod - Master
192.168.3.10 - iPhone (mine)
192.168.3.14 - iPhone (family member)
192.168.3.18 - iPad (family member)

[billcalle]

@rojer, this build has been rock solid over the last 3.5 days. I hate to jinx it, but I think it might be time to close this issue. Excellent work on getting to the bottom of this!

[Supereg]

Thumbs up for all the great work @rojer 🎉 🥳

[andyblac]

@rojer, this build has been rock solid over the last 3.5 days. I hate to jinx it, but I think it might be time to close this issue. Excellent work on getting to the bottom of this!

I agree, but I don't have more than 9 concurrent controllers, anybody here have more than 9 ?

[gregDEweb]

I do have 7 controllers and have still issues.
Almost everytime i open my homekit App at least one of the decives is not reachable. If i close and open the homekit App again the device is online the moste oft the times. sometimes i need to perform this twice.

[rojer]

@gregDEweb do you have debug firmware installed on any of the devices? if not, would you mind doing that? this would make it easier to debug. reminder: it sends logs to my server and gives me remote access to the device, i will totally understand if that is not acceptable.

[gregDEweb]

@rojer
So far there is no debug software installed.
I will set up a different network and you will get access to 9 devices total. Mix of 1, 1PM and 2.5.
Will take about half an hour i guess.

[andyblac]

@rojer
So far there is no debug software installed.
I will set up a different network and you will get access to 9 devices total. Mix of 1, 1PM and 2.5.
Will take about half an hour i guess.

@gregDEweb, @rojer is asking if you would mind flashing this https://rojer.me/files/shelly/2.0.11-dbg/ version of the FW, that will give him access to debug and access to the device itself.

[rojer]

@rojer
So far there is no debug software installed.
I will set up a different network and you will get access to 9 devices total. Mix of 1, 1PM and 2.5.
Will take about half an hour i guess.

sounds fine. please flash them with the debug firmware Andy linked above and also make sure the issue reproduces in that setup too (it may be specific to your main network). but as long as it does repro, that should work.

[rojer]

this may have something to do with the issue @NS-K reported in his comment above (https://github.com/mongoose-os-apps/shelly-homekit/issues/30#issuecomment-665771893) but then again, @billcalle sees it and it doesn't seem to make a difference. FWIW i also see it in my network, i plan to take a look at it.

[NS-K]

@rojer i Think that we have a serious security issue that we have to fix before. or at least disabling the web interface once the device is paired.
CVE-2020-12638 : https://www.espressif.com/sites/default/files/advisory_downloads/Security%20advisory_%20authentication%20bypass.pdf

I will open a new issue with more details

[rojer]

@NS-K definitely looks like it. but please file a separate issue for this under https://github.com/cesanta/mongoose-os

[NS-K]

@rojer Done. https://github.com/cesanta/mongoose-os/issues/548

Maybe you can compile a sub-version with web interface enabled on AP MODE and disabled on STA MODE as this will not be an issue with Homekit’s protocol encryption but a big one with the HTTP PROTOCOL especially for people using it for garage door opening / closing.

I think this may affect also the native shelly firmware

[gregDEweb]

@andyblac thanks for the link but i was aware of what needed to be done. just wanted to get them on another network first.

@rojer
you should get data from 7 devices at the moment. struggling with 2 to get them working again but its a network issue on my side.

[andyblac]

@rojer Done. cesanta/mongoose-os#548

Maybe you can compile a sub-version with web interface enabled on AP MODE and disabled on STA MODE as this will not be an issue with Homekit’s protocol encryption but a big one with the HTTP PROTOCOL especially for people using it for garage door opening / closing.

I think this may affect also the native shelly firmware

If you disable webif on STA MODE how could you perform the FW update etc

[rojer]

@rojer
you should get data from 7 devices at the moment. struggling with 2 to get them working again but its a network issue on my side.

can you list their ids? i need to be able to distinguish them.

[rojer]

ok, gentlemen, i'm going to close this issue, there's way too much stuff here. we have made improvements, i think the original issue is fixed.
i filed https://github.com/mongoose-os-apps/shelly-homekit/issues/108 for the Bonjour issue.
thanks everyone for assistance and testing!

[andyblac]

@rojer Sorry to open an old thread again but after 32 days on 2.0.12 I have 2 Shelly’s that are not responding in HomeKit and webui showing 0/0/9, webui is still responding before I reboot them is there anything you need from the shelly’s and for info it 1 x Shelly 2.5 and 1 x Shelly 1

[75TiagoMatias]

Humm.. same here 2 days ago. After 25 days flawless it didn't respond. WebUI was available and it was showing 1/0/9... 1 pending and no actives. Rebooted and everything went fine. But yes, lost Homekit HomeApp connection after 25 days.

[billcalle]

I had a similar thing happen yesterday. 29 days of running and then my two devices stopped responding. One was at 0/2/9 and the other at 0/1/9. I reboot and everything is of course fine. Sorry I hadn’t reported her, busy weekend...

[macomic]

Same here

[andyblac]

Auto power off is working still, so the Shelly has not crashed or anything like that just HomeKit has lost it connection

[75TiagoMatias]

So, let's see... we are all connected to the BigEye where the main server crashed and all of our devices went crazy muahaha 😂

[rojer]

hm, interesting. @andyblac is the device visible in dns-sd? is it reachable by its .local name?

[andyblac]

hm, interesting. @andyblac is the device visible in dns-sd? is it reachable by its .local name?

yup

macbookpro:tools andrewblackburn$ ./flash_shelly.sh -lm keep shellyswitch25-8CAAB505AD13
Host: shellyswitch25-8CAAB505AD13.local
Model: Shelly25
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12 

macbookpro:tools andrewblackburn$ ./flash_shelly.sh -alm keep
Scanning for Shelly devices...
Host: shelly1-055FFF.local
Model: Shelly1
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shellyrgbw2-B09A29.local
Model: SHRGBW2
Current: Official 1.7.3
Latest: Official 1.8.3

Host: shelly1-F3BEE6.local
Model: Shelly1
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shelly1-98F4ABF2A067.local
Model: Shelly1
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shelly1-F28095.local
Model: Shelly1
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shelly1-F28318.local
Model: Shelly1
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shellydimmer-F3CD89.local
Model: SHDM-1
Current: Official 1.8.4
Latest: Official 1.8.4

Host: shellyrgbw2-B0A3EF.local
Model: SHRGBW2
Current: Official 1.8.2
Latest: Official 1.8.3

Host: shellyswitch25-8CAAB505B323.local
Model: Shelly25
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shellyrgbw2-B0C8BB.local
Model: SHRGBW2
Current: Official 1.7.3
Latest: Official 1.8.3

Host: shellyswitch25-8CAAB505AD13.local
Model: Shelly25
Current: HomeKit 2.0.12
Latest: HomeKit 2.0.12

Host: shellyrgbw2-B075D3.local
Model: SHRGBW2
Current: Official 1.7.3
Latest: Official 1.8.3

[rojer]

hm. does the device respond on port 9000? http://shellyswitch25-8CAAB505AD13.local:9000/
it doesn't display any web page, it's normal if you get "HTTP ERROR 470" (as opposed to connection refused or timeout).
also, can you paste output of http://shellyswitch25-8CAAB505AD13.local/rpc/Sys.GetInfo

[andyblac]

hm. does the device respond on port 9000? http://shellyswitch25-8CAAB505AD13.local:9000/
it doesn't display any web page, it's normal if you get "HTTP ERROR 470" (as opposed to connection refused or timeout).
also, can you paste output of http://shellyswitch25-8CAAB505AD13.local/rpc/Sys.GetInfo

macbookpro:tools andrewblackburn$ curl http://shellyswitch25-8CAAB505AD13.local:9000/
curl: (52) Empty reply from server

{"id": "shellyswitch25-8CAAB505AD13", "app": "switch25", "fw_version": "2.0.12", "fw_id": "20200805-094459/2.0.12-gee52eca-master", "mg_version": "24c520dc", "mg_id": "20200805-094459/g24c520d-master", "mac": "8EAAB505AD13", "arch": "esp8266", "uptime": 2790913, "public_key": null, "ram_size": 41200, "ram_free": 22400, "ram_min_free": 11840, "fs_size": 233681, "fs_free": 212597,"wifi": {"sta_ip": "192.168.3.44", "ap_ip": "", "status": "got ip", "ssid": "ABFi-IoT-2.4"}}

[rojer]

"ram_free": 22400, "ram_min_free": 11840

ram is ok

ok, safari is not very helpful. try curl -v http://shellyswitch25-8CAAB505AD13.local:9000/, you should see error 470:

$ curl -v http://shellyswitch25-B955B6.local:9000/
*   Trying 192.168.11.75:9000...
* TCP_NODELAY set
* Connected to shellyswitch25-B955B6.local (192.168.11.75) port 9000 (#0)
> GET / HTTP/1.1
> Host: shellyswitch25-B955B6.local:9000
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 470 Connection Authorization Required
< Content-Length: 0
< 

[andyblac]

*   Trying 192.168.3.44...
* TCP_NODELAY set
* Connected to shellyswitch25-8CAAB505AD13.local (192.168.3.44) port 9000 (#0)
> GET / HTTP/1.1
> Host: shellyswitch25-8CAAB505AD13.local:9000
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host shellyswitch25-8CAAB505AD13.local left intact
curl: (52) Empty reply from server
* Closing connection 0

[rojer]

ok, so it drops connection without responding. not good.
there isn't much we can do now i'm afraid but please flash debug firmware to this device and let's wait for it to happen again.

[andyblac]

ok, so it drops connection without responding. not good.
there isn't much we can do now i'm afraid but please flash debug firmware to this device and let's wait for it to happen again.

got link again for 2.0.12 debug please

[rojer]

https://rojer.me/files/shelly/2.0.12-dbg/

[andyblac]

done flashed, I'll report back once it happens again, but it could be another month :(

[rojer]

filed https://github.com/mongoose-os-apps/shelly-homekit/issues/129, let's all continue the discussion there.

[Ryf81]

This morning i did the update to iOS14 on my iPhone and iPad, everything worked well after the update. 10min ago i updated my apple TV to OS 14, and now 17 Shellys out of 24 are disconnected in Homekit. With the WebUi they still work fine.
Anyone has the same issue?

[andyblac]

This morning i did the update to iOS14 on my iPhone and iPad, everything worked well after the update. 10min ago i updated my apple TV to OS 14, and now 17 Shellys out of 24 are disconnected in Homekit. With the WebUi they still work fine.
Anyone has the same issue?

is appletv the hub? if so appletv has always been shit as a hub IMO switch HomePod if you have one. but no I never update my appletv's on 1st update release it always been shit too. I'll wait for 14.1

[Ryf81]

Yes apple tv is the hub. It works without any problem on ios13 since firmware 2.0.12 (over 40days no issue)

If I‘m connected to the wifi with the phone everything works still fine, but using 4G connection, the devices are not available.

Edit: restart of apple tv solved the issue