Shelly-homekit: HAP HTTP server drops connections

[rojer]

Reported by @andyblac in https://github.com/mongoose-os-apps/shelly-homekit/issues/30#issuecomment-687673015, confirmed by other users.
Apparently, after a while (~30 days) the device becomes unavailable in the Home app.

• still available on port 80, web ui works fine
• RPC works fine
• free ram is normal, so not a memory leak
• HAP connection counts are all at 0
  HAP HTTP server drops connection without sending a response - https://github.com/mongoose-os-apps/shelly-homekit/issues/30#issuecomment-687847662

[pivale]

Hey,

Having the same problem:

0/0/9 pending/active/max
Uptime: 25:16:33:46

Noticed just because automation on babycam stopped working.

Anything other information I can provide before rebooting?

[andyblac]

nope, if you want to load the debug version, so if you get it again, it will allow @rojer to login to the device to see why it is happening download from below

https://rojer.me/files/shelly/2.0.12-dbg/

[pivale]

nope, if you want to load the debug version, so if you get it again, it will allow @rojer to login to the device to see why it is happening download from below

https://rojer.me/files/shelly/2.0.12-dbg/

2 quick questions, as I would like to help:

• what do you mean connecting to the device? as an ssh connection there? or just having more info I could then provide?
• when changing/loading/updating the firmware, does the connection with homekit needs to be redone and all other configurations?

Thank you!

[andyblac]

nope, if you want to load the debug version, so if you get it again, it will allow @rojer to login to the device to see why it is happening download from below
https://rojer.me/files/shelly/2.0.12-dbg/

2 quick questions, as I would like to help:

• what do you mean connecting to the device? as an ssh connection there? or just having more info I could then provide?
• when changing/loading/updating the firmware, does the connection with homekit needs to be redone and all other configurations?

Thank you!

you'll have to ask @rojer about question 1, but as for Q2 no just flash it, all settings are saved.

[rojer]

i ran a "connection stress test" overnight, with about 25000 connection attempts total - everything worked as expected.

what do you mean connecting to the device? as an ssh connection there? or just having more info I could then provide?

debug firmware sends serial console logs to my server and opens an MQTT connection back to my server, so i can send RPC requests to the device remotely, to reconfigure it and gather debug information. it also gives me ability to reflash the device remotely, so pretty much full control over a device on your network. i totally understand if you are not willing to provide that kind of access, but it's really invaluable in terms of debugging and being able to figure out what's wrong directly.

[rojer]

i have one suspicion: just over 24 days is the overflow value for a signed 32-bit counter of milliseconds. i need to check if we use 32-bit uptime somewhere in HAP code.

[andyblac]

@rojer only had 4 devices do this atm, other devices still going strong, if it helps, it seems to be devices that are used in HomeKit automations either as a trigger or an action.

ie my bathroom SW2 is a fan that is controlled via an automation from a Humidity Sensor if humidity gets over 75% it kicks in fan and turns off once it drops below 65%

[andyblac]

i have one suspicion: just over 24 days is the overflow value for a signed 32-bit counter of milliseconds. i need to check if we use 32-bit uptime somewhere in HAP code.

hmm, now you mention counter, could it be the auto off feature ? these same devices are using that too ?

my bathroom SW1 is using the auto off feature. (thats the one you have access too)

[billcalle]

Both of mine died and I do not use auto off or automation. I think that is a red herring.

[andyblac]

Both of mine died and I do not use auto off or automation. I think that is a red herring.

thanks for confirming. only 4 of mine have needed a reboot, all the rest are still running spot on. 34 days and continuing

[rebuss0507]

Hi, just wanted to confirm the exact same issue for me. I was also having the issue reported in #30 (comment) and where upgrading after fix firmware where online. All of my Shelly actually died a couple of days ago already (but not exactly sure when). Rebooted them now after being online for 29 days, now they are all working again......

Except two where one of it is indeed included in an automation.....But I will check on these extra separately

[andyblac]

@rojer on day 25, so we should be seeing something soon

[andyblac]

@rojer it happened again, can you check logs ?? shellyswitch25-8CAAB505AD13 was ok this morning but now 'no response` in HomeKit

App: switch25 2.0.12 20200805-094606/2.0.12-1-ga4caed4-debug
Uptime: 2236689.69
RAM: 22276 free, 12488 min free
HAP config number: 14
HAP connection stats: 0/3/9
HAP connections:
  192.168.3.25:49174 nc 0x3fff48bcf 0x200000 io 176732 ts 0x3fff3554 rd 176727
  192.168.3.3:49166 nc 0x3fff672cf 0x200000 io 234150 ts 0x3fff67b4 rd 234143
  192.168.3.18:51807 nc 0x3fff5f14f 0x200000 io 23072 ts 0x3fff54bc rd 23071
 Total: 3

[andyblac]

every now and then log shows 1 pending but it does not get a new connection.

[rojer]

[Oct  2 14:47:29.773] shellyswitch25-8CAAB505AD13 497165 1601650049.912 2|shelly_main.cpp:276     Uptime: 2236751.25, conns 0/3/9, RAM: 41200, 24664 free                                                                                                                                             
[Oct  2 14:47:30.389] shellyswitch25-8CAAB505AD13 497166 1601650050.472 2|HAPPlatformTCPStrea:120 0x3fff4834 192.168.3.24:50151 Incoming HAP connection, ns 0/3/9                                                                                                                                     
[Oct  2 14:47:30.389] shellyswitch25-8CAAB505AD13 497167 1601650050.482 2|HAPPlatformTCPStrea:274 0x3fff4834 192.168.3.24:50151 Accepted HAP connection, ns 0/4/9 ts 0x3fff68ec                                                                                                                       
[Oct  2 14:47:30.389] shellyswitch25-8CAAB505AD13 497168 1601650050.502 2|HAPPlatformLog.c:65     IPAccessoryServer Failed to allocate session. (Number of supported accessory server sessions should be consistent with the maximum number of concurrent streams supported by TCP stream manager.)   

hm, that's odd. something is off with session management. i'll need to take a closer look

[andyblac]

you want me to leave it in this state for now ?

[rojer]

there isn't much use, unfortunately, as core dump functionality is broken.
i'll review the relevant code, for now feel free to reboot and upgrade this device.

[rojer]

clearly HAP sessions get out of sync with TCP connections. i'll add a bunch of debug output around that code.

[andyblac]

if you want me test another firmware thats no problem, but it will take another 25 or so till it happens again.

[andyblac]

let me know when this is built, I'll update and enable debug logging, will that open port for you ?

[rojer]

it's out with 2.2.0, please install it on the device.
it will provide additional information i can correlate with logs to figure out when the discrepancy occurs.
i should be able to notice it earlier, once first "phantom" session appears instead of when we run out of session slots.
this will manifest as number of sessions getting out of sync with number of active connections.

[rojer]

...aaaand it went silent. install the 2.2.0-debug build please - http://rojer.me/files/shelly/2.2.0-dbg/

[andyblac]

...aaaand it went silent. install the 2.2.0-debug build please - http://rojer.me/files/shelly/2.2.0-dbg/

done.

[rojer]

yep, it's back. ok, will keep an eye on it.

[andyblac]

cheers.