Shellyforhass: Shelly 2.5 shortly becoming unavailable every minute or so

[gaggio]

My Shelly 2.5 started immediately to work correctly with you compoonent, being recognized as a cover. Unfortunately, I can notice from the events log that it shortly becomes unavailable every minute or few minutes, generally for less tha a minute.
Is there an explanation to this behaviour, which of course might not depend on the component?
Updated to latest firmware 1.5.0

Thanks

[hakana]

The component set the switch unavailable if it not got any CoAP messages from it under 20 sec. I will check this with my Shelly 2.5 and see if it is some problem. You can find a simple python script that printing out the CoAP messages in raw format here: https://github.com/StyraHem/ShellyForHASS/blob/master/util/test.py . You can try it and se how the traffic look like.

[hakana]

@gaggio Have you tested some more?

[gaggio]

Haven't had a chance to do much more, I can say the behaviour is still the same after some days the shelly has been connected.
I have tried to test your script but it gives me Invalid Argument at line 11, both on python 2.7 and 3.7.
Must be an issue with the socket package version... I'll investigate

[gaggio]

After few minutes of running your test script, this is what I get dumped in a file:

b'P\x1e\x04\x10\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x11\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x12\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x13\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x14\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x15\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x16\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x17\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x18\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x19\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x1a\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x1b\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x1c\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x1d\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x1e\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04\x1f\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04X\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}' b'P\x1e\x04Y\xb3cit\x01s\xed\x0b\xec\x03SHSW-25#73C0D2#1\xd2C\x96\x00\x82\x0c\x00\xff{"G":[[0,112,0],[0,122,0],[0,113,0],[0,111,0.000000],[0,121,0.000000]]}'

[hakana]

How long time is it between each message you get?

[xenio]

Same for me, Shelly 2.5 is recognised and regular added to the entities but it becomes unavailable and back online after some seconds.
My Shelly 2.5 does not have this options checked:
"Enable action execution via CoAP (CoIOT) protocol"
I have only Shelly cloud.
The test.py does not show anything.

[gaggio]

How long time is it between each message you get?

I tried to run again test.py to answer your question. The interval is generally almost exactly 15s, then occasionally becomes 2min30s or so. During that time it is marked unavailable. I also noticed that during those "pauses" the shelly's web interface becomes unresponsive as well.

EDIT: the interval occasionally becomes also greater than 10min

[hakana]

Ok, sounds like there is a problem with the Shelly or your network. Can you ping the Shelly during this time? Can you ping something else?

[hakana]

Same for me, Shelly 2.5 is recognised and regular added to the entities but it becomes unavailable and back online after some seconds.
My Shelly 2.5 does not have this options checked:
"Enable action execution via CoAP (CoIOT) protocol"
I have only Shelly cloud.
The test.py does not show anything.

Are you running test.py on the same machine as HASS?

[xenio]

@hakana No, the python script is on my Mac and Hass is on a Intel Nuc. I thought that the broadcast protocol was catch in all the network. I'll try the test.py on the Nuc.
What about that option on the Shelly "Enable action execution via CoAP" has to be on?

[hakana]

@xenio You don't need to turn on "Enable action execution via CoAP".
You probably have a firewall in your Mac that not is open for the broadcasts. Test on the Nuce and see if it works and how often you get the broadcasts.

[WeaselBreath]

Installed this today and I'm seeing the same issue with a Shelly1 switch. I keeps cycling to 'Unavailable" about once per minute. I previously had the switch set up with MQTT in the Shelly app and it worked flawlessly.

[hakana]

I'm running it self with 30+ Shellies and I can't see this problem at all. Each Shelly sending a CoAP message around every 10 sec and they never got offline for me. So the question is why it not working in some networks. It could depend on firewalls, switches etc that block the traffic. I need more input to track down this issue.

[samlof]

I have a similar problem with Shelly Bulb. The test script only receives update messages rarely. On and off commands do go through, but hass state doesn't update.
The web ui and mobile app still work.

I rebooted the bulb from web ui and the updates are sent for 1-5 minutes. Then it stops again.

I have Zyxel router and tried disabling it's firewall. Shelly has firmware 20190531-075722/v1.5.0-hotfix2@022ec015

[hakana]

Can everyone with this problem test to set the flag igmp_fix in config.yaml .

[gaggio]

Tried, restarted and left it on since this morning, doesn't seem to make any difference.

[samlof]

Didn't change anything. I have put mqtt on before and turned it off to test this.

[gaggio]

Yesterday I connected another shelly 2.5, so I have the opportunity to test if the first one has hardware malfunction. The second one behaves the same in home assistant. I also reduced the home_interval to 1 minute to capture even short wifi disconnections of both shellies, will update in one or two days with the outcomes.

[gaggio]

Yesterday I connected another shelly 2.5, so I have the opportunity to test if the first one has hardware malfunction. The second one behaves the same in home assistant. I also reduced the home_interval to 1 minute to capture even short wifi disconnections of both shellies, will update in one or two days with the outcomes.

Did the above and also shortened scan_inerval to 30s, result is that both shellies never get disconnected from wifi, yet they are still marked unavailable every minute in home assistant. Tried to activate the switches both from shelly app and through voice commands (google home) and it works also during unavailability in home assistant.

[gaggio]

Other thing I noticed today, is that the two roller shutters suspiciously become unavailable at the same time, see the stripes in the history view that are strongly correlated:

[hakana]

Is there anyone that get this that know python and can try to debug it?

[rufik]

Is there anyone that get this that know python and can try to debug it?

Do you have any hints where exactly look into code for debugging this issue? I have rare symptoms like that, I can put some effort and try to gain some additional info...

[gaggio]

Is there anyone that get this that know python and can try to debug it?

I know python but I honestly I wouldn't be able to understand where to look in such a complex project. I can try and with some hints be probably able to write additional debugging info to the logs...

[hakana]

@gaggio and @rufik, thank you for helping me with this.

We can just start just to ensure your HASS server receive all CoAP messages correctly. I have put together a python script that you can run. If you are running docker please also try to run it inside the docker container.

Change the 134E13 (line 18) to the ID of one of your devices. When you run the script it will print the time stamp every time it receive a CoAP message from the Shelly device. Shelly sending this message every 15 second, so verify you get all the messages.

How are your HASS environment look like, are you running HassIO or HASS, in docker or Raspberry PI etc?

import datetime
import socket
import struct

UDP_IP = "224.0.1.187"
UDP_PORT = 5683

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', UDP_PORT))
mreq = struct.pack("4sl", socket.inet_aton(UDP_IP), socket.INADDR_ANY)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

while True:
  try:
    data, addr = sock.recvfrom(10240)
    #print(addr)
    if str(data).find("134E13") >= 0:
        print(str(datetime.datetime.now()))
        #print str(data)
  except Exception as e:
    print ('exception ' + str(e))

[rufik]

I'm running your script right now monitoring shelly 1. Directly on host (Ubuntu 18.04, arm64, kernel 5.1.16) using python 3.6. I let you know about result.

My HAS is running in docker using host network mode. I had some only minor unavailability so far:
https://imgur.com/AvdPgiU

[rufik]

OK, I've got first results. HAS logbook first:

August 1, 2019
10:15 PM Shelly 1 PM [609304] turned off
10:15 PM Shelly 1 PM [609304] changed to unavailable

Your script output then:

[2019-08-01 22:13:50.711487] b'P\x1e\xc3\xbb\xb3cit\x01s\xed\x0b\xec\x03SHSW-PM#609304#1\xd2C\x96\x00\x82\x03\x00\xff{"G":[[0,111,0.00],[0,112,0]]}'
[2019-08-01 22:14:05.712607] b'P\x1e\xc3\xbc\xb3cit\x01s\xed\x0b\xec\x03SHSW-PM#609304#1\xd2C\x96\x00\x82\x03\x00\xff{"G":[[0,111,0.00],[0,112,0]]}'
[2019-08-01 22:14:20.713407] b'P\x1e\xc3\xbd\xb3cit\x01s\xed\x0b\xec\x03SHSW-PM#609304#1\xd2C\x96\x00\x82\x03\x00\xff{"G":[[0,111,0.00],[0,112,0]]}'
[2019-08-01 22:15:05.768656] b'P\x1e\xc3\xc0\xb3cit\x01s\xed\x0b\xec\x03SHSW-PM#609304#1\xd2C\x96\x00\x82\x03\x00\xff{"G":[[0,111,0.00],[0,112,0]]}'

As you can see, there are only two CoAP messages at 2019-08-01 22:14 but should be four. Router's log is empty at this timestamp.

[hakana]

Ok, so the component working as it should... So the question is if the Shelly missing to send some messages or if they are disappeared on its way. Can you run the script on another computer at the same time and see if they show same result?

I can add a parameter to the plugin where you can change the time for the Shelly to go in to unavailable state. The time to be unavailable now is 20 sec.

[rufik]

I'll try to run on another host and will see.
Maybe you should set 60 sec timeout for unavailable state? Maybe just obey scan_interval setting?
In my case unavailability is just one minute long, so extending this "timeout" to 60 sec will do the trick I think.

[gaggio]

I'm running HASS.io. See below what I get with your script (why the messages are not logged, I don't know), and the status of the same shelly from home assistant history:

2019-08-05 17:08:49.392561
2019-08-05 17:09:04.400565
2019-08-05 17:09:19.424844
2019-08-05 17:09:34.456194
2019-08-05 17:09:49.506272
2019-08-05 17:10:04.516213
2019-08-05 17:10:19.526251
2019-08-05 17:24:35.379485
2019-08-05 17:24:50.390298
2019-08-05 17:25:05.398846
2019-08-05 17:36:36.083707
2019-08-05 17:36:51.109751
2019-08-05 17:37:06.108667
2019-08-05 17:37:21.134392
2019-08-05 17:37:36.131975
2019-08-05 17:37:51.139442
2019-08-05 17:38:06.152545
2019-08-05 17:47:21.696106
2019-08-05 17:47:36.695894
2019-08-05 17:47:51.695905
2019-08-05 17:50:21.875431
2019-08-05 17:50:36.901650
2019-08-05 17:50:51.928500
2019-08-05 17:51:06.951234
2019-08-05 17:51:21.970940
2019-08-05 17:51:36.975607
2019-08-05 17:51:51.997920
2019-08-05 17:55:07.209115
2019-08-05 17:55:22.220632
2019-08-05 17:55:37.228242
2019-08-05 17:55:52.248871
2019-08-05 17:56:07.250847
2019-08-05 17:56:22.255708
2019-08-05 17:56:37.271999
2019-08-05 17:56:52.301388
2019-08-05 18:24:54.096784
2019-08-05 18:25:09.115747
2019-08-05 18:25:24.125967
2019-08-05 18:25:39.144566
2019-08-05 18:25:54.165763
2019-08-05 18:26:09.195244
2019-08-05 18:26:24.220001
2019-08-05 18:26:39.230683
2019-08-05 18:50:40.644112
2019-08-05 18:50:55.668570
2019-08-05 18:51:10.688363
2019-08-05 19:13:41.920701
2019-08-05 19:13:56.922940
2019-08-05 19:14:11.932202
2019-08-05 19:14:26.938413
2019-08-05 19:14:41.953686
2019-08-05 19:14:56.960080
2019-08-05 19:52:14.380600
2019-08-05 19:52:29.380739
2019-08-05 19:52:44.383024
2019-08-05 19:57:14.626576
2019-08-05 19:57:29.654678
2019-08-05 19:57:44.684959
2019-08-05 20:14:00.623090
2019-08-05 20:14:15.648830
2019-08-05 20:14:30.671516
2019-08-05 20:21:53.796489
2019-08-05 20:22:08.818946
2019-08-05 20:22:23.841057
2019-08-05 20:22:38.846267
2019-08-05 20:22:53.868583
2019-08-05 20:23:08.885193
2019-08-05 20:23:23.895201
2019-08-05 20:23:38.895605
2019-08-05 20:23:53.896195
2019-08-05 20:24:08.900986
2019-08-05 20:24:23.926280
2019-08-05 20:24:38.948228
2019-08-05 20:24:53.977461
2019-08-05 20:25:08.982632
2019-08-05 20:25:23.997956
2019-08-05 20:47:40.312163
2019-08-05 20:47:55.316793
2019-08-05 20:48:10.329716
2019-08-05 20:48:25.338584
2019-08-05 20:48:40.359436
2019-08-05 20:48:55.365012
2019-08-05 20:49:10.385370
2019-08-05 20:49:25.411547
2019-08-05 20:49:40.425536
2019-08-05 20:49:55.426249
2019-08-05 20:50:10.450416
2019-08-05 21:11:26.845486
2019-08-05 21:11:41.854944
2019-08-05 21:11:56.876286
2019-08-05 21:12:11.905891
2019-08-05 21:12:26.918225
2019-08-05 21:12:41.923508
2019-08-05 21:12:56.937204
2019-08-05 21:18:12.209876
2019-08-05 21:18:27.210519
2019-08-05 21:18:42.213493
2019-08-05 21:18:57.237574
2019-08-05 21:31:12.977029
2019-08-05 21:31:28.001595
2019-08-05 21:31:43.029471
2019-08-05 21:31:58.032337

[urihersh]

I'm experiencing the same thing. I got 6 shelly 2.5, all of them are discovered, but becomes unavailable quickly. I run homeassistant on a docker on ubuntu VM (bridged to my home network).

[ispiropoulos]

Experiencing the exact same issue with several Shelly 2.5 and 1 Shelly One

2019-09-05 18:07:53.585711
2019-09-05 18:08:08.534303
2019-09-05 18:08:38.639504
2019-09-05 18:10:53.910280
2019-09-05 18:11:38.966643
2019-09-05 18:11:54.020254
2019-09-05 18:13:39.082594
2019-09-05 18:14:39.192504

[jotacor]

Same problem here.
I'm running HA in a docker, Ubuntu server 18.04 host, over WLAN.

[chemelli74]

@gaggio, @ispiropoulos, @jotacor, can you retest with 0.1.6b7 and latest firmware v1.5.9 as well ?

Thank you,

Simone

[chemelli74]

Hi all, just to inform that after 3 weeks of inactivities on very old thread, I'll proceed to close them.
If later one someone verify that the issue is still there, a new issue need to be opened.

Thank you,

Simone

[rufik]

It''s fine for me now.

[chemelli74]

No update for 21 days, closing

Simone