Core: Somfy, reduce polling rate.

[chriscn]

The problem

I have just recieved an email from Somfy detailing that we are polling the endpoints very highly. I wondered if we could add a polling rate

Environment

• Home Assistant Core release with the issue: 0.115.6
• Last working Home Assistant Core release (if known):
• Operating environment (OS/Container/Supervised/Core): Docker on a Rapsberry Pi
• Integration causing this issue: somfy
• Link to integration documentation on our website: [https://www.home-assistant.io/integrations/somfy/)(https://www.home-assistant.io/integrations/somfy/)

Problem-relevant configuration.yaml

Traceback/Error logs

Additional information

[Tacchi1983]

Got the same e-mail !

[probot-home-assistant[bot]]

Hey there @tetienne, mind taking a look at this issue as its been labeled with an integration (somfy) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[Gotanius]

Got the same letter. Somfy is setting a deadline at 2020/10/31 after which they might start banning.

[christopheBfr]

I sent an email to Somfy API developer support to ask what is the allowable poll rate , also I ask them to extend the dead line, Home Assistant sprint duration is not compatible with the 31th of october deadline.

I also ask them to not limit or ban us (their customers !) before we find a solution.

@tetienne The job is for you.

Merci @tetienne , nous comptons sur toi !

[Hypfer]

This _might_ be considered off-topic, but FYI at least for somfy rts, there seems to be a solution which avoids their cloud and therefore this issue altogether.

It can be found here: https://github.com/filipmaelbrancke/ha-rflink-rts

Furthermore, it should be possible to locally control newer somfy io devices via the velux home assistant integration.
You might want to check that out as well

[thomasloven]

Do not reply to this issue only to say you also got the email. Just leave a thumbs up on the original post.

[GijsvanDulmen]

I also send some feedback towards Somfy (beginning of this year and today again) about providing a way of receiving events through a webhook push or such. This would seem a better way to reduce the polling. I would suggest providing the same feedback.

[christopheBfr]

I also send some feedback towards Somfy (beginning of this year and today again) about providing a way of receiving events through a webhook push or such. This would seem a better way to reduce the polling. I would suggest providing the same feedback.

Perhaps also another way can be to have a simple REST API on the Tahoma box itself to poll the Tahoma box locally rather than to reach Somfy site... But we can dream :-)

[balloob]

We will include a fix in 116 to reduce the polling rate. If anyone wants to open a PR, please. Otherwise I will hack something together but I won't be able to test it. Will just change some knobs.

[balloob]

They talk about Somfy, is it just Somfy or also Tahoma integration ?

[iMicknl]

They talk about Somfy, is it just Somfy or also Tahoma integration ?

@balloob the TaHoma integration is broken for a while, see (Home Assistant Alert). We just opened the first PR to fix it, however this will take some time. https://github.com/home-assistant/core/pull/41267

We will include a fix in 116 to reduce the polling rate. If anyone wants to open a PR, please. Otherwise I will hack something together but I won't be able to test it. Will just change some knobs.

This could help, but I am afraid the issue is more complex. I believe that @tetienne already contacted Somfy, to understand what the culprit could be...

In the new TaHoma (custom) component we are also tracking this issue: https://github.com/iMicknl/ha-tahoma/issues/289. It is still unclear what the real issue is, since we fully rewrote the TaHoma component to limit the use of /setup and to use an event listener which can be pulled up to once per second.
The Somfy component doesn't even use this /setup endpoint, thus I am not sure if just lowering the polling interval will solve this issue.

[tetienne]

Hi all, I will contact the Somfy support with different Somfy maintainers (like Homey) to have a clear response on what they expect. As said @iMicknl, the Somfy component does not have any /setup entry point, so that's unclear on what they expect from us.

[tetienne]

Email sent to the Somfy support. I hope they will clarify their message.

[WhiteDogBe]

Such arrogance of Somfy. They provide no local connectivity, promise HomeKit support for years and don't seem to ever respond to any developer requests. If they Google for 5 minutes they know the 95% of integrations like HA, Homey, ... and if they take 5 minutes they can send you all a nice mail with what they would like to address. All this for an API, which is basically the exchange of some text, which any decent cheapass webserver can do 1000 of per second. Things like ratelimiting and caching would be the first I'd implement in my API...

But nope, they prefer to take the Logitech route, piss off all their users and generate some bad PR on the internet. These guys live on a different planet.

[tetienne]

So I got an answer from the Somfy support:

I am preparing a more complete answer with the team to clarify as soon as possible.

I hope we will get really more details about what they expect.

@WhiteDogBe At least, the Logitech Harmony has a local API ;)

[louwersj]

Such arrogance of Somfy. They provide no local connectivity, promise HomeKit support for years and don't seem to ever respond to any developer requests. If they Google for 5 minutes they know the 95% of integrations like HA, Homey, ... and if they take 5 minutes they can send you all a nice mail with what they would like to address. All this for an API, which is basically the exchange of some text, which any decent cheapass webserver can do 1000 of per second. Things like ratelimiting and caching would be the first I'd implement in my API...

But nope, they prefer to take the Logitech route, piss off all their users and generate some bad PR on the internet. These guys live on a different planet.

I think this shows a bit the face of a management that is unwilling or unable to face the new way consumers want to use products. The issue being, consumers like to integrate and they have the desire to move away from products that can't integrate. The fact that their backend is having issues should be a blessing, as it is an indicator people like to use their product. I would argue, the more integrations the better, the more API calls the better. It means your products are adopted, integrated in third party products and you get free marketing (a lot)

The solution to this "problem" could have been three folded in my opinion.
1) Hire some developers that will actively donate time and code to this project as it is most likely one of the biggest sources of API calls
2) write some propper SDKs to share with the community and upfront set limits, however... make them fair limits.
3) Fix the backend to that it can easily accommodate a large set of request.

Just my 2 cents

[iMicknl]

The solution to this "problem" could have been three folded in my opinion.

1. Hire some developers that will actively donate time and code to this project as it is most likely one of the biggest sources of API calls
2. write some propper SDKs to share with the community and upfront set limits, however... make them fair limits.
3. Fix the backend to that it can easily accommodate a large set of request.

The main issue is that Somfy's official API is still very limited in terms of device support. Many devices are not supported and the way you can interact with them is limited as well. Still no support for webhooks, or even a basic event fetching mechanism. The roadmap is unclear, but based on the conversations with Somfy that have been shared with me, they don’t invest heavily in their API.

I still believe that their 'warning' is not based on recent data. Both of the integrations don't depend heavily on the /setup endpoint. The previous tahoma integration (still in core) did this, thus possibly their message is based on historical data.

[tetienne]

Hi all,

So I got an answer from the Somfy support. And the update is not really nice... To sum up the content of the email:

They expect from the 3rd party plugins to behave as the tahomalink UI. Meaning that we should use the API when the end-user truly needs it. The device states can be updated when the user is manually connected to the dashboards.

in our opinion (@iMicknl, @vlebourl and myself), the solution to limit the use to UI interactions just means the end of this integration in Home Assistant.

Hopefully, there is one positive sentence in this email:

If you don't agree, please let me know what we could miss.

[christopheBfr]

Hopefully, there is one positive sentence in this email:

If you don't agree, please let me know what we could miss.

So, a technical advice we can give to somfy ingenieers, is to implement webhooks to maintain the synchro between HA and Tahoma without polling the Somfy api site each 2 minutes to save CPU / bandwidth consuption... and let us to sparingly change the cover position few times per day... Netatmo and al other companies succeed in doing webhooks , why Somfy (with the help of a python trainee perhaps ) cannot do the same ?

[iMicknl]

So, a technical advice we can give to somfy ingenieers, is to implement webhooks to maintain the synchro between HA and Tahoma without polling the Somfy api site each 2 minutes to save CPU / bandwidth consuption... and let us to sparingly change the cover position few times per day... Netatmo and al other companies succeed in doing webhooks , why Somfy (with the help of a python trainee perhaps ) cannot do the same ?

To be honest, I have no confidence in Somfy delivering a better API. They are not prioritizing their Somfy Open API at all. They announced HomeKit support in early 2018 and today they only support the Indoor / Outdoor camera via HomeKit.

Hopefully they will allow the event listener 24/7 or add a webhook approach. I am using the new TaHoma integration for a few months and I haven't received the email from Somfy. My polling interval is set to a minute.

[cjoha]

Is there a way we can frame this in a positive way for the Somfy engineers? I spoke to them on email today and they seemed positive to engage the community.

If we have an approach which would lessen their API charges or load then surely we should try to make that recommendation. Personally I don’t want to see the integration go away, I call Somfy blinds on a daily basis (sun-rise / sun-set), as well as integrating them with Alexa routines via Nabu Casa.

[iMicknl]

Is there a way we can frame this in a positive way for the Somfy engineers? I spoke to them on email today and they seemed positive to engage the community.

Currently I have seen quite some communication, but no actual steps to reach out to the community. They could also proactively have reached out to some of the developers on GitHub, since almost all of the integrations / libraries add a User Agent.
I am happy to have 'positive' discussions, but it would be good to see any concrete actions from their side :-).

If we have an approach which would lessen their API charges or load then surely we should try to make that recommendation. Personally I don’t want to see the integration go away, I call Somfy blinds on a daily basis (sun-rise / sun-set), as well as integrating them with Alexa routines via Nabu Casa.

We could have a look if we can only open the session and event listener when you initiate an action in Home Assistant, however this means that you need to control your devices only via Home Assistant. Changes outside Home Assistant won't be reflected.
This will also mean that sensors (door, co2 and others) won't be useable, since you won't know the state. In the end, you don't have any benefit of having state aware IO devices, if you have to treat them like RTS devices.

[Hypfer]

Is there a way we can frame this in a positive way for the Somfy engineers?

I don't think that the engineers are at fault here.
The issue in general, the wording of the initial email and the "open API" which isn't ment to be used as one all have the same completely detached management fingerprints all over them.

The engineers are most likely just relaying this nonsense.

[DarkwellZ]

Just a thought (I'm basing this on my experience with there connexoon box):

Are we sure there is an actual problem? It seems the e-mail went out to everyone using a their API.
I do not feel Home Assistant is abusing their API. Can it be that some apps are inefficiently programmed and calling there API a lot and as a result they send out this e-mail to all of us?

I don't know how Home Assistant is calling the API, but it doesn't seem to frequent. If someone in my house (read, my wife) uses the remote from somfy, the status of the blinds doesn't seem to update in Home Assistant. My conclusion, it polls the state after it send a command (?), this seems very conservative and in line with if not use less API calls as the connexoon app which already updates everything on load.

just means the end of this integration in Home Assistant.

When I read the above posts it sounds like we are already expecting to get banned. Maybe it will turn out to be less integration killing then we fear?

[vlebourl]

Several points regarding your comment. First the email only went out to the 1200 heaviest users, all platforms included (meaning homey, openhab and ha if I'm correct), so not every users yet, fortunately. Second, the email was intended for users of the unofficial API. For HA that would be the tahoma integration, not the somfy one. The core Tahoma integration is broken ATM but we are working on it. Finally regarding the use of the remote, this is a problem on Somfy's side, it seems their not retrieving the state of a device operated through a physical remote unless you open the phone or web app, hence the state not being updated in ha.

[Mosibi]

Second, the email was intended for users of the unofficial API. For HA that would be the tahoma integration, not the somfy one.

I use the HA Somfy integration and not the Tahoma integration, but also got the email.

[vlebourl]

I think they got mixed up... Another thing they can't do properly 😁
But in the answer @tetienne got it is clearly said:

First of all, we would like to apologize for the lack of clarity for users of the official SOMFY OPEN API who received the email. The communication was targeting users of the unofficial one.

[OrbitingOcelot]

While I agree the situation is not ideal, I can only tell you that it could be much much worse, just take a look at LG and their ThinQ API. Looks nice and shiny on the website, in reality they dont give you API keys unless you partner with them ($$$), and all integrations have to reverse engineer the mobile app, with calls to cryptic backends that give out responses in Korean etc. LG Thinq API sucks and LG does not give a damn about their users or 3rd party open source devs.

In comparison, at least on the Somfy side there seems to be some willingness to engage in dialogue and find a solution that works for everybody, so I am grateful that the devs have reached out and that the discussion is going.

I for myself could live with a reduced polling rate without any issues. Most important would be that the cover status is updated some seconds after an operation is done via HA to check cover position/results. Also maybe periodic polling updates of the covers could be made optional in the config, with some very conservative default setting. I for myself am not using any remotes at all, everything is done via HA, and I am sure there are similar users, for which no polling or very infrequent polling would suffice. If anyone needs shorter intervals, they could change the config (on their own responsibility of getting throttled/banned etc.).

[bdraco]

If all devices are assumed state (RTS) #42462 reduces the polling frequency to once per hour. That should help reduce the load on somfy's servers without any noticeable change for RTS users.

This applies only to the somfy integration.

[pawlizio]

That's why I changed to velux component using Velux KLF200 gateway to control all my io-homecontrol devices (all of them are from Somfy). Now I'm able to control them locally without relying on a cloud service. Another plus is the reaction time of control commands, they are now much quicker than via Somfy cloud.

[MSkeem]

That's why I changed to velux component using Velux KLF200 gateway to control all my io-homecontrol devices (all of them are from Somfy). Now I'm able to control them locally without relying on a cloud service. Another plus is the reaction time of control commands, they are now much quicker than via Somfy cloud.

That sounds really great. Do you have some instructions (a link or something) on how to do this?

[pawlizio]

No, I have no detailed instruction, I just bought a KLF200, connected it via LAN to my network.
Due to the fact that I already had a Tahoma Box and a configured io-homecontrol network, I just had to share the secret key with the Velux KLF200 device. On tahomalink.com there is a functionality to send the sectret key, and in the Webinterface of KLF200 you have the function to receive secret keys. So it took me just a few minutes to discover all of my devices with KLF200. Now I'm able to control via both Tahoma Box and Velux KLF200, but actually I'm only using KLF200. You may just check in advance whether all of your devices are supported by KLF200. You can check chapter Appendix 2 of the below API specification if your device is listed: https://velcdn.azureedge.net/-/media/com/api/klf200/technical%20specification%20for%20klf%20200%20api-ver3-18.pdf

[MSkeem]

Nice - it sounds like I have more or less the same setup as you do. 5 awnings connected to Tahoma, which works ok, but with some delay, as you also mention (which is quite annoying). So I might also be able to get my devices to the KLF in that simple way. I will buy one tomorrow :)

[MSkeem]

No, I have no detailed instruction, I just bought a KLF200, connected it via LAN to my network.
Due to the fact that I already had a Tahoma Box and a configured io-homecontrol network, I just had to share the secret key with the Velux KLF200 device. On tahomalink.com there is a functionality to send the sectret key, and in the Webinterface of KLF200 you have the function to receive secret keys. So it took me just a few minutes to discover all of my devices with KLF200. Now I'm able to control via both Tahoma Box and Velux KLF200, but actually I'm only using KLF200. You may just check in advance whether all of your devices are supported by KLF200. You can check chapter Appendix 2 of the below API specification if your device is listed: https://velcdn.azureedge.net/-/media/com/api/klf200/technical%20specification%20for%20klf%20200%20api-ver3-18.pdf

Which firmware are you running on the KLF - the latest? version 2.0.0.71, date 2018-09-27.

[pawlizio]

Yes, the latest one. If you miss some functionality, like tilt function, use my HACS custom component: https://github.com/pawlizio/my_velux