Core: Netatmo weather sensors not working anymore in 0.104.0

[rosch100]

Home Assistant release with the issue:
0.104.0

Last working Home Assistant release (if known):
0.103.x

Operating environment (Hass.io/Docker/Windows/etc.):
hass.io

Integration:
https://www.home-assistant.io/integrations/netatmo/

Description of problem:
Since the upgrade to 0.104.0, my netatmo weather sensors are not connected anymore. The thermostats are fine though.
There are repeated warnings in the log:

2020-01-16 17:01:38 WARNING (SyncWorker_13) [homeassistant.components.netatmo.sensor] No data found for ....

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

netatmo:
  api_key: apikeyapikey
  secret_key: keykeykey
  username: [email protected]
  password: xxxxxxxxx

[probot-home-assistant[bot]]

Hey there @cgtobi, mind taking a look at this issue as its been labeled with a integration (netatmo) you are listed as a codeowner for? Thanks!

[cgtobi]

This might be a duplicate of https://github.com/home-assistant/home-assistant/issues/30291

[rosch100]

Could be possible - my account is connected to four stations, of which one is currently not available.

[cgtobi]

Ok, that is fixed upstream already. I’ll prioritise it.

[cgtobi]

@rosch100 in case you have a dev or test environment could you possibly give this fix a try?

[rosch100]

I do not have a test environment, but I’d like to try this fix anyway. However, I’m not sure how to do that. Where could I find docs about it?

Am 17.01.2020 um 13:06 schrieb cgtobi [e-mail]:

@rosch100 https://github.com/rosch100 in case you have a dev or test environment could you possibly give this fix https://github.com/home-assistant/home-assistant/pull/30882 a try?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/home-assistant/home-assistant/issues/30842?email_source=notifications&email_token=ABL2USEOXP55I6LXRTSJ5T3Q6GNLNA5CNFSM4KHWPAT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHPPTI#issuecomment-575600589, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABL2USCIOIBA6JSTU4R3JJ3Q6GNLNANCNFSM4KHWPATQ.

[cgtobi]

As it is more or less just a update of a dependency you can try this:

pip3 install pyatmo
pip3 show pyatmo
# should give you version 3.2.2
hass --skip-pip

Running hass without --skip-pip will downgrade pyatmo to the original version.

[rosch100]

Somehow, I’d need to incorporate your changes… (How?)
When I pip3 install pyatmo inside the homeassistant container (I’m on hassio), I get:
bash-5.0# pip3 install pyatmo
Requirement already satisfied: pyatmo in /usr/local/lib/python3.7/site-packages (3.1.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/site-packages (from pyatmo) (2.22.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/site-packages (from requests->pyatmo) (2019.11.28)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.7/site-packages (from requests->pyatmo) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/site-packages (from requests->pyatmo) (1.25.7)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/site-packages (from requests->pyatmo) (3.0.4)
bash-5.0# pip3 show pyatmo
Name: pyatmo
Version: 3.1.0

Am 17.01.2020 um 13:47 schrieb cgtobi [e-mail]:

As it is more or less just a update of a dependency you can try this:

pip3 install pyatmo
pip3 show pyatmo

should give you version 3.2.2

hass --skip-pip
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/home-assistant/home-assistant/issues/30842?email_source=notifications&email_token=ABL2USHQMDDRHCTTVAXJD7LQ6GSFLA5CNFSM4KHWPAT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHSLPI#issuecomment-575612349, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABL2USAP3QUT56GD4VGXQNTQ6GSFLANCNFSM4KHWPATQ.

[cgtobi]

Try pip install --upgrade pyatmo. The other changes are not so relevant right now.

[rosch100]

Does not work unfortunately:

bash-5.0# pip install —upgrade pyatmo
ERROR: Invalid requirement: '—upgrade'

[springstan]

Try two hyphens in front of upgrade, e.g. pip install --upgrade pyatmo

[rosch100]

Same result:

bash-5.0# pip install —-upgrade pyatmo
ERROR: Invalid requirement: '—-upgrade'

[springstan]

How about pip3 install pyatmo==3.2.2?

[rosch100]

pip3 install pyatmo==3.2.2 works! Now the question is how to restart the docker container homeassistant/raspberrypi3-homeassistant:0.104.1 without resetting this...

[springstan]

Well you could make the same change for the manifest.json file as in #30882 or wait for the next release which will include this fix.

[rosch100]

Restarting the raspi did not revert and I can confirm now, that the fix works.
Thanks for all your efforts!

[cgtobi]

Thanks for testing. Did you test only the new pyatmo module or the changes to sensor.py as well?

[rosch100]

@cgtobi Tried both "pyatmo 3.2.2" and pyatmo + sensor.py.
Both work!

[rosch100]

@cgtobi After a while, netatmo sensors and climate settings stop working now. After a restart, everything works again for a while. That's why I did not find out about it initially.
There are a lot of recurring errors in the log. These two errors are logged for each entity:

oauthlib.oauth2.rfc6749.errors.InvalidClientError: (invalid_client)
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired) 

Full error log for one entity:

oauthlib.oauth2.rfc6749.errors.InvalidClientError: (invalid_client) 
2020-01-21 07:50:10 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.netatmo_terrasse_temperatur_battery_percent fails
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 114, in query
    return self._oauth.post(url=url, data=params, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 478, in request
    url, http_method=method, body=data, headers=headers
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 198, in add_token
    raise TokenExpiredError()
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired) 
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 461, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 257, in update
    self.netatmo_data.update()
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 576, in update
    self.station_data = self.data_class(self.auth)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/weather_station.py", line 24, in __init__
    resp = self.authData.post_request(url=self.urlReq)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 119, in post_request
    resp = query(url, params, timeout, 3)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 116, in query
    self._oauth.token = self.refresh_tokens()
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 88, in refresh_tokens
    token = self._oauth.refresh_token(_AUTH_REQ)
  File "/usr/local/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 446, in refresh_token
    self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
    self.token = parse_token_response(body, scope=scope)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
    validate_token_parameters(params)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 438, in validate_token_parameters
    raise_from_error(params.get('error'), params)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 405, in raise_from_error
    raise cls(**kwargs)
oauthlib.oauth2.rfc6749.errors.InvalidClientError: (invalid_client) 
....

[rosch100]

Logged the first occurrence of the error:

2020-01-21 11:11:49 WARNING (SyncWorker_5) [homeassistant.components.netatmo.sensor] No data found for Borkheide Borkheide Regen
2020-01-21 11:11:49 WARNING (SyncWorker_0) [homeassistant.components.netatmo.sensor] No data found for Borkheide Borkheide Terrasse
2020-01-21 11:11:49 WARNING (SyncWorker_8) [homeassistant.components.netatmo.sensor] No data found for Borkheide Borkheide Terrasse
2020-01-21 11:12:20 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.netatmo_borkheide_borkheide_regen_radio fails
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 114, in query
    return self._oauth.post(url=url, data=params, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 478, in request
    url, http_method=method, body=data, headers=headers
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 198, in add_token
    raise TokenExpiredError()
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired) 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 461, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 257, in update
    self.netatmo_data.update()
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 576, in update
    self.station_data = self.data_class(self.auth)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/weather_station.py", line 24, in __init__
    resp = self.authData.post_request(url=self.urlReq)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 119, in post_request
    resp = query(url, params, timeout, 3)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 116, in query
    self._oauth.token = self.refresh_tokens()
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 88, in refresh_tokens
    token = self._oauth.refresh_token(_AUTH_REQ)
  File "/usr/local/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 446, in refresh_token
    self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
    self.token = parse_token_response(body, scope=scope)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
    validate_token_parameters(params)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 438, in validate_token_parameters
    raise_from_error(params.get('error'), params)
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 405, in raise_from_error
    raise cls(**kwargs)
oauthlib.oauth2.rfc6749.errors.InvalidClientError: (invalid_client) 
2020-01-21 11:12:20 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.netatmo_schlafzimmer_battery_percent fails
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 114, in query
    return self._oauth.post(url=url, data=params, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 478, in request
    url, http_method=method, body=data, headers=headers
  File "/usr/local/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 198, in add_token
    raise TokenExpiredError()
oauthlib.oauth2.rfc6749.errors.TokenExpiredError: (token_expired) 

During handling of the above exception, another exception occurred:
......

[cgtobi]

This is probably related to changes in the pyatmo module. I'd suggest to roll back the local changes and wait for the next beta (next week).