Core: Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds

[MrUserY]

The problem

Allready discussed this error on Discord with cgtobi. Putting it in github to track the progress of the issue out of curiosisty. No hurry everyhing works. But the system seems to be unhappy. The log is shouting daily:

Error handling request
Update for camera.netatmo_presence_achtertuin fails
Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
Too many retries

Environment

Multiple devices in house displaying the lovelace frontend using the offical homeassistant app.

arch | armv7l
-- | --
dev | false
docker | true
hassio | true
os_name | Linux
os_version | 4.19.114-v7l
python_version | 3.7.7
timezone | Europe/Amsterdam
version | 0.109.6
virtualenv | false

• Home Assistant Core release with the issue: Unknown
• Last working Home Assistant Core release (if known): It works
• Operating environment (Home Assistant/Supervised/Docker/venv): Home Assistant 0.109.6
  Supervisor: 222 and HassOS 3.13
• Integration causing this issue: Netatmo
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/netatmo/

Problem-relevant configuration.yaml

netatmo:
client_id: thesecretid
client_secret: thesecretcode

Traceback/Error logs

2020-05-09 17:20:31 ERROR (MainThread) [homeassistant.components.netatmo] Error during webhook registration - 400 - Bad request - Allowed ports for webhooks are 443 and 80. (21) when accessing 'https://api.netatmo.com/api/addwebhook'
2020-05-09 22:35:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-09 22:51:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-09 22:51:24 ERROR (SyncWorker_11) [pyatmo.auth] Too many retries
2020-05-09 22:51:24 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_presence_achtertuin fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 472, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 212, in update
    self._data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 261, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 30, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 168, in post_request
    if "application/json" in resp.headers.get("content-type")
AttributeError: 'NoneType' object has no attribute 'headers'
2020-05-09 23:38:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-09 23:38:24 ERROR (SyncWorker_6) [pyatmo.auth] Too many retries
2020-05-09 23:38:24 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_presence_achtertuin fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 472, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 212, in update
    self._data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 261, in update
    self.camera_data = pyatmo.CameraData(self.auth, size=100)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 30, in __init__
    resp = self.authData.post_request(url=_GETHOMEDATA_REQ, params=postParams)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 168, in post_request
    if "application/json" in resp.headers.get("content-type")
AttributeError: 'NoneType' object has no attribute 'headers'
2020-05-10 00:10:17 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-10 00:10:48 WARNING (MainThread) [homeassistant.helpers.entity] Update of camera.netatmo_presence_achtertuin is taking over 10 seconds
2020-05-10 00:10:56 ERROR (SyncWorker_17) [pyatmo.auth] Too many retries
2020-05-10 00:10:56 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_presence_achtertuin fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 472, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 217, in update
    cid=self._camera_id
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 284, in camera_urls
    temp_local_url = check_url(vpn_url)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 277, in check_url
    resp = self.authData.post_request(url=f"{url}/command/ping")
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 168, in post_request
    if "application/json" in resp.headers.get("content-type")
AttributeError: 'NoneType' object has no attribute 'headers'
2020-05-10 00:11:03 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 437, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.7/http/client.py", line 457, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.7/http/client.py", line 501, in readinto
    n = self.fp.readinto(b)
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 99, in camera_image
    f"{self._localurl}/live/snapshot_720.jpg", timeout=10
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 683, in send
    r.content
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/real_ip.py", line 39, in real_ip_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 73, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 127, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 125, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 490, in get
    return await self.handle(request, camera)
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 507, in handle
    image = await camera.async_camera_image()
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 376, in async_camera_image
    return await self.hass.async_add_executor_job(self.camera_image)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 118, in camera_image
    cid=self._camera_id
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 284, in camera_urls
    temp_local_url = check_url(vpn_url)
  File "/usr/local/lib/python3.7/site-packages/pyatmo/camera.py", line 277, in check_url
    resp = self.authData.post_request(url=f"{url}/command/ping")
  File "/usr/local/lib/python3.7/site-packages/pyatmo/auth.py", line 158, in post_request
    f"{resp.status_code} - "
KeyError: 502

Additional information

Using 2 netatmo presence's.
Using the new config flow (:8123/config/integrations> choose yaml config)
Multiple display's in the house (official app on android and ios).

Sometimes netatmo's don't show camera picture in lovelace when the ios/android app wasn't used for a while and lost connection. You need to force refresh then the get the camera's again, while other integrations just (seem to) display, but offcourse their an other type of integration.

[probot-home-assistant[bot]]

netatmo documentation
netatmo source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @cgtobi, mind taking a look at this issue as its been labeled with a integration (netatmo) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[Trouble-51]

Same here, more than 10 seconds, too many retries etc. It’s clogging up the logs! Getting rid of all these messages would be a real improvement. Thanks for looking into the issues.

[cgtobi]

Hi, I’d strongly encourage you to test the pre beta custom component until it’s released.

https://community.home-assistant.io/t/testers-wanted-netatmo-integration/203729?u=cgtobi

[MrUserY]

Okeejj still looking for beta testers or are you about to launch?

[cgtobi]

Beta testers are still welcome. Especially with concrete issues.

[MrUserY]

Using beta. One error (below) that clogs up the logs. Hope you can disable this warning:

Log Details (WARNING)
Logger: homeassistant.components.light
Source: helpers/entity_platform.py:202
Integration: Light (documentation, issues)
First occurred: July 20, 2020, 7:56:46 PM (492 occurrences)

[MrUserY]

For the record:

2020-07-21 20:44:59 WARNING (MainThread) [homeassistant.components.light] Platform netatmo not ready yet. Retrying in 30 seconds.

2020-07-21 20:47:02 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/netatmo/__init__.py", line 157, in register_webhook
    hass.data[DOMAIN][entry.entry_id][AUTH].addwebhook, webhook_url
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/netatmo/pyatmo/auth.py", line 216, in addwebhook
    resp = self.post_request(WEBHOOK_URL_ADD, post_params)
  File "/config/custom_components/netatmo/pyatmo/auth.py", line 167, in post_request
    f"{resp.status_code} - "
pyatmo.exceptions.ApiError: 400 - Bad request - Allowed ports for webhooks are 443 and 80. (21) when accessing 'https://api.netatmo.com/api/addwebhook'

2020-07-21 20:47:03 WARNING (MainThread) [homeassistant.components.light] Platform netatmo not ready yet. Retrying in 60 seconds.

2020-07-21 20:48:04 WARNING (MainThread) [homeassistant.components.light] Platform netatmo not ready yet. Retrying in 90 seconds.

Etc.

[MrUserY]

The "lights" warning issue is solved with the new version today!

Ain't it possible to solve the "ApiError: 400 - Bad request - Allowed ports for webhooks are 443 and 80" etc also?
For people who don't use webhooks (like me)?

Just thinking out loud to make the standard error-log nice and clean for real troublesome issues :-)

[cgtobi]

That exception should actually be caught and return a nicer error message like:
Error during webhook registration - 400 - Bad request - Allowed ports for webhooks are 443 and 80. (21) when accessing 'https://api.netatmo.com/api/addwebhook'

I tried to replicate the behaviour but so far no luck. You still get that exception with the latest version, right?

[MrUserY]

The full error is:

2020-07-22 13:24:04 ERROR (MainThread) [custom_components.netatmo] Error during webhook registration - 400 - Bad request - Allowed ports for webhooks are 443 and 80. (21) when accessing 'https://api.netatmo.com/api/addwebhook'

This error is persistent allready for a long time, also with the official HA version. I've reported/We've talked about this about +/- a year ago I think. Even with clean installing everything. I was allready used to it, and am not using webhooks just 2 camera's only :-) But no error in the log would be nicer :-)

[MrUserY]

Updated to 113.1 and new Netato hacs from yesterday (dunnow the version).

Error stil present: ERROR (MainThread) [custom_components.netatmo] Error during webhook registration - 400 - Bad request - Allowed ports for webhooks are 443 and 80. (21) when accessing 'https://api.netatmo.com/api/addwebhook'

I've got a new error that mentions Netatmo. IP's and UUID's are anonymized.

2020-07-26 06:47:50 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xa56bc970>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.43.23', port=80): Max retries exceeded with url: /bdee54c12344fad91500c38e8b555446/live/snapshot_720.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xa56bc970>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/config/custom_components/netatmo/camera.py", line 199, in camera_image
    response = requests.get(
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.43.23', port=80): Max retries exceeded with url: /efae54c23445fac91500c38e8b837466/live/snapshot_720.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xa56bc970>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/real_ip.py", line 39, in real_ip_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 73, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 127, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 129, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 499, in get
    return await self.handle(request, camera)
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 516, in handle
    image = await camera.async_camera_image()
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 385, in async_camera_image
    return await self.hass.async_add_executor_job(self.camera_image)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/netatmo/camera.py", line 215, in camera_image
    self._data.update_camera_urls(camera_id=self._id)
  File "/config/custom_components/netatmo/pyatmo/camera.py", line 177, in update_camera_urls
    temp_local_url = check_url(vpn_url)
  File "/config/custom_components/netatmo/pyatmo/camera.py", line 167, in check_url
    resp = self.auth.post_request(url=f"{url}/command/ping")
  File "/config/custom_components/netatmo/pyatmo/auth.py", line 167, in post_request
    f"{resp.status_code} - "
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[cgtobi]

Thanks for reporting this. I'll make sure to capture these exceptions. The error about the wrong webhook ports is an error and will stay. I'll take a look at making this optional somehow in the future to explicitly disable webhook usage but it really provides much better responsiveness overall except weather stations and it is an error reported by the API, so I think it is beneficial to notify the user accordingly.

[MrUserY]

Dunnow what it means, yesterday:

Logger: homeassistant
Source: custom_components/netatmo/pyatmo/camera.py:175
First occurred: August 4, 2020, 6:40:02 AM (1 occurrences)
Last logged: August 4, 2020, 6:40:02 AM

Error doing job: Task exception was never retrieved

Traceback (most recent call last):
File "/config/custom_components/netatmo/data_handler.py", line 89, in async_update
await self.async_fetch_data(
File "/config/custom_components/netatmo/data_handler.py", line 113, in async_fetch_data
self.data[data_class_entry] = await self.hass.async_add_executor_job(
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/config/custom_components/netatmo/pyatmo/camera.py", line 122, in __init__
self.update_camera_urls(camera_id)
File "/config/custom_components/netatmo/pyatmo/camera.py", line 177, in update_camera_urls
temp_local_url = check_url(vpn_url)
File "/config/custom_components/netatmo/pyatmo/camera.py", line 175, in check_url
return resp.get("local_url")
AttributeError: 'NoneType' object has no attribute 'get'

AND

Log Details (ERROR)

Logger: custom_components.netatmo.pyatmo.auth
Source: custom_components/netatmo/pyatmo/auth.py:156
Integration: Netatmo (documentation)
First occurred: August 4, 2020, 6:40:02 AM (2 occurrences)
Last logged: August 4, 2020, 6:42:07 AM

Too many retries

[Trouble-51]

I don’t see any cameras anymore but also no error in the log

From: MrUser [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Wednesday, 5 August 2020 at 13:54
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Comment [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

Today (dunnow what it means):

Logger: homeassistant
Source: custom_components/netatmo/pyatmo/camera.py:175
First occurred: August 4, 2020, 6:40:02 AM (1 occurrences)
Last logged: August 4, 2020, 6:40:02 AM

Error doing job: Task exception was never retrieved

Traceback (most recent call last):
File "/config/custom_components/netatmo/data_handler.py", line 89, in async_update
await self.async_fetch_data(
File "/config/custom_components/netatmo/data_handler.py", line 113, in async_fetch_data
self.data[data_class_entry] = await self.hass.async_add_executor_job(
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/config/custom_components/netatmo/pyatmo/camera.py", line 122, in init
self.update_camera_urls(camera_id)
File "/config/custom_components/netatmo/pyatmo/camera.py", line 177, in update_camera_urls
temp_local_url = check_url(vpn_url)
File "/config/custom_components/netatmo/pyatmo/camera.py", line 175, in check_url
return resp.get("local_url")
AttributeError: 'NoneType' object has no attribute 'get'

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cgtobi]

@Trouble-51 Could you please give some details about your setup? No camera usually indicates that you used HA cloud link with Netatmo which currently does not support cameras.

[Trouble-51]

I solved the problem yesterday by deleting the files in custom components, deleting the integration, restart HA, copying the HACS test files to custom components, restarting HA and run the integration again. After the cameras were back!

From: cgtobi [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Friday, 7 August 2020 at 15:59
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Mention [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

@Trouble-51 Could you please give some details about your setup? No camera usually indicates that you used HA cloud link with Netatmo which currently does not support cameras.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cgtobi]

@MrUserY That happens when the connection to the camera is lost. Did this happen with an up to date version of the custom component?

[Trouble-51]

No it happened when I did an update the latest version of HA

From: cgtobi [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Friday, 7 August 2020 at 16:04
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Mention [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

@MrUserY That happens when the connection to the camera is lost. Did this happen with an up to date version of the custom component?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[MrUserY]

Yes, with the version available at that moment I made the call on github. It could be the WIFI was lost for a particular short moment. My network is really great but the Netatmo Présence is a disaster in WIFI reception. I’ve littery put an extra accesspoint against the outside wall for one of the camera’s. The other camera is within limits, the reception is good enough. But I can’t guarantee when a truck with bad sparkplugs runs by the RF spectrum is interference free for a moment. So if you say that is the error... I can agree. (Maybe I have to lower the log level then, so it’s not Complaining anymore).

[cgtobi]

Just saying that is the cause. It should be handled more gracefully. I've put that on my list.

[cgtobi]

@MrUserY Please update the custom component. I committed an attempt to fix that camera issue.

[Trouble-51]

I get the following error:

Logger: homeassistant.setup
Source: setup.py:280
First occurred: 10:52:14 (1 occurrences)
Last logged: 10:52:14

Unable to prepare setup for platform netatmo.camera: Platform not found (No module named 'custom_components.netatmo.auth')

From: cgtobi [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Saturday, 8 August 2020 at 10:11
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Mention [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

@MrUserY Please update the custom component. I committed an attempt to fix that camera issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cgtobi]

@Trouble-51 could you please fix the formatting of you post. It is incredibly hard to read.

[Trouble-51]

Here you go, sorry about that. I copied the new file camera.py, deleted the integration restarted and installed the Netatmo integration was redirected to the website. Restarted HA again and this appeared in the log

Log Details (ERROR)

Logger: homeassistant.setup
Source: setup.py:280
First occurred: 10:52:14 (1 occurrences)
Last logged: 10:52:14

Unable to prepare setup for platform netatmo.camera: Platform not found (No module named 'custom_components.netatmo.auth').

From: cgtobi [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Saturday, 8 August 2020 at 12:29
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Mention [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

@Trouble-51 could you please fix the formatting of you post. It is incredibly hard to read.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cgtobi]

Did you install this via HACS? There is no need to copy individual files.

[Trouble-51]

Now installed via HACS and it’s working again. FYI, this is the debug information.

2020-08-08 13:47:12 DEBUG (SyncWorker_0) [homeassistant.components.netatmo.climate] Invalid home None

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] weather station detected!

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Study 70:ee:50:03:c7:14

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Rain gauge 05:00:00:00:9b:82

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Gym 03:00:00:03:29:84

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Wind Gauge 06:00:00:01:2d:5a

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Living Room 03:00:00:04:b2:70

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Outdoor 02:00:00:27:76:a0

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] Adding module Master Bedroom 03:00:00:08:11:b2

2020-08-08 13:47:12 DEBUG (SyncWorker_5) [homeassistant.components.netatmo.sensor] No home coach entities found

2020-08-08 13:47:13 DEBUG (SyncWorker_6) [homeassistant.components.netatmo.camera] Setting up camera 70:ee:50:2d:25:96 Garage

2020-08-08 13:47:13 DEBUG (SyncWorker_6) [homeassistant.components.netatmo.camera] Setting up camera 70:ee:50:28:2b:36 Back Yard

From: cgtobi [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Saturday, 8 August 2020 at 13:16
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Mention [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

Did you install this via HACS? There is no need to copy individual files.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Trouble-51]

A little later and now got this and cameras no longer showing anything:

2020-08-08 13:50:01 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_back_yard fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 465, in async_device_update await self.hass.async_add_executor_job( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 216, in update self._vpnurl, self._localurl = self._data.camera_data.camera_urls( File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 284, in camera_urls temp_local_url = check_url(vpn_url) File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 277, in check_url resp = self.authData.post_request(url=f"{url}/command/ping") File "/usr/local/lib/python3.8/site-packages/pyatmo/auth.py", line 157, in post_request f"{resp.status_code} - " File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[cgtobi]

2020-08-08 13:50:01 ERROR (MainThread) [homeassistant.helpers.entity] Update for camera.netatmo_back_yard fails Traceback (most recent call last): 
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() 
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 465, in async_device_update await self.hass.async_add_executor_job( 
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) 
File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 216, in update self._vpnurl, self._localurl = self._data.camera_data.camera_urls( 
File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 284, in camera_urls temp_local_url = check_url(vpn_url) 
File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 277, in check_url resp = self.authData.post_request(url=f"{url}/command/ping") 
File "/usr/local/lib/python3.8/site-packages/pyatmo/auth.py", line 157, in post_request f"{resp.status_code} - " 
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) 
File "/usr/local/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads return _default_decoder.decode(s) 
File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) 
File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) 
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[Trouble-51]

And the latest log errors for both cameras:

Log Details (ERROR)

Logger: homeassistant.helpers.entity
Source: components/netatmo/camera.py:216
First occurred: 13:50:01 (39 occurrences)
Last logged: 16:37:31
Update for camera.netatmo_back_yard fails
Update for camera.netatmo_garage fails
Traceback (most recent call last):

  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state

    await self.async_device_update()

  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 465, in async_device_update

    await self.hass.async_add_executor_job(

  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run

    result = self.fn(self.args, *self.kwargs)

  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 216, in update

    self._vpnurl, self._localurl = self._data.camera_data.camera_urls(

  File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 284, in camera_urls

    temp_local_url = check_url(vpn_url)

  File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 277, in check_url

    resp = self.authData.post_request(url=f"{url}/command/ping")

  File "/usr/local/lib/python3.8/site-packages/pyatmo/auth.py", line 157, in post_request

   f"{resp.status_code} - "

  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json

    return complexjson.loads(self.text, **kwargs)

  File "/usr/local/lib/python3.8/site-packages/simplejson/__init__.py", line 525, in loads

   return _default_decoder.decode(s)

  File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 370, in decode

    obj, end = self.raw_decode(s)

  File "/usr/local/lib/python3.8/site-packages/simplejson/decoder.py", line 400, in raw_decode

    return self.scan_once(s, idx=_w(s, idx).end())

simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Log Details (ERROR)

Logger: homeassistant.helpers.entity
Source: components/netatmo/camera.py:216
First occurred: 13:53:45 (52 occurrences)
Last logged: 16:35:46
Update for camera.netatmo_back_yard fails
Update for camera.netatmo_garage fails
Traceback (most recent call last):

  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state

    await self.async_device_update()

  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 465, in async_device_update

    await self.hass.async_add_executor_job(

  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run

    result = self.fn(self.args, *self.kwargs)

  File "/usr/src/homeassistant/homeassistant/components/netatmo/camera.py", line 216, in update

    self._vpnurl, self._localurl = self._data.camera_data.camera_urls(

  File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 284, in camera_urls

    temp_local_url = check_url(vpn_url)

  File "/usr/local/lib/python3.8/site-packages/pyatmo/camera.py", line 282, in check_url

    return resp.get("local_url")

AttributeError: 'NoneType' object has no attribute 'get'

From: cgtobi [e-mail]
Reply to: home-assistant/core [e-mail]
Date: Saturday, 8 August 2020 at 13:16
To: home-assistant/core [e-mail]
Cc: Trouble-51 [e-mail], Mention [e-mail]
Subject: Re: [home-assistant/core] Netatmo Error handling request AND Update for camera.netatmo_presence_fails AND Too many retries AND taking over 10 seconds (#35450)

Did you install this via HACS? There is no need to copy individual files.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[cgtobi]

Thanks for all the logs, but would you mind posting them via the web interface? The email reply is really killing the formatting.

[cgtobi]

Just for completeness, the cause for the absence of @Trouble-51 cameras was that he was using HA cloud link, which unfortunately does not yet support cameras yet due to a limitation by Netatmo. Using Netatmo dev key/secret in configuration.yaml fixed the issue.

[MrUserY]

Seems to be fine. Time for a official merge?

[cgtobi]

This has been merged for 0.114. For some reason the notes did not end up in the release notes.

[MrUserY]

Oh oke. Then i'll remove the hacs thing an try it out. Thanks for your efforts. I can close this case.