Core: All ambient weather sensors are offline

[geiseri]

The problem

Ambient weather sensors are all offline event though there are no websocket errors.

Environment

arch | x86_64
-- | --
chassis | embedded
dev | false
docker | true
docker_version | 19.03.8
hassio | true
host_os | HassOS 4.11
installation_type | Home Assistant OS
os_name | Linux
os_version | 5.4.50
python_version | 3.8.3
supervisor | 229
timezone | America/New_York
version | 0.113.3
virtualenv | false

• Integration causing this issue: Ambient weather station
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/ambient_station

Problem-relevant configuration.yaml

Configured via UI, so no clue.

Traceback/Error logs

This is the only thing associated with their webservice i see:

2020-08-03 08:58:22 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection.139970222331504] Connected
2020-08-03 08:58:22 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection.139970222331504] Sending {'type': 'auth_required', 'ha_version': '0.113.3'}
2020-08-03 08:58:22 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection.139970222331504] Received {'type': 'auth', 'access_token': 'XXXXXXXXX'}
2020-08-03 08:58:22 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection.139970222331504] Received access_token
2020-08-03 08:58:22 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection.139970222331504] Auth OK
2020-08-03 08:58:22 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection.139970222331504] Sending {'type': 'auth_ok', 'ha_version': '0.113.3'}

Then nothing.

Additional information

It seems to have stopped working around 11am EST on 08/02/2020

[probot-home-assistant[bot]]

ambient documentation
ambient source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @bachya, mind taking a look at this issue as its been labeled with an integration (ambient_station) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[probot-home-assistant[bot]]

ambient_station documentation
ambient_station source
_{^{(message by IssueLinks)}}

[psperry]

I have been having the same issue for the last few days. At first, I thought it was related to version 0.113.3, so rolled back which seemed to fix the issue. But I think it was not really the latest version because it started happening again and it comes and goes. For instance, it went offline this morning at 5:29. Nothing was being done at the time. No related error in the logs that I can see.

I am guessing that it is an issue with Ambient.

[mmartoccia]

I too am having an issue. Same version (0.113.3) and I've removed my current configuration and attempted to recreate the integration. When using the Application and API key on the integration's page where it asks for both to complete the integration, it says "Invalid API Key and/or Application Key".

I've also tried it with ambient_weather in the configuration.yaml, it sees it as a newly discovered device on reboot but still asks for both Applicaiton and API Key on the Integration page to configure. Same message, "Invalid API Key and/or Application Key".

I suspect it's with the parsing of the key details or the structure to verify those details at the endpoint have changed.

[brimo2k]

I will just echo the sentiment of others here: The integration stops updating sporadically (if at all) and have tried upgrading Home Assistant to a different version and still see the issue. Nothing specific in the logs to clue me in as to what might have changed.

[geiseri]

This uses websockets, did something change in that area recently? Or did Ambient change (break) their webservice?

[brimo2k]

I do see that they updated their API documents with changes to the 'real-time' event data:
https://github.com/ambient-weather/api-docs/commit/0fa7335aad36c193cf366bc9766c0c9bbb03d79a#diff-2625016b50d68d922257f74801cac29c

[bachya]

Hi everyone – I'm looking into it.

@mmartoccia Thanks for your info; I, too, an experiencing the Invalid API Key and/or Application Key on a fresh install. Will need to dig in further.

[bachya]

Quick follow-up: this was also reported in aioambient, the library powering the integration. About 22 hours ago, running aioambient by itself consistently produced data from the websocket (https://github.com/bachya/aioambient/issues/68#issuecomment-667715828). Now, running the same test script, I don't see much:

INFO:engineio.client:Attempting WebSocket connection to wss://dash2.ambientweather.net/socket.io/?api=1&applicationKey=REDACTED&transport=websocket&EIO=3
INFO:engineio.client:WebSocket connection accepted with {'sid': 'REDACTED', 'upgrades': [], 'pingInterval': 25000, 'pingTimeout': 60000}
INFO:socketio.client:Engine.IO connection established
INFO:root:Simulating some other task occurring...
INFO:engineio.client:Sending packet PING data None
INFO:engineio.client:Received packet PONG data None
INFO:root:Simulating some other task occurring...

When I test the REST API, I get this:

ERROR:root:There was an error: Error requesting data from https://dash2.ambientweather.net/v1/devices: 401, message='Unauthorized', url=URL('https://dash2.ambientweather.net/v1/devices?apiKey=REDACTRED&applicationKey=REDACTED')

So, something's clearly different on their side. I'll take a look at the link @brimo2k sent and see if something has changed.

[bachya]

Okay, thankfully, it looks like a small thing. A long time ago, the Ambient devs told me that I should use https://dash2.ambientweather.net instead of https://api.ambientweather.net (as it was deemed more stable). It now looks like https://dash2.ambientweather.net is no longer in service; using https://api.ambientweather.net seems to work fine.

I'll do some more testing.

[bachya]

Testing looks good. PR coming.

[chrismdann]

You guys are amazing. Thank you

[mmartoccia]

It appears someone kicked the server for the bad endpoint. I'm receiving data again with no update.

[psperry]

Yeah, but it was still coming and going as of yesterday -- and once it stops, the only way I could get it back was to restart. The fix is now in beta and seems to be working well so far.

[mmartoccia]

Awesome, cant wait. Appreciate all the work put into this. I'd love for a few more sensors to be included e.g. Lightning, P2.5, etc... May need to contribute to the code base.