Core: Netatmo, second base station (probably) causing error setting up platform

[freemann]

The problem

After adding a second base station to my netatmo account, the new station didn't show up automatically. After removing and adding the integration I get the below error.

Environment

• Home Assistant Core release with the issue:
  0.116.2
• Last working Home Assistant Core release (if known):
  0.116.2
• Operating environment (OS/Container/Supervised/Core):
  Core
• Integration causing this issue:
  Netatmo
• Link to integration documentation on our website:
  ...

Problem-relevant configuration.yaml

Didn't change the config of this integration :)

Traceback/Error logs

Logger: homeassistant.components.sensor
Source: components/netatmo/sensor.py:252
Integration: Sensor (documentation, issues)
First occurred: 11:34:25 (2 occurrences)
Last logged: 14:26:09

Error while setting up netatmo platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 193, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 159, in async_setup_entry
    async_add_entities(await find_entities(data_class_name), True)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 150, in find_entities
    NetatmoSensor(data_handler, data_class_name, module, condition)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 252, in __init__
    f"{MANUFACTURER} {self._device_name} {SENSOR_TYPES[sensor_type][0]}"
KeyError: 'temp_trend'

Additional information

[springstan]

Please fill out the issue template, you can find it by clicking on New Issue and then selecting Report a bug with Home Assistant. Copy and paste the provided template into this already created issue, click preview to take a look at the empty template and then fill out as much as you can. Any additional information can be helpful!

Thanks for understanding :)

[peytoncasper]

@freemann Is the second base station that you added a different model than the first one?

[probot-home-assistant[bot]]

netatmo documentation
netatmo source
_{^{(message by IssueLinks)}}

[freemann]

No, just another weather base station.

[probot-home-assistant[bot]]

Hey there @cgtobi, mind taking a look at this issue as its been labeled with an integration (netatmo) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[peytoncasper]

I think this is related to the update to pyatmo by @cgtobi on October 6th [1] that pulls additional data from the base station like temp_trend and pressure_trend. I think the fix is rather straight forward and we just need to update the current map of SENSOR_TYPES to include this new data. [2] That being said, I would love to put together a PR, but I, unfortunately, don't own a Netatmo to test whether these are the only breaking changes.

EDIT: pyatmo was bumped to 4.1.0 in Home Assistant 0.116.1, so its possible that rolling back to 0.116.0 might resolve the issue in the meantime. [3]

[1] https://github.com/jabesq/netatmo-api-python/commit/28b1c824dfb441ce3527cf9909243302f786b04a#diff-18ae2186e963b59400a982d4a46cf9fb

[2] https://github.com/peytoncasper/core/blob/0.116.2/homeassistant/components/netatmo/sensor.py#L50

[3] https://github.com/home-assistant/core/releases/tag/0.116.1

[freemann]

Ok, so downgrading to 0.116.0 should fix it?

Just downgraded to 0.116.0 and now getting the following error:

```Logger: homeassistant.components.sensor
Source: components/netatmo/sensor.py:120
Integration: Sensor (documentation, issues)
First occurred: 21:24:39 (1 occurrences)
Last logged: 21:24:39

Error while setting up netatmo platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 193, in _async_setup_platform
await asyncio.shield(task)
File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 159, in async_setup_entry
async_add_entities(await find_entities(data_class_name), True)
File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 120, in find_entities
for module_id in data_class.get_modules(station_id):
File "/usr/local/lib/python3.8/site-packages/pyatmo/weather_station.py", line 88, in get_modules
"station_name": station["station_name"],
KeyError: 'station_name'
```

So probably 0.116.1 is creating some issues, but also add my second weather base station to my account.

[peytoncasper]

I'll just reiterate, I don't own one much less two so purely guessing based off the code. However, my question would be whether or not you've finished setting up the second station and gave it a name during the process? For some reason, Home Assistant can't find the name and that might be due to a lack of name or Netatmo might do some sort of pairing process in which a secondary base station doesn't get a name.

Hopefully, that makes sense, regardless let's see what cgtobi has to say as I'm sure he has a much better understanding of their API.

[freemann]

Both stations have a name:

I will wait for @cgtobi 😃

[cgtobi]

@freemann can we hook up via Discord or the forums to exchange debug information which might potentially contain sensible information?

[freemann]

That's possible and I prefer the forum.
I'm there with the same account as on GitHub.

[cgtobi]

A fix is in the making.

[jsimon3]

@cgtobi, not sure if it helps but I downloaded the test script (gen_test_data.py) you mentioned in a parallel thread on what seems to be the same or very similar issue. I can confirm that the script returns the same error as the HA integration:

Checking for available weather stations...
Traceback (most recent call last):
  File "gen_test_data.py", line 124, in <module>
    (i, s["station_name"]) for i, s in weather_data.stations.items()
  File "gen_test_data.py", line 124, in <listcomp>
    (i, s["station_name"]) for i, s in weather_data.stations.items()
KeyError: 'station_name'

To rule out issues on Netatmo API side, I downloaded their Swagger file and using the same authentication method as the integration, I tested the getstationsdata method - which was successful. Let me know if the response I get with Swagger would help you with debugging.

Since I dont know Python well enough to be of much help, I bought you a couple of coffees ;-)

[cgtobi]

Yes, I am aware of this. A fix is already merged upstream and will land as soon as a new release of pyatmo is available.

Thanks for the coffee, I'll enjoy every sip while coding for HA/pyatmo/etc.
Greatly appreciated.

[cgtobi]

If your comfortable modifying your system by hand for the time being I can provide an unofficial pre-release wheel to be installed via pip as a workaround. Let me know if this of interest.

[jsimon3]

Sure, I would give it a shot if it can easily be removed once the official release comes out.

[freemann]

Here also interested in the unofficial release 😃

[cgtobi]

Yeah, it would simply be overwritten.

https://github.com/cgtobi/netatmo-api-python/releases/download/v4.1.0b1/pyatmo-4.1.0-py3-none-any.whl
Use this at your own risk. Just pip install this and restart HA.

[jsimon3]

I installed the pyatmo:

config $ pip install pyatmo-4.1.0-py3-none-any.whl 
Processing ./pyatmo-4.1.0-py3-none-any.whl
Collecting requests-oauthlib
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: requests in /usr/lib/python3.8/site-packages (from pyatmo==4.1.0) (2.23.0)
Collecting oauthlib
  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 3.2 MB/s 
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (1.25.9)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (2020.4.5.1)
Installing collected packages: oauthlib, requests-oauthlib, pyatmo
Successfully installed oauthlib-3.1.0 pyatmo-4.1.0 requests-oauthlib-1.3.0

Then restarted HA, installed the Netatmo integration .- but unfortunately I still get the same error message - I tried both yaml and Cloud method.

2020-10-13 23:18:08 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up netatmo platform for sensor                          
Traceback (most recent call last):                                                                                                                   
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 193, in _async_setup_platform                                         
    await asyncio.shield(task)                                                                                                     
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 160, in async_setup_entry                         
    async_add_entities(await find_entities(data_class_name), True)                                                                 
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 151, in find_entities                             
    NetatmoSensor(data_handler, data_class_name, module, condition)                                                                
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 248, in __init__                                  
    self._device_name = module_info["station_name"]                                                                                
KeyError: 'station_name'   

[cgtobi]

Can you please test with the self test script you already downloaded? Just pip install the wheel there and try the script again.

[jsimon3]

Same result:

(venv) config $ pip install pyatmo-4.1.0-py3-none-any.whl
Requirement already satisfied: pyatmo==4.1.0 from file:///config/pyatmo-4.1.0-py3-none-any.whl in ./venv/lib/python3.8/site-packages (4.1.0)
Requirement already satisfied: requests in ./venv/lib/python3.8/site-packages (from pyatmo==4.1.0) (2.24.0)
Requirement already satisfied: oauthlib in ./venv/lib/python3.8/site-packages (from pyatmo==4.1.0) (3.1.0)
Requirement already satisfied: requests-oauthlib in ./venv/lib/python3.8/site-packages (from pyatmo==4.1.0) (1.3.0)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (2020.6.20)
Requirement already satisfied: chardet<4,>=3.0.2 in ./venv/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./venv/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (1.25.10)
Requirement already satisfied: idna<3,>=2.5 in ./venv/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (2.10)
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/config/venv/bin/python3 -m pip install --upgrade pip' command.

python gen_test_data.py --CLIENT_ID xxx --CLIENT_SECRET xxx --USERNAME 'xxx' --PASSWORD 'xxx' --mode long
Energy Public Camera WeatherStation HomeCoach Smokedetector

Checking for available weather stations...
Traceback (most recent call last):
  File "gen_test_data.py", line 123, in <module>
    stations = [
  File "gen_test_data.py", line 124, in <listcomp>
    (i, s["station_name"]) for i, s in weather_data.stations.items()
KeyError: 'station_name'

[cgtobi]

Ok, that is interesting. Are you on Discord to exchange some (potentially sensible) data from your system?

[jsimon3]

I am now

[cgtobi]

Great. Just ping me on Discord and I get back to you asap.

[freemann]

Here the same error:

Energy Public Camera WeatherStation HomeCoach Smokedetector
Traceback (most recent call last):
  File "gen_test_data.py", line 123, in <module>
    stations = [
  File "gen_test_data.py", line 124, in <listcomp>
    (i, s["station_name"]) for i, s in weather_data.stations.items()
KeyError: 'station_name'

[freemann]

@cgtobi
I see some contributions on your account, is there something we can test?

[cgtobi]

That has nothing to do with this issue but I talked to Netatmo today and now I know where those issues come from at least.

[cgtobi]

You can try the fix:

ssh <your_instance>
docker exec -it $(docker ps -f name=homeassistant -q) bash
pip install --upgrade https://github.com/cgtobi/netatmo-api-python/releases/download/v4.1.0b1/pyatmo-4.1.0-py3-none-any.whl

[freemann]

With portainer installed the new pyatmo.

With this result;
```Logger: homeassistant.components.sensor
Source: components/netatmo/sensor.py:120
Integration: Sensor (documentation, issues)
First occurred: 18:44:52 (2 occurrences)
Last logged: 18:46:11

Error while setting up netatmo platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 193, in _async_setup_platform
await asyncio.shield(task)
File /usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 159, in async_setup_entry
async_add_entities(await find_entities(data_class_name), True)
File
/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 120, in find_entities
for module_id in data_class.get_modules(station_id):
File "/usr/local/lib/python3.8/site-packages/pyatmo/weather_station.py", line 88, in get_modules
"station_name": station["station_name"],
KeyError: 'station_name```

[cgtobi]

Then the fix isn't installed. Line 88 contains return {}.

[freemann]

4.1.0 uninstalled, 4.1.0 installed.

[cgtobi]

Are you sure you're in the right container/environment?

[freemann]

Is a restart of the container necessary?

[jsimon3]

I can confirm that the fix worked for me - thanks to help from cgtobi. Yes, a restart of HA was necessary. Dont know if a container restart would have done it as well

[freemann]

Tried from putty to reinstall the package with the following result:

Also opened the weather_station.py inside the container:

The cursor left in the middle is on line 88.
Is this the new or old version of the file?

This the command I used and the location of pyatmo I got of the error message:

sudo docker exec -it home_assistant vi /usr/local/lib/python3.8/site-packages/pyatmo/weath
er_station.py

[cgtobi]

Of course you need to restart HA.

[freemann]

Ok, been there done that.
Will wait for the official release and hoping it fixes the issue.

[cgtobi]

@freeman please show me line 51 of that file. It should contain if "station_name" not in item:.

[freemann]

Ok can't get around this;

First I uninstall pyatmo 4.0.0;

xxx@xxx:~$ sudo docker exec -it home_assistant pip uninstall pyatmo
Found existing installation: pyatmo 4.0.0
Uninstalling pyatmo-4.0.0:
  Would remove:
    /usr/local/lib/python3.8/site-packages/pyatmo-4.0.0.dist-info/*
    /usr/local/lib/python3.8/site-packages/pyatmo/*
Proceed (y/n)? y
  Successfully uninstalled pyatmo-4.0.0

Then I install 4.1.0 beta;

xxx@xxx:~$ sudo docker exec -it home_assistant pip install https://github.com/cgtobi/netatmo-api-python/releases/download/v4.1.0b1/pyatmo-4.1.0-py3-none-any.whl
Collecting pyatmo==4.1.0
  Downloading https://github.com/cgtobi/netatmo-api-python/releases/download/v4.1.0b1/pyatmo-4.1.0-py3-none-any.whl (19 kB)
Requirement already satisfied: requests in /usr/local/lib/python3.8/site-packages (from pyatmo==4.1.0) (2.24.0)
Requirement already satisfied: oauthlib in /usr/local/lib/python3.8/site-packages (from pyatmo==4.1.0) (3.1.0)
Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.8/site-packages (from pyatmo==4.1.0) (1.3.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (2020.6.20)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/site-packages (from requests->pyatmo==4.1.0) (1.25.10)
Installing collected packages: pyatmo
Successfully installed pyatmo-4.1.0

Then I check with the uninstall command if 4.1.0 is installed and it is;

xxx@xxx:~$ sudo docker exec -it home_assistant pip uninstall pyatmo
Found existing installation: pyatmo 4.1.0
Uninstalling pyatmo-4.1.0:
  Would remove:
    /usr/local/lib/python3.8/site-packages/pyatmo-4.1.0.dist-info/*
    /usr/local/lib/python3.8/site-packages/pyatmo/*
Proceed (y/n)? n

Then I restart HA with the service command call; homeassistant.restart

After the restart I check the installed pyatmo version again and its back to 4.0.0;

xxx@xxx:~$ sudo docker exec -it home_assistant pip uninstall pyatmo
Found existing installation: pyatmo 4.0.0
Uninstalling pyatmo-4.0.0:
  Would remove:
    /usr/local/lib/python3.8/site-packages/pyatmo-4.0.0.dist-info/*
    /usr/local/lib/python3.8/site-packages/pyatmo/*
Proceed (y/n)? n

So the restart of HA is don't strange things in the Docker container.....

I have no idea what's happening here and/or what to do with it....

[cgtobi]

What HA version are you using? You have to be on 0.116.2.

[freemann]

Ah OK, that's 1 step further because it stays on 4.1.0 package.

Tried so many things that I lost the HA version out of side... now i'm back on 0.116.2 😊

But.... New error message in town;

Logboekdetails (ERROR)
Logger: homeassistant.components.sensor
Source: components/netatmo/sensor.py:248
Integration: Sensor (documentation, issues)
First occurred: 13:29:31 (2 occurrences)
Last logged: 13:30:40

Error while setting up netatmo platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 193, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 160, in async_setup_entry
    async_add_entities(await find_entities(data_class_name), True)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 151, in find_entities
    NetatmoSensor(data_handler, data_class_name, module, condition)
  File "/usr/src/homeassistant/homeassistant/components/netatmo/sensor.py", line 248, in __init__
    self._device_name = module_info["station_name"]
KeyError: 'station_name'

Most recent HA file version;
https://github.com/home-assistant/core/blob/dev/homeassistant/components/netatmo/sensor.py#L246

Back in the day it was;
https://github.com/home-assistant/core/blob/da2f9db62e401a82fd9153887a052fbe47c667fe/homeassistant/components/netatmo/sensor.py#L248

[cgtobi]

@freemann Now that you're on 0.116.2 did you run pip install --upgrade https://github.com/cgtobi/netatmo-api-python/releases/download/v4.1.0b1/pyatmo-4.1.0-py3-none-any.whl inside that container again?

[freemann]

Just updated to 0.116.3, checked pyatmo version and my docker shows 4.1.0, restarted the container, still 4.1.0 is there and sensors are reloading.
The lack of a stationname is a bit disappointing, do you have ideas how i could fix that on the NetAtmo side?

Thanks for all you time and effort!

[cgtobi]

Are you using their new app?

You gotta upgrade the pyatmo lib linked above. It won't show a different version though.

[freemann]

The lib version is on 4.1.0 and what I mean is shown below. Here you see the co2 sensors of both the 2 base stations, where one is "wk" and the other "kinderkamer" and I have no idea which is which(of course I known which is which)

The sub sensors have a name and could be identified, here you see "bk" what's the sensor name;

New app old app? Which app on what system/site?

Both base station have a new, see this screenshot in the android app;

[cgtobi]

Ok, I'll take a look.

I was referring to the iOS app from Netatmo.

[freemann]

I think I'm using the new Android NetAtmo app(v3.0.0.17).

[cgtobi]

@freemann please check my PM on the forum. Thx