Core: OpenWeatherMap: Update to 0.118.1 breaks integration

[f45tb00t]

The problem

After updating from v0.117.6 to 0.118.1 the openweathermap integration does not load anymore

Environment

HW: Raspberry Pi 3 B+
Installation: Manual (venv)
Affected Version: 0.118.1
Python Version: v3.8.6

• Home Assistant Core release with the issue: 0.118.1
• Last working Home Assistant Core release (if known): 0.117.6
• Operating environment (OS/Container/Supervised/Core): Core
• Integration causing this issue: openweathermap
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/openweathermap/

Problem-relevant configuration.yaml

No config set in configuration.yaml

Traceback/Error logs

2020-11-20 12:54:30 ERROR (MainThread) [homeassistant.setup] Setup failed for openweathermap: Unable to import component: No module named '_bz2'

Additional information

Rollback to v0.117.6 fixed the issue

[probot-home-assistant[bot]]

openweathermap documentation
openweathermap source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @fabaff, @freekode, @nzapponi, mind taking a look at this issue as its been labeled with an integration (openweathermap) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[f45tb00t]

@frenck
How can I change/add that label myself in the future? Just to reduce your workload. Or can this be set only by admins/moderators/contributors/...?

[frenck]

This can only set by collaborators of the GitHub org or repo.

[f45tb00t]

This can only set by collaborators of the GitHub org or repo.

okay. got ya

[nzapponi]

Do you guys have any idea what _bz2 refers to? I had no issues with the migration so far.

[f45tb00t]

Do you guys have any idea what _bz2 refers to? I had no issues with the migration so far.

It seems that "libbz2-dev" is needed when compiling python. But in fact I have not installed this before. In addition it worked with 0.117.6 before very well. No missing dependencies at all. So no idea what's causing the issue yet.
But well... I am not a coder at all. :)

[nzapponi]

Looks really weird, I don't see any references to bz2, neither in the openweathermap component, nor in the Home Assistant core repo at all.

Given that you've installed HA through venv, did you try installing libbz2-dev at all? Or, could you uninstall openweathermap and then upgrade to 0.118.1, and report here if the issue goes away?

Thanks!

[f45tb00t]

Given that you've installed HA through venv, did you try installing libbz2-dev at all? Or, could you uninstall openweathermap and then upgrade to 0.118.1, and report here if the issue goes away?

Hi!
Yes I did install it. Then tried again with openweathermap, but just fulfilling the depedencies did not make it run.
Okay, I will remove openweathermap. Upgrade to recent version and try to install it again.
Side Note: I found for a topic not related to HA, that if the above error occurs, you have to install libbz2-dev and recompile phyton. But I really would like to prevent this. Especially since it worked before like a charme without any issues.

I'll keep you updated.

[f45tb00t]

Update:

1. stay on 0.117.6 && remove openweathermap integration
2. upgrade to 0.118.2
3. Restart HA
4. Install integration openweathermap
   Result: 2020-11-21 17:35:38 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading configuration flow for integration openweathermap: No module named '_bz2'

So no, its not working at all. Ideas?

[marc-gist]

To solve I did this (i'm running python 3.8.6 venv):
> sudo apt install libbz2-dev
then under Python-3.8.6
> sudo ./configure --enable-optimizations
> sudo make -j 4; sudo make altinstall

Restarted HA and all worked fine (i did not have to create the venv again).

[nzapponi]

Sounds like the issue was related to your environment specifically... I'll close the issue for now, if anyone else runs into this please reopen and we can take a proper look at it!

[marc-gist]

Sounds like the issue was related to your environment specifically... I'll close the issue for now, if anyone else runs into this please reopen and we can take a proper look at it!

perhaps an update to docs for what is needed in the python env? not sure why the packages is even needed for a single integration.

[nzapponi]

A search for import bz2 in my virtual env developer environment shows that actually bz2 is required in a bunch of places, not just for openweathermap.

My gut feeling is that using the venv installation method gets you into all sorts of weird errors when upgrading and new dependencies come in.

We can probably update the venv documentation, but I'd recommend moving to Docker or hass.io if I were you.

I recently moved from venv to Docker and it's much more stable & predictable, and easier to upgrade!

[marc-gist]

We can probably update the venv documentation, but I'd recommend moving to Docker or hass.io if I were you.

I recently moved from venv to Docker and it's much more stable & predictable, and easier to upgrade!

You are probably right; but I've been doing it this way since like version 0.6 :D so hard to change. And really, I've had less issues upgrading because I can readily see the log output, quickly start and stop HA, add/remove packages like the color log output, etc. So many benefits to running core.

[marc-gist]

so i think at https://developers.home-assistant.io/docs/development_environment/
should update the line for

sudo apt-get install python3-pip python3-dev python3-venv autoconf libssl-dev libxml2-dev libxslt1-dev libjpeg-dev libffi-dev libudev-dev zlib1g-dev pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libavresample-dev libavfilter-dev ffmpeg

to include libbz2-dev

[f45tb00t]

Sounds like the issue was related to your environment specifically... I'll close the issue for now, if anyone else runs into this please reopen and we can take a proper look at it!

No, I don't think this is specifically related to my environment. I just followed the installation instructions. If it's a needed dependency, I propose to mention it in the RTFM.

Moving to docker is also not that good imho. The more docker containers you run, the more overhead you will have. Not sure about the overhead comparison with a venv and docker. But usually you will have an impact when running docker.
Also hass.io is not an option. Because as far as I can see this has very limited functionality in comparison to a raspbian light.
So in fact using the core installation has the most flexible environment in my point of view.

I just recompilled it. Now it's working again as expected.

configure --prefix=/usr/local/opt/python-3.8.6 --enable-loadable-sqlite-extensions --enable-optimizations
sudo make -j 4 && sudo make altinstall
done...

cheers

fastboot