Core: Incompatible PiWheels built wheels when on Debian Stretch or lower

[jurgenweber]

Home Assistant release with the issue:
0.96.0

Last working Home Assistant release (if known):
0.95.4, but rollback does not work

Operating environment (Hass.io/Docker/Windows/etc.):
hassbian, python 3.7

Component/platform:
cloud

Description of problem:

2019-07-18 21:28:31 ERROR (MainThread) [homeassistant.setup] Error during setup of component cloud
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/setup.py", line 153, in _async_setup_component
    hass, processed_config)
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/components/cloud/__init__.py", line 142, in async_setup
    from hass_nabucasa import Cloud
  File "/srv/homeassistant/lib/python3.7/site-packages/hass_nabucasa/__init__.py", line 16, in <module>
    from .remote import RemoteUI
  File "/srv/homeassistant/lib/python3.7/site-packages/hass_nabucasa/remote.py", line 16, in <module>
    from .acme import AcmeClientError, AcmeHandler
  File "/srv/homeassistant/lib/python3.7/site-packages/hass_nabucasa/acme.py", line 9, in <module>
    import OpenSSL
  File "/srv/homeassistant/lib/python3.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/srv/homeassistant/lib/python3.7/site-packages/OpenSSL/crypto.py", line 16, in <module>
    from OpenSSL._util import (
  File "/srv/homeassistant/lib/python3.7/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/srv/homeassistant/lib/python3.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: /usr/lib/arm-linux-gnueabihf/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /srv/homeassistant/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so)

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:
I upgraded to 0.96.0, and there is no cloud.
I down graded hassbian-config upgrade homeassistant=0.95.4 same problem.

not sure what is going on here. nothing with ssl works now.

also # cat /etc/debian_version 9.9

[cgtobi]

Did you install/update libssl-dev?

[reaper7]

update libssl-dev on rpi will not change anything
because the available version is still 1.1.0

pi@rpi:~ $ apt-cache policy libssl-dev
libssl-dev:
  Installed: 1.1.0k-1~deb9u1
  Candidate: 1.1.0k-1~deb9u1
  Version table:
 *** 1.1.0k-1~deb9u1 500
        500 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages
        100 /var/lib/dpkg/status

for me this issue broke also Xiaomi IR Remote component based on miio

[jurgenweber]

yeah, exactly. If you are on < 'buster' its broke.

[DubhAd]

FWIW, I upgraded my test box running Python 3.7.3, and Debian 9.8, to 0.96. That has OpenSSL 1.1.0 and 1.0.2. I've signed it up for a trial Cloud account, and it has no issues.

ii  libssl1.0.2:amd64                  1.0.2r-1~deb9u1                   amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64                    1.1.0j-1~deb9u1                   amd64        Secure Sockets Layer toolkit - shared libraries

I'll need to upgrade my test Pi to Python 3.7 before I can test that.

[DubhAd]

Ok, my Pi sees the same error.

ImportError: /usr/lib/arm-linux-gnueabihf/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /home/homeassistant/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so)

[Ernst79]

Seems to affect a number of components, all related to this OPENSSL issue.

asuswrt
https://community.home-assistant.io/t/0-96-notion-updated-sidebar-advanced-mode/126889/82

xiaomi_miio
https://community.home-assistant.io/t/0-96-notion-updated-sidebar-advanced-mode/126889/77

broadlink
https://community.home-assistant.io/t/0-96-notion-updated-sidebar-advanced-mode/126889/76

[Jc2k]

I have seen that error before when a virtualenv was created and then python3 was updated underneath it. Are you using virtualenv?

[reaper7]

I am using virtualenv with python 3.7.3

[jurgenweber]

I have seen that error before when a virtualenv was created and then python3 was updated underneath it. Are you using virtualenv?

I am not sure what you mean by this, yes. I created a new venv when I found out python3.5 was getting deprecated.

what python3 do you refer to? The system one or python inside the venv?

[Ernst79]

I'm using virtualenv with python 3.7.3. too. Upgraded python a few (HA) versions earlier, by the upgrade script (hassbian tools), but has worked fine till now. I thought this script also creates a new venv.

[Jc2k]

When you create a virtualenv it copies the system one into the virtualenv, but it still depends on the system one. So if the system one changes but the one in the virtualenv doesn't you can get weird errors like this. I have actually seen it complain about OPENSSL_1_1_1 because of this on Ubuntu, but not on rasbpian/debian before. Can someone confirm they created a fresh virtualenv after upgrading to python 3.7 and then haven't done an apt upgrade that updated python recently and yet still have this problem? Can they also confirm the steps to create a fresh virtualenv they are using.

[Jc2k]

If you are on amd64 and Debian/Ubuntu i'd expect it to use a wheel package which is precompiled with a suitable statically compiled version of OpenSSL actually in the wheel package. I'm not sure why that isnt happening - try 'pip install wheel; pip uninstall cryptography; pip install --no-cache cryptography'

On raspbian the same is normally true because of piwheels. It looks like there is a piwheels version of cryptography here -> https://www.piwheels.org/project/cryptography/. The last time i installed raspbian there was a /etc/pip.conf which contained:

[global]
extra-index-url=https://www.piwheels.org/simple

With that in place my pi stuff used the cryptography wheels and used the right openssl without having to install any openssl packages via apt.

[jurgenweber]

I used

hassbian-config upgrade python --force

to upgrade my venv.

but yeah, the point is python is very different:

In the venv:

(homeassistant) root@warvm-pi94 ~ # python --version
Python 3.7.3

The system:

pytroot@warvm-pi94 ~ # python --version
Python 2.7.13
root@warvm-pi94 ~ # python3 --version
Python 3.5.3

[Jc2k]

@jurgenweber what happens if you try this in your venv:

pip install --no-cache https://www.piwheels.org/simple/cryptography/cryptography-2.7-cp37-cp37m-linux_armv7l.whl

And then start HA?

[jurgenweber]

yeah, same thing.

(homeassistant) root@warvm-pi94 /srv/homeassistant # pip freeze | grep crypt
cryptography==2.7

ImportError: /usr/lib/arm-linux-gnueabihf/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /srv/homeassistant/lib/python3.7/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so)

[Jc2k]

Did it definitely uninstall and reinstall? Might need to do a pip uninstall cryptography first as version wasnt changed.

[jurgenweber]

yeah, it did

 # pip install --no-cache https://www.piwheels.org/simple/cryptography/cryptography-2.7-cp37-cp37m-linux_armv7l.whl
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting cryptography==2.7 from https://www.piwheels.org/simple/cryptography/cryptography-2.7-cp37-cp37m-linux_armv7l.whl
  Downloading https://www.piwheels.org/simple/cryptography/cryptography-2.7-cp37-cp37m-linux_armv7l.whl (842kB)
     |████████████████████████████████| 849kB 116kB/s
Requirement already satisfied: six>=1.4.1 in ./lib/python3.7/site-packages (from cryptography==2.7) (1.12.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in ./lib/python3.7/site-packages (from cryptography==2.7) (1.12.3)
Requirement already satisfied: asn1crypto>=0.21.0 in ./lib/python3.7/site-packages (from cryptography==2.7) (0.24.0)
Requirement already satisfied: pycparser in ./lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography==2.7) (2.19)
ERROR: homeassistant 0.95.4 has requirement cryptography==2.6.1, but you'll have cryptography 2.7 which is incompatible.
Installing collected packages: cryptography
  Found existing installation: cryptography 2.6.1
    Uninstalling cryptography-2.6.1:
      Successfully uninstalled cryptography-2.6.1
Successfully installed cryptography-2.7

[Jc2k]

And you are on buster?

[jurgenweber]

no. :) I did say that in my original message....

also # cat /etc/debian_version 9.9

[Jc2k]

Nice edit ;) I guess i haven't memorised them...

If you are on < buster and have your pip pointing at piwheels then pip will merrily install buster wheels that depend on buster versions of things, so you either have to upgrade to buster or turn off piwheels.org and reinstall cryptography.

I guess pip uninstall cryptography; pip install cryptography==2.6.1 --no-cache --no-binary :all: would do it. But you may will need to install truckloads of -dev packages.

If your pip is old enough it might not understand --no-binary and so there is also --no-use-wheel.

[jurgenweber]

yeah, I figured the best way forward would be just to upgrade to the latest hassbian which is buster.

a problem for tomorrow.

THanks, tis late here.

[DubhAd]

I have seen that error before when a virtualenv was created and then python3 was updated underneath it. Are you using virtualenv?

I built a fresh 3.7 venv for this test.

Can someone confirm they created a fresh virtualenv after upgrading to python 3.7 and then haven't done an apt upgrade that updated python recently and yet still have this problem? Can they also confirm the steps to create a fresh virtualenv they are using.

I followed https://www.home-assistant.io/docs/installation/raspberry-pi/, which says to install wheel.

[Jc2k]

@DubhAd What version of Debian/Raspbian do you have on the pi? AIUI if its not buster then it wont work because the wheels for python 3.7 from piwheels.org assume you are on buster. You need to upgrade to buster or install with wheels switched off.

[DubhAd]

9.8, so not Buster

[frenck]

The reason why this broke, PiWheels builds wheels against Debian Buster, which has a newer glibc, which now causes issues if you are still on Debian Stretch.

There are 2 possible option to get this fixed:

Option 1: _IMHO the best / future proof /preferred option_

Upgrade your distribution to Debian Buster. After upgrade, everything works as intended.

Pros: more future proof, Piwheels work.
Cons: Upgrades your whole system and might affect other things you might be running on your device.

Steps:

• Edit stretch to buster in the following files:
  
  
  
  • /etc/apt/sources.list
  
  
  • /etc/apt/sources.list.d/raspi.list
  
  
  • /etc/apt/sources.list.d/hassbian.list
  
  
• sudo apt-get update
• sudo apt-get -y dist-upgrade
• sudo reboot

Option 2:

Disable Piwheels prebuild packages.

Pros: Quick and easy.
Cons: You'll end up with a possible slow updating/upgrading/installation when new packages need to be installed, since it needs to compile them from source. You are still on Stretch, which is not really future proof.

Steps:

• Edit /etc/pip.conf and comment out the extra-index-url like this:

```ini
[global]
# extra-index-url=https://www.piwheels.org/simple

```

• source /srv/homeassistant/bin/activate
• pip uninstall cryptography
• pip install --no-cache cryptography==2.7

[frenck]

Closing up this issue, since there is nothing left to do. See the answer above for a solution.

[jurgenweber]

yeah, I upgraded to Debian Buster and life is good again. :) Thanks team!