Cryptography: latest version of cryptography removed Cryptography_HAS_TLSEXT_HOSTNAME

[bundgaard]

Accidently upgraded to latest version, as it was not specified, left us in a puzzlement as to why the HAS_TLSEXT_HOSTNAME was removed.

Downgrade solved the problem. Currently using version 3.0

exceptions.AttributeError: 'module' object has no attribute 'Cryptography_HAS_TLSEXT_HOSTNAME'
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  <redacted files>
    import requests
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/requests/__init__.py", line 52, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 54, in <module>
    import OpenSSL.SSL
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/opt/buildagent/work/<redacted>/env/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 431, in <module>
    _lib.Cryptography_HAS_TLSEXT_HOSTNAME, "SNI not available"
AttributeError: 'module' object has no attribute 'Cryptography_HAS_TLSEXT_HOSTNAME'

[Katafalkas]

👍

[tgdfool2]

Same issue noticed here.

[RSulzmann]

see similar errors in the latest cryptography version 3.3:
module 'lib' has no attribute 'Cryptography_HAS_TLSEXT_HOSTNAME'

cryptography version 3.2.1 is working fine.

[reaperhulk]

Upgrading your pyOpenSSL will resolve this issue.

[omermizr]

Hey @reaperhulk, I actually think this should be reopened and addressed with a patch (as version 3.3 implies no breaking changes compared to 3.2).

[tiran]

Just install latest PyOpenSSL 20.0.0.

This is an unfortunate limitation of Python's packaging format. It cannot declare incompatibility without adding a hard dependency. Otherwise cryptography could declare that it is not compatible with PyOpenSSL < 20.0.0.

[reaperhulk]

pyOpenSSL 19.1 or better will fix this. Cryptography does not use semantic versioning (for a variety of reasons), so you should check the changelog for incompatibility information. This wasn’t caught because the dependency that expects it has been updated for more than a year. We're not currently planning to release a patch for this since upgrading the dependency will resolve the issue.

[omermizr]

Ok, I see. I didn't know you don't use SemVer.
Thank you @reaperhulk and @tiran for your quick responses!

[sabretus]

Actually pip3 is broken after this is installed, how should I install pyOpenSSL?

$ pip3 list
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 33, in vendored
    __import__(vendored_name, globals(), locals(), level=0)
ModuleNotFoundError: No module named 'pip._vendor.cachecontrol'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
  File "/usr/lib/python3/dist-packages/pip/__init__.py", line 22, in <module>
    from pip._vendor.requests.packages.urllib3.exceptions import DependencyWarning
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 64, in <module>
    vendored("cachecontrol")
  File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 36, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/__init__.py", line 84, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module>
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 675, in <module>
    _lib.Cryptography_HAS_TLSEXT_HOSTNAME, "SNI not available"
AttributeError: module 'lib' has no attribute 'Cryptography_HAS_TLSEXT_HOSTNAME'

[reaperhulk]

@sabretus You can try directly installing the wheel with pip (go download it from pypi.org), but that's likely to still fail. You can delete pyOpenSSL directly (via rm) and that should allow pip to install packages again.

As an aside, urllib3 hasn't auto-injected pyOpenSSL in several years so you're mixing significantly older packages with brand new ones. Your paths also suggest you're mixing distribution packages with packages supplied from pip, which will almost always lead to tears.

[rhuddleston]

Looks like it's still here in the latest urllib3: https://github.com/urllib3/urllib3/blob/master/src/urllib3/contrib/pyopenssl.py

[reaperhulk]

@rhuddleston it is present but inject_into_urllib3 is not automatically called by requests as of a release in ~May 2020 I believe. So my dates were wrong (I am working off memory of advocating for this to not occur except on very old Python years ago and didn't recall when it had actually landed). However latest pip vendors a requests that does not do this. An unfortunate confluence of bugs for those affected...

I think we're going to go ahead and do a 3.3.1 since this breakage can burn people so badly. You should still upgrade to later versions, but we'll do a release later tonight to resolve this.

[sabretus]

This problem started after I installed azure-identity and azure-storage-blob on clean Ubuntu 18.04.4

Here is the full dependency log

Collecting azure-identity
  Downloading https://files.pythonhosted.org/packages/2a/35/64e29615e7709c10c4f1d4310a8c13a6770142e9fcb9358fb8fa4d9b1578/azure_identity-1.5.0-py2.py3-none-any.whl (103kB)
    100% |████████████████████████████████| 112kB 6.7MB/s
Collecting azure-storage-blob
  Downloading https://files.pythonhosted.org/packages/e7/5d/0bb4ed37da2523c393789b1d8ecbf56b1d35fa344af30fe423da2c06cbe9/azure_storage_blob-12.6.0-py2.py3-none-any.whl (328kB)
    100% |████████████████████████████████| 337kB 3.8MB/s
Collecting six>=1.6 (from azure-identity)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting msal<2.0.0,>=1.6.0 (from azure-identity)
  Downloading https://files.pythonhosted.org/packages/14/66/8f9e84f1f5545381e97ba104805f58521f2354f33625b33002910f9e3032/msal-1.7.0-py2.py3-none-any.whl (59kB)
    100% |████████████████████████████████| 61kB 10.5MB/s
Collecting msal-extensions~=0.3.0 (from azure-identity)
  Downloading https://files.pythonhosted.org/packages/49/cb/c833ffa0f97c3098b146ac19bb2266c2d84b2119ffff83fdf001bb59d3ae/msal_extensions-0.3.0-py2.py3-none-any.whl
Collecting cryptography>=2.1.4 (from azure-identity)
  Downloading https://files.pythonhosted.org/packages/33/a3/f2b1d0e004055d897d687365130e5baf04e8d364b4b852716ef34e934ea3/cryptography-3.3-cp36-abi3-manylinux1_x86_64.whl (2.7MB)
    100% |████████████████████████████████| 2.7MB 609kB/s
Collecting azure-core<2.0.0,>=1.0.0 (from azure-identity)
  Downloading https://files.pythonhosted.org/packages/f8/4b/ea7faaafac956a168ab9a95a7ebe583f9d308e8332a68af0ed3128ef520c/azure_core-1.9.0-py2.py3-none-any.whl (124kB)
    100% |████████████████████████████████| 133kB 12.5MB/s
Collecting msrest>=0.6.10 (from azure-storage-blob)
  Downloading https://files.pythonhosted.org/packages/fa/f5/9e315fe8cb985b0ce052b34bcb767883dc739f46fadb62f05a7e6d6eedbe/msrest-0.6.19-py2.py3-none-any.whl (84kB)
    100% |████████████████████████████████| 92kB 15.1MB/s
Collecting PyJWT[crypto]<2,>=1.0.0 (from msal<2.0.0,>=1.6.0->azure-identity)
  Downloading https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl
Collecting requests<3,>=2.0.0 (from msal<2.0.0,>=1.6.0->azure-identity)
  Using cached https://files.pythonhosted.org/packages/39/fc/f91eac5a39a65f75a7adb58eac7fa78871ea9872283fb9c44e6545998134/requests-2.25.0-py2.py3-none-any.whl
Collecting portalocker~=1.0; platform_system != "Windows" (from msal-extensions~=0.3.0->azure-identity)
  Downloading https://files.pythonhosted.org/packages/3b/e7/ceef002a300a98a208232fab593183249b6964b306ee7dabb29908419cca/portalocker-1.7.1-py2.py3-none-any.whl
Collecting cffi>=1.12 (from cryptography>=2.1.4->azure-identity)
  Downloading https://files.pythonhosted.org/packages/1c/1a/90fa7e7ee05d91d0339ef264bd8c008f57292aba4a91ec512a0bbb379d1d/cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401kB)
    100% |████████████████████████████████| 409kB 3.9MB/s
Collecting certifi>=2017.4.17 (from msrest>=0.6.10->azure-storage-blob)
  Using cached https://files.pythonhosted.org/packages/5e/a0/5f06e1e1d463903cf0c0eebeb751791119ed7a4b3737fdc9a77f1cdfb51f/certifi-2020.12.5-py2.py3-none-any.whl
Collecting requests-oauthlib>=0.5.0 (from msrest>=0.6.10->azure-storage-blob)
  Downloading https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl
Collecting isodate>=0.6.0 (from msrest>=0.6.10->azure-storage-blob)
  Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)
    100% |████████████████████████████████| 51kB 15.5MB/s
Collecting chardet<4,>=3.0.2 (from requests<3,>=2.0.0->msal<2.0.0,>=1.6.0->azure-identity)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests<3,>=2.0.0->msal<2.0.0,>=1.6.0->azure-identity)
  Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl
Collecting urllib3<1.27,>=1.21.1 (from requests<3,>=2.0.0->msal<2.0.0,>=1.6.0->azure-identity)
  Using cached https://files.pythonhosted.org/packages/f5/71/45d36a8df68f3ebb098d6861b2c017f3d094538c0fb98fa61d4dc43e69b9/urllib3-1.26.2-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.12->cryptography>=2.1.4->azure-identity)
  Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
    100% |████████████████████████████████| 112kB 13.3MB/s
Collecting oauthlib>=3.0.0 (from requests-oauthlib>=0.5.0->msrest>=0.6.10->azure-storage-blob)
  Downloading https://files.pythonhosted.org/packages/05/57/ce2e7a8fa7c0afb54a0581b14a65b56e62b5759dbc98e80627142b8a3704/oauthlib-3.1.0-py2.py3-none-any.whl (147kB)
    100% |████████████████████████████████| 153kB 10.3MB/s

so it installed urllib3-1.26.2 which is the latest one?