Certbot: '_openssl' has no function, constant or global variable named 'Cryptography_HAS_RSA_OAEP_MD'

[mhsabbagh]

My operating system is (include version):

Ubuntu 16.04.

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

APT repository from the official website (ppa:certbot/certbot).

I ran this command and it produced this output:

root@myserver:/etc/php/7.0/fpm/pool.d# certbot --nginx
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.21.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/usr/lib/python3/dist-packages/josepy/__init__.py", line 41, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/usr/lib/python3/dist-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/usr/lib/python3/dist-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 16, in <module>
    from OpenSSL._util import (
  File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 156, in <module>
    Binding.init_static_locks()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 137, in init_static_locks
    cls._ensure_ffi_initialized()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 124, in _ensure_ffi_initialized
    cls.lib = build_conditional_library(lib, CONDITIONAL_NAMES)
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 84, in build_conditional_library
    if not getattr(lib, condition):
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_RSA_OAEP_MD'

Certbot's behavior differed from what I expected because:

It didn't work, crashed on start.

[joohoi]

This looks like version mismatch between some of our dependencies. Namely python libraries pyOpenSSL (python-openssl) which depends on cryptography (python-cryptography) which then depends on libssl package. I don't have a direct answer to this issue, but please check that all the packages have been upgraded from the Certbot PPA. Let me know about the status after checking, and we'll work from there.

[mhsabbagh]

Yep, everything is upgraded and up to date:

root@myserver:~# apt-cache policy $(dpkg --get-selections | grep -v deinstall$ | awk '{ print $1 }') | perl -e '@a = <>; $a=join("", @a); $a =~ s/\n(\S)/\n\n$1/g;  @packages = split("\n\n", $a); foreach $p (@packages) {print "$1: $2\n" if $p =~ /^(.*?):.*?500 http:\/\/ppa\.launchpad\.net\/(.*?)\s/s}'

certbot: certbot/certbot/ubuntu
python-certbot-nginx: certbot/certbot/ubuntu
python-pkg-resources: certbot/certbot/ubuntu
python-setuptools: certbot/certbot/ubuntu
python3-acme: certbot/certbot/ubuntu
python3-asn1crypto: certbot/certbot/ubuntu
python3-certbot: certbot/certbot/ubuntu
python3-certbot-nginx: certbot/certbot/ubuntu
python3-certifi: certbot/certbot/ubuntu
python3-cffi-backend: certbot/certbot/ubuntu
python3-chardet: certbot/certbot/ubuntu
python3-configargparse: certbot/certbot/ubuntu
python3-configobj: certbot/certbot/ubuntu
python3-cryptography: certbot/certbot/ubuntu
python3-future: certbot/certbot/ubuntu
python3-idna: certbot/certbot/ubuntu
python3-josepy: certbot/certbot/ubuntu
python3-ndg-httpsclient: certbot/certbot/ubuntu
python3-openssl: certbot/certbot/ubuntu
python3-parsedatetime: certbot/certbot/ubuntu
python3-pkg-resources: certbot/certbot/ubuntu
python3-pyasn1: certbot/certbot/ubuntu
python3-requests: certbot/certbot/ubuntu
python3-rfc3339: certbot/certbot/ubuntu
python3-six: certbot/certbot/ubuntu
python3-urllib3: certbot/certbot/ubuntu
python3-zope.component: certbot/certbot/ubuntu
python3-zope.hookable: certbot/certbot/ubuntu
python3-zope.interface: certbot/certbot/ubuntu

[joohoi]

I wonder if @hlieberman or @oerdnj would have ideas about what's the culprit behind this mismatch and/or how to fix it?

[hlieberman]

Hm. That's a version mismatch for sure; Cryptography_HAS_RSA_OAEP_MD was introduced in python-cryptography version 1.4, which is a couple of years old.

I'm wondering if you've got a duplicate lib somewhere in your PYTHONPATH.

Can you give me the output of python3 -dv /usr/bin/certbot --nginx?

[mhsabbagh]

It is: https://pastebin.com/raw/eGTdxLFS

(Couldn't put it on GitHub, too long)

[hlieberman]

Hm. That looks right.

Can I see dpkg -l python3-{cryptography,openssl,cffi,cffi-backend,certbot,certbot-nginx,acme} openssl and lsb_release -a

[mhsabbagh]

Sure:

root@mail:~# dpkg -l python3-{cryptography,openssl,cffi,cffi-backend,certbot,certbot-nginx,acme} openssl and lsb_release -a
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                 Version                 Architecture            Description
+++-====================================-=======================-=======================-=============================================================================
ii  openssl                              1.0.2g-1ubuntu4.10      amd64                   Secure Sockets Layer toolkit - cryptographic utility
ii  python3-acme                         0.21.1-1+ubuntu16.04.1+ all                     ACME protocol library for Python 3
ii  python3-certbot                      0.21.1-1+ubuntu16.04.1+ all                     main library for certbot
ii  python3-certbot-nginx                0.21.1-1+ubuntu16.04.1+ all                     Nginx plugin for Certbot
un  python3-cffi                         <none>                  <none>                  (no description available)
ii  python3-cffi-backend                 1.10.0-0.1+ubuntu16.04. amd64                   Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography                 1.9-1+ubuntu16.04.1+cer amd64                   Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-openssl                      17.3.0-1~0+ubuntu16.04. all                     Python 3 wrapper around the OpenSSL library
dpkg-query: no packages found matching and
dpkg-query: no packages found matching lsb_release
dpkg-query: no packages found matching -a

[creativetags]

I have a similar issue running Ubuntu 16.04.4 LTS

Did anyone find the solution?

Name | Version | Architecture | Description
---------------------------|-------------------|---------------------|------------------
ii openssl |1.1.0g-2.1+ubuntu16.04.1 |amd64 |Secure Sockets Layer toolkit - cryptographic utility
ii python3-acme |0.21.1-1+ubuntu16.04.1+c |all |ACME protocol library for Python 3
ii python3-certbot |0.21.1-1+ubuntu16.04.1+c |all |main library for certbot
ii python3-certbot-nginx |0.21.1-1+ubuntu16.04.1+c |all |Nginx plugin for Certbot
un python3-cffi | |(no description available)
ii python3-cffi-backend |1.10.0-0.1+ubuntu16.04.1 |amd64 |Foreign Function Interface for Python 3 calling C code - runtime
ii python3-cryptography |1.9-1+ubuntu16.04.1+cert |amd64 | Python library exposing cryptographic recipes and primitives (Python 3)
ii python3-openssl |17.3.0-1~0+ubuntu16.04.1 |all |Python 3 wrapper around the OpenSSL library

Error varies each time I run certbot renew --dry-run
'_openssl' has no function, constant or global variable named 'Cryptography_HAS_SCT'
or Cryptography_HAS_EVP_PKEY_DHX
or Cryptography_HAS_SCRYPT
or Cryptography_HAS_DTLS
or Cryptography_HAS_RSA_OAEP_MD
or Cryptography_HAS_SET_ECDH_AUTO
or Cryptography_HAS_SSL_ST
or Cryptography_HAS_DTLS
or Cryptography_HAS_X509_STORE_CTX_GET_ISSUER
or Cryptography_HAS_LOCKING_CALLBACKS

[oerdnj]

Is your system fully upgraded? e.g. apt-get update && apt-get dist-upgrade

[creativetags]

@oerdnj Yes, just checked

[mhsabbagh]

I have some domains' certificates expiring and sadly I couldn't renew them :(

No solution so far.

[creativetags]

@mhsabbagh I found a temporary workaround with a manual install of certbot-auto https://certbot.eff.org/docs/install.html#certbot-auto

Then use ./certbot-auto instead of certbot

[mhsabbagh]

@creativetags Thank you! Confirmed to be working here too.

[creativetags]

Still broken in 0.22.2

[xzhuz]

Have you fix it yet? The same problem occur to me now. Need help!

[Fyb3roptik]

This is still an issue. Cannot use Certbot :(

[Hakujou]

I also have this issue on a 16.04.5 fully updated system. Any update on that matter ?

[vriyeit]

Was there any solution found on this?

[joohoi]

Was there any solution found on this?

AFAIK the only reliable solution right now is to use certbot-auto as pointed out in https://github.com/certbot/certbot/issues/5651#issuecomment-370853487

The reason for the issue is a dependency mismatch with some other package, probably installed from another PPA, that has installed a mismatching version of one of our python dependencies globally. certbot-auto is able to work around this, as it's creating it's isolated Python virtual environment with the correct versions of the required Python dependencies.

[ConfusedVorlon]

Hi - Hoping for an update on this.
I got the email about having to upgrade certbot to move off ACME TLS-SNI-01
I have upgraded everything to the best of my understanding

||/ Name                            Version              Architecture         Description
+++-===============================-====================-====================-====================================================================
ii  openssl                         1.0.2g-1ubuntu4.14   amd64                Secure Sockets Layer toolkit - cryptographic utility
ii  python3-acme                    0.28.0-1+ubuntu16.04 all                  ACME protocol library for Python 3
ii  python3-certbot                 0.28.0-1+ubuntu16.04 all                  main library for certbot
un  python3-certbot-nginx           <none>               <none>               (no description available)
un  python3-cffi                    <none>               <none>               (no description available)
ii  python3-cffi-backend            1.10.0-0.1+ubuntu16. amd64                Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography            1.9-1+ubuntu16.04.1+ amd64                Python library exposing cryptographic recipes and primitives (Python
ii  python3-openssl                 17.3.0-1~0+ubuntu16. all                  Python 3 wrapper around the OpenSSL library

Ubuntu 16.04.5 LTS

certbot --version (or anything else) generates

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.28.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/usr/lib/python3/dist-packages/josepy/__init__.py", line 44, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/usr/lib/python3/dist-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/usr/lib/python3/dist-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 16, in <module>
    from OpenSSL._util import (
  File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 156, in <module>
    Binding.init_static_locks()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 137, in init_static_locks
    cls._ensure_ffi_initialized()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 124, in _ensure_ffi_initialized
    cls.lib = build_conditional_library(lib, CONDITIONAL_NAMES)
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 84, in build_conditional_library
    if not getattr(lib, condition):
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_DTLS'

not sure if it is relevant, but I'm running on Digital Ocean, so regular ubuntu updates probably come from their mirrors

[bmw]

@ConfusedVorlon, the ends of the version strings in your dpkg output seems to be cut off a bit. Can you run the command again and share the full output?

Another thing you can try is reinstalling the cryptography package which is where this error is occurring. The command to run would be:

sudo apt-get update && sudo apt-get install --reinstall python3-cryptography

After running that command, you can check if certbot --version is working for you.

While I'd like to figure out the problem here and get it fixed, another thing you can do is use certbot-auto. I believe you said you were using Apache in which case the instructions on how to do this can be found at https://certbot.eff.org/lets-encrypt/ubuntuother-apache.

[ConfusedVorlon]

I ran dpkg again

dpkg -l python3-{cryptography,openssl,cffi,cffi-backend,certbot,certbot-nginx,acme} openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                             Version               Architecture          Description
+++-================================-=====================-=====================-=====================================================================
ii  openssl                          1.0.2g-1ubuntu4.14    amd64                 Secure Sockets Layer toolkit - cryptographic utility
ii  python3-acme                     0.28.0-1+ubuntu16.04. all                   ACME protocol library for Python 3
ii  python3-certbot                  0.28.0-1+ubuntu16.04. all                   main library for certbot
un  python3-certbot-nginx            <none>                <none>                (no description available)
un  python3-cffi                     <none>                <none>                (no description available)
ii  python3-cffi-backend             1.10.0-0.1+ubuntu16.0 amd64                 Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography             1.9-1+ubuntu16.04.1+c amd64                 Python library exposing cryptographic recipes and primitives (Python 
ii  python3-openssl                  17.3.0-1~0+ubuntu16.0 all                   Python 3 wrapper around the OpenSSL library

then I ran

sudo apt-get update && sudo apt-get install --reinstall python3-cryptography

~# sudo apt-get update && sudo apt-get install --reinstall python3-cryptography
sudo: unable to resolve host nurtureit-blog-2
Hit:1 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease                                                             
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]                                                                
Hit:4 https://repos.sonar.digitalocean.com/apt main InRelease                                                                             
Hit:5 http://lon1.mirrors.digitalocean.com/ubuntu xenial InRelease                  
Get:6 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates InRelease [109 kB] 
Get:7 http://lon1.mirrors.digitalocean.com/ubuntu xenial-backports InRelease [107 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [101 kB]  
Get:9 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [425 kB]      
Get:10 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates/main Sources [330 kB]
Get:11 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates/universe Sources [245 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [170 kB]             
Get:13 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates/main amd64 Packages [912 kB]
Get:14 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates/universe amd64 Packages [727 kB]
Get:15 http://lon1.mirrors.digitalocean.com/ubuntu xenial-updates/universe Translation-en [300 kB]
Fetched 3,535 kB in 1s (2,030 kB/s)                              
Reading package lists... Done
sudo: unable to resolve host nurtureit-blog-2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 2 not upgraded.
Need to get 211 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial/main amd64 python3-cryptography amd64 1.9-1+ubuntu16.04.1+certbot+2 [211 kB]
Fetched 211 kB in 0s (6,212 kB/s)        
(Reading database ... 115159 files and directories currently installed.)
Preparing to unpack .../python3-cryptography_1.9-1+ubuntu16.04.1+certbot+2_amd64.deb ...
Unpacking python3-cryptography (1.9-1+ubuntu16.04.1+certbot+2) over (1.9-1+ubuntu16.04.1+certbot+2) ...
Setting up python3-cryptography (1.9-1+ubuntu16.04.1+certbot+2) ...

this did not resolve the issue

certbot --version
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.28.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/usr/lib/python3/dist-packages/josepy/__init__.py", line 44, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/usr/lib/python3/dist-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/usr/lib/python3/dist-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 16, in <module>
    from OpenSSL._util import (
  File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 156, in <module>
    Binding.init_static_locks()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 137, in init_static_locks
    cls._ensure_ffi_initialized()
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 124, in _ensure_ffi_initialized
    cls.lib = build_conditional_library(lib, CONDITIONAL_NAMES)
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 84, in build_conditional_library
    if not getattr(lib, condition):
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_SCT'

so I ran dpkg again for good measure

dpkg -l python3-{cryptography,openssl,cffi,cffi-backend,certbot,certbot-nginx,acme} openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                        Version                    Architecture               Description
+++-===========================================-==========================-==========================-===========================================================================================
ii  openssl                                     1.0.2g-1ubuntu4.14         amd64                      Secure Sockets Layer toolkit - cryptographic utility
ii  python3-acme                                0.28.0-1+ubuntu16.04.1+cer all                        ACME protocol library for Python 3
ii  python3-certbot                             0.28.0-1+ubuntu16.04.1+cer all                        main library for certbot
un  python3-certbot-nginx                       <none>                     <none>                     (no description available)
un  python3-cffi                                <none>                     <none>                     (no description available)
ii  python3-cffi-backend                        1.10.0-0.1+ubuntu16.04.1+c amd64                      Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-cryptography                        1.9-1+ubuntu16.04.1+certbo amd64                      Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-openssl                             17.3.0-1~0+ubuntu16.04.1+c all                        Python 3 wrapper around the OpenSSL library

any suggestions?

[bmw]

What's the output of:

sudo dpkg -l libssl*

[ConfusedVorlon]

sudo dpkg -l libssl*
sudo: unable to resolve host nurtureit-blog-2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                        Version                    Architecture               Description
+++-===========================================-==========================-==========================-===========================================================================================
ii  libssl1.0.0:amd64                           1.0.2g-1ubuntu4.14         amd64                      Secure Sockets Layer toolkit - shared libraries

[bmw]

@reaperhulk, if you get any spare cycles to take a look, do you have any idea what could be causing the failures @ConfusedVorlon and others are seeing here?

To summarize, they seem to be using the standard OpenSSL packages on Ubuntu with the cryptography and pyopenssl packages from our PPA, but when they run Certbot they get errors like:

AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_*'

The main things we've looked at are what packages they have installed with dpkg which seem correct and the Python tracebacks don't hint at things like a mix of packages from pip.

I'm not exactly sure where we should look next to continue trying to debug this problem. I am not able to reproduce the problem on a clean installation.

[reaperhulk]

This is pretty much always a symbol mismatch between multiple versions of cryptography. Cryptography_HAS_* is a set of invented symbols we add to do feature detection, but if a newer version loads an older version's shared object that symbol won't be present and you'll see that sort of error.

This issue doesn't come up much but the manner in which it expresses is extremely opaque. Maybe we should consider adding the current cryptography version as a symbol during compile and comparing it on import and erroring if there's a mismatch. I think we will try to do that! It doesn't solve the problem, but it will make it very clear what the problem is.

[kikoreis]

I've tracked down one possible root cause of this down to a packaging issue in python-cryptography. The version certbot is distributing via its PPA (1.9) comes with its own .so files:

/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_padding.abi3.so
/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_openssl.abi3.so
/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/_constant_time.abi3.so

However, it fails to remove any .so files that previous versions of the package may have installed. On a stock 16.04 LTS installation, python-cryptography 1.2.3 includes its own .so files:

/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_constant_time.x86_64-linux-gnu.so
/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_openssl.x86_64-linux-gnu.so
/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_padding.x86_64-linux-gnu.so

If you add the PPA and simply upgrade to the version in the PPA, you'll end up with both sets of .so files. The python runtime prefers the filename of the old .so, which is missing key symbols, and you end up with the traceback above. As a workaround, you can simply delete the old .so files. But the packaging should really be fixed to remove them upon upgrade.

[kikoreis]

Here are potential instances of this problem:

https://community.letsencrypt.org/t/update-for-tls-sni-01-now-certbot-fails-on-cryptography-has-dtls/86052
https://blog.yeon.me/ja/goto/928

[kikoreis]

Incidentally, the symbol missing indicated in the error message will vary, because there are a large number of missing symbols, and the loop which checks for symbols isn't ordered. So the first missing symbol of the unordered set will be returned -- the effect is that the error message changes. Here are the results of 5 failing runs:

AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_SCT'
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_TLS_ST'
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_DTLS'
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_RSA_OAEP_MD'
AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_EVP_PKEY_DHX'

[bmw]

@kikoreis, are you able to create something like a Dockerfile that reproduces this problem? I'm unable to reproduce it on a clean installation. I do see both sets of .so files you described, however, one is for Python 2 and the other is for Python 3.

[kikoreis]

Yeah. Sadly, this is one of those times where my initial analysis is all wrong 8). The reason I had leftover Python .so files was nothing to do with the certbot python-cryptography packaging (which is removing old versions correctly, as I just confirmed in a lxd 16.04 LTS instance) and everything to do with a restore from backup. Sorry for the noise.

[kikoreis]

But for the people stuck in this bug, do an ls -l /usr/lib/python3/dist-packages/cryptography/hazmat/bindings/ -- if you see _openssl.cpython-35m-x86_64-linux-gnu.so, or anything _openssl* that differs from _openssl.abi3.so, then you've found found the problem. Just mv *x86_64-linux-gnu.so /tmp or whatever and try again.

[stale[bot]]

We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

[stale[bot]]

This issue has been closed due to lack of activity, but if you think it should be reopened, please open a new issue with a link to this one and we'll take a look.