Cryptography: LibreSSL on FreeBSD include directory mix-up

[unrelentingtech]

Currently, pip install cryptography without specifying LDFLAGS="-L/usr/local/lib" CFLAGS="-I/usr/local/include" and with LibreSSL installed on FreeBSD results in a non-working build that fails like this:

_openssl.abi3.so: Undefined symbol "CMS_decrypt"

or this:

_openssl.abi3.so: Undefined symbol "d2i_DHxparams"

I have figured it out! So the library ends up linked to the /usr/local/lib libssl (package installed *SSL), but configuration headers are read from /usr/include (base system OpenSSL) instead of /usr/local/include!

I'm not very familiar with the cffi build system, but pkg-config returns -I/usr/local/include -L/usr/local/lib, so I have no idea how /usr/include ends up used…

[reaperhulk]

The cffi build system is really just the setuptools/distutils machinery, so it's using whatever defaults your Python would use for building a C module. I don't believe Python uses pkg-config at all so it will just use whatever the compiler's defaults are for include and lib dirs, although it's possible that's not true on all platforms.

If you want to see what the exact compiler invocations are you can pip install -vvv cryptography and you should see all the output.

[reaperhulk]

Closing this due to lack of activity, but it can be reopened if there's more information.

[unrelentingtech]

What more information do you need?

You should use pkg-config (e.g. like this) when it's available.

[fyfe]

Installing cryptography like so works on FreeBSD with openssl/libressl installed from ports

pip install --global-option=build_ext --global-option="-I/usr/local/include/" --global-option="-L/usr/local/lib" cryptography

[fyfe]

Install log for pip install --no-cache-dir -vvv cryptography

Install log for pip install --no-cache-dir -vvv --global-option=build_ext --global-option="-I/usr/local/include/" --global-option="-L/usr/local/lib" cryptography

Search for_openssl.o and -lssl for the relevant lines, I don't know how the build system works but the logs show the first command is adding -L/usr/local/lib with no -I causing the module to have a version mismatch between headers and libraries.

[mundschenk-at]

@reaperhulk I've run into this issue as well. What information do you need to debug this?

[tykling]

hello @reaperhulk and everyone. I am having a related issue, although not with LibreSSL, the base problem is the same:

• FreeBSD has two versions of OpenSSL installed, one in /usr/lib and another in /usr/local/lib
• cryptography appears to detect the headers from the base OpenSSL in /usr/lib but links against the ports OpenSSL in /usr/local/lib
• So a pip install cryptography fails like so:

[tykling@dns3 ~]$ rm -rf venv && virtualenv venv && ./venv/bin/pip install cryptography && ./venv/bin/python -c 'from cryptography.hazmat.bindings._openssl import ffi, lib'                                                                                       
New python executable in /usr/home/tykling/venv/bin/python2.7
Also creating executable in /usr/home/tykling/venv/bin/python
Installing setuptools, pip, wheel...done.
Collecting cryptography
Collecting enum34; python_version < "3" (from cryptography)
  Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting idna>=2.1 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.7 (from cryptography)
Collecting six>=1.4.1 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography)
  Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography)
Installing collected packages: enum34, idna, asn1crypto, pycparser, cffi, six, ipaddress, cryptography
Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 idna-2.8 ipaddress-1.0.22 pycparser-2.19 six-1.12.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/home/tykling/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so: Undefined symbol "EVP_MD_CTX_new"
[tykling@dns3 ~]$ 

• With the flags provided by @fyfe earlier in this issue it installs and works well:

[tykling@dns3 ~]$ rm -rf venv && virtualenv venv && ./venv/bin/pip install --global-option=build_ext --global-option="-I/usr/local/include/" --global-option="-L/usr/local/lib" cryptography && ./venv/bin/python -c 'from cryptography.hazmat.bindings._openssl import ffi, lib'
New python executable in /usr/home/tykling/venv/bin/python2.7
Also creating executable in /usr/home/tykling/venv/bin/python
Installing setuptools, pip, wheel...done.
/usr/home/tykling/venv/lib/python2.7/site-packages/pip/_internal/commands/install.py:211: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting cryptography
  Using cached https://files.pythonhosted.org/packages/f3/39/d3904df7c56f8654691c4ae1bdb270c1c9220d6da79bd3b1fbad91afd0e1/cryptography-2.4.2.tar.gz
  Installing build dependencies ... done
Collecting idna>=2.1 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz
Collecting asn1crypto>=0.21.0 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/fc/f1/8db7daa71f414ddabfa056c4ef792e1461ff655c2ae2928a2b675bfed6b4/asn1crypto-0.24.0.tar.gz
Collecting six>=1.4.1 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/dd/bf/4138e7bfb757de47d1f4b6994648ec67a51efe58fa907c1e11e350cddfca/six-1.12.0.tar.gz
Collecting cffi!=1.11.3,>=1.7 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz
Collecting enum34 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/bf/3e/31d502c25302814a7c2f1d3959d2a3b3f78e509002ba91aea64993936876/enum34-1.1.6.tar.gz
Collecting ipaddress (from cryptography)
  Using cached https://files.pythonhosted.org/packages/97/8d/77b8cedcfbf93676148518036c6b1ce7f8e14bf07e95d7fd4ddcb8cc052f/ipaddress-1.0.22.tar.gz
Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography)
  Using cached https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz
Skipping bdist_wheel for cryptography, due to binaries being disabled for it.
Skipping bdist_wheel for idna, due to binaries being disabled for it.
Skipping bdist_wheel for asn1crypto, due to binaries being disabled for it.
Skipping bdist_wheel for six, due to binaries being disabled for it.
Skipping bdist_wheel for cffi, due to binaries being disabled for it.
Skipping bdist_wheel for enum34, due to binaries being disabled for it.
Skipping bdist_wheel for ipaddress, due to binaries being disabled for it.
Skipping bdist_wheel for pycparser, due to binaries being disabled for it.
Installing collected packages: idna, asn1crypto, six, pycparser, cffi, enum34, ipaddress, cryptography
  Running setup.py install for idna ... done
  Running setup.py install for asn1crypto ... done
  Running setup.py install for six ... done
  Running setup.py install for pycparser ... done
  Running setup.py install for cffi ... done
  Running setup.py install for enum34 ... done
  Running setup.py install for ipaddress ... done
  Running setup.py install for cryptography ... done
Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 idna-2.8 ipaddress-1.0.22 pycparser-2.19 six-1.12.0
[tykling@dns3 ~]$ 

This is a problem on FreeBSD 12, where the current situation is that the base OpenSSL is newer than the ports OpenSSL right now.

Is there anything that can be fixed in setup.py or somewhere to make this work? Please let me know if you need any further information. Thank you :+1:

[reaperhulk]

This is an issue related to linker and include default ordering. The workaround you're doing is the best solution until the ports issue is resolved.

[ecederstrand]

I'm getting this again with newer pip versions. With pip 19.3.1:

$ rm -rf venv \
  && python3 -m venv venv \
  && source venv/bin/activate.csh \
  && pip3 install -U pip==19.3.1 \
  && pip3 install --global-option=build_ext --global-option="-I/usr/local/include/" --global-option="-L/usr/local/lib" cryptography \
  && python3 -c 'from cryptography.hazmat.bindings._openssl import ffi, lib'
[...]
Successfully installed cffi-1.13.2 cryptography-2.8 pycparser-2.19 six-1.13.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/home/erik/venv/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "d2i_DHxparams"
$

With pip 18.1:

$ rm -rf venv \
  && python3 -m venv venv \
  && source venv/bin/activate.csh \
  && pip3 install -U pip==18.1 \
  && pip3 install --global-option=build_ext --global-option="-I/usr/local/include/" --global-option="-L/usr/local/lib" cryptography \
  && python3 -c 'from cryptography.hazmat.bindings._openssl import ffi, lib'
[...]
Successfully installed cffi-1.13.2 cryptography-2.8 pycparser-2.19 six-1.13.0
You are using pip version 18.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$

[tykling]

I very recently had yet another round of fighting this when I switched python versions, I figured out in the end that the --global-option -I and -L options are added too late in the commandline - after another -I and -L which includes the base libs. I also figured out that setting CFLAGS and LDFLAGS env vars instead of setting --global-option on pip solved it. So I ended up with this lovely commit in my ansible roles, maybe you can try the same https://github.com/tykling/ansible-roles/commit/8a68351d0b2057b3fc2e89582e7af40a60783df4#diff-616bf354c31e534a0753dcc12d15f681

It is tricky because when checking the _openssl.so file with ldd it shows that it has linked against the ports libs, but not before getting the symbols from the base libs. Annoying stuff.

[ecederstrand]

Thanks for the explanation, @tykling! The CFLAGS and LDFLAGS paths in the referenced commit didn't work (I see you changed them in a later commit), This command worked for me (tested with pip==19.3.1).

CFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/include" pip3 install --global-option=build_ext cryptography