Cryptography: if homebrew openssl is linked, cryptography builds an unusable wheel

[glyph]

You can see the exception below:

http://asciinema.org/a/0n4rvtw727jfn5lhs981hjusx

Please enjoy _real-time_ Cryptography wheel build times in that video, as I had to enjoy them about 50 times while regressing this.

In short, the error is Symbol not found: _BIO_new_CMS. This has been seen before, by OpenStack developers, and apparently the folk wisdom in that community is "use a linux VM" which avoids this issue.

However, I can _properly_ build a homebrew OpenSSL wheel by exporting LDFLAGS, CPPFLAGS, CFLAGS, and so on, _without_ doing a brew link openssl first. I think that something is getting confused about the difference between /usr/local/lib/lib(crypto|ssl).dylib and /usr/lib/lib(crypto|ssl).dylib

[glyph]

Oops, the "by OpenStack developers" was supposed to be a link, like this: by OpenStack developers

[glyph]

Apparently I misspoke, I _also_ can't build a wheel against homebrew openssl manually; the error is:

cffi.ffiplatform.VerificationError: importing '.../_Cryptography_cffi_a269d620xd5c405b7.so': dlopen(/.../_Cryptography_cffi_a269d620xd5c405b7.so, 2): Library not loaded: /usr/local/lib/libssl.1.0.0.dylib

[glyph]

DYLD_LIBRARY_PATH=/usr/local/opt/openssl/lib does fix the problem but I suspect that this is not the intended solution.

[alex]

I beleive this si a dupe of #2006

[hynek]

I’m not sure it’s a dupe; 2006 is about how the cache confuses ppl. This sounds like a general build problem?

[reaperhulk]

Yeah this is a new thing, and actually a reasonably serious issue since it turns out homebrew no longer makes OpenSSL keg only as of two days ago (see: https://github.com/Homebrew/homebrew/commit/9ca3c054c070cc825870a6fc0fc5c5a22f7d8559#diff-5f8300a99f5c14b861dfa8eff2905a16)

Apple's CLI tools for El Capitan do not ship OpenSSL headers any more. So, hopefully this is something we can control, although I'm not sure how.

[reaperhulk]

The only thing required to replicate this on my machine is homebrew (with linked openssl. This is the default behavior as of 1.0.2d) and running pip install cryptography --no-use-wheel.

I believe that it's looking at the headers for /usr/local/include/openssl but then linking against /usr/lib/lib{crypto,ssl}.

When running pip install cryptography --no-use-wheel I get:

_Cryptography_cffi_a269d620xd5c405b7.so:
    /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)

If I run LDFLAGS="-L/usr/local/opt/openssl/lib" pip install cryptography --no-use-wheel then everything works as expected since it overrides the (apparently incorrect) order of priority for the linker. "-L/usr/local/lib" works as well.

I thought the linker default path in OS X was $(HOME)/lib:/usr/local/lib:/lib:/usr/lib but maybe I'm wrong. Any ideas?

[reaperhulk]

https://github.com/Homebrew/homebrew/issues/41613

[tdsmith]

FWIW, man ld says: "The default library search path is /usr/lib then /usr/local/lib. The default framework search path is /Library/Frameworks then /System/Library/Frameworks."

[reaperhulk]

I'm removing the milestone since the fix for this is not under our control (other than some potential documentation work).

[glyph]

Worth noting: this happens with system python, python.org python, and pyenv python, but _not_, for some reason, with homebrew python. I still haven't figured out exactly why.

[glyph]

I think that tweaks to setup.py _could_ fix this - or at least work around it - in cryptography. Not to mention the fact that shipping static wheels for OS X will address the problem.

[tdsmith]

Worth noting: this happens with system python, python.org python, and pyenv python, but not, for some reason, with homebrew python. I still haven't figured out exactly why.

We discussed this in IRC; after setting DYLD_PRINT_LIBRARIES, we saw that cffi imports hashlib, which (for Homebrew python) is linked against libssl and libcrypto libraries that match the configuration in the include files that were picked up during the build. Those appear to mask the system libraries against which the cytography .so's are nominally linked.

That makes me worried about whether bundling dynamic libraries in a wheel with e.g. delocate will work; if system Python's hashlib is linked against the system libssl, will that mask the wheel's libssl?

[glyph]

Aah, thanks for the reference @tdsmith, I didn't appreciate the nuance of "because something's using hashlib" in that conversation.

I am still at a bit of a loss for what the "right" answer really is here. Would it make sense to invoke the C compiler explicitly in setup.py to discover which headers are in use, and then explicitly set LDFLAGS to point -L at the parallel lib location? It seems like the use-case for _not_ wanting that to happen would be sufficiently obscure that you could add a special environment variable to disable that check, rather than leaving it broken by default if you have different things in /usr/local and /usr.

[hynek]

GUISE STAHP YOU’RE BUILDING AUTOCONF ;)

[glyph]

Those who do not understand autoconf are doomed to repeat it.

On the other hand, those who _do_ understand autoconf are doomed to _use_ it.

So... not really sure which side of that tradeoff we want to be on.

[tdsmith]

Homebrew's openssl has been removed from /usr/local in Homebrew/homebrew@2e191b19b8e.

I don't know if this is the place to ask, but are you expecting Python.org's python distribution to begin bundling openssl and matching headers, since the headers are going away in OS X 10.11? How will cryptography find them?

[tdsmith]

(Or just resort to static linking, I guess.)

[glyph]

/me updates brew and re-creates every single virtual environment _again_...

[glyph]

Just to summarize the state here; this is covered a bit in https://github.com/Homebrew/homebrew/issues/41613 -

• Cryptgraphy wants to just use whatever OpenSSL is ambiently available in your environment. It achieves this by doing #include <openssl/...> in its headers and and -lcrypto -lssl on its build command line.
• On OS X, the C preprocessor puts /usr/local/include first by default, but the linker puts /usr/lib first by default. Therefore, if an OpenSSL is installed into /usr/local, the wheel will be broken (homebrew is just one instance of this problem).
• Cryptography detects conditionally-available names, like _BIO_new_CMS, by putting #ifdefs into its verify section; if the names are not declared by the headers its including, then they're defined by Cryptography itself; they always exist in the cdef section but the definitions will be dummies from the verify section if the preprocessor check fails. So this is done entirely in the preprocessor: nothing about the linking process during compilation notices the missing symbols, so it happily links to the wrong library.
• The install_name of the wrong library is written into Cryptography's .sos, making them persistently broken.
• El Capitan is going to stop including header files for OpenSSL, but keep shipping dylibs, so without homebrew (or equivalent) Cryptography just won't build at all; with it, it will build something broken by default.

Fundamentally, this appears to boil down to:

• Cryptography wants to depend on the native platform's C toolchain dependency management, and OS X's native platform C toolchain is broken by default.

[public]

Speaking of re-inventing autoconf. Why don't we shell out to pkg-config if it's available?

[glyph]

@public - that actually seems pretty reasonable to me. homebrew provides it, and on my system (which is now back to up-to-date homebrew, meaning, keg-only openssl):

$ pkg-config openssl --cflags 
-I/usr/local/Cellar/openssl/1.0.2d/include 
$ pkg-config openssl --libs
-L/usr/local/Cellar/openssl/1.0.2d/lib -lssl -lcrypto 

Of course, I'm sure that there will be some times when pkg-config does the _wrong_ thing, so is there some way to tell it what location to point at?

[public]

ENVIRONMENT VARIABLES
PKG_CONFIG_PATH
A colon-separated (on Windows, semicolon-separated) list of directories to search for .pc files. The default directory will always be searched
after searching the path; the default is libdir/pkgconfig:datadir/pkgconfig where libdir is the libdir for pkg-config and datadir is the datadir
for pkg-config when it was installed.

[glyph]

I did see PKG_CONFIG_PATH, but that seems to only be additive:

$ PKG_CONFIG_PATH=nope pkg-config openssl --libs
-L/usr/local/Cellar/openssl/1.0.2d/lib -lssl -lcrypto 

[public]

PKG_CONFIG_LIBDIR seems like it might do what you are after?

[fqx]

So, does anybody know how to install cryptography without home-brew under El Capitan?

[reaperhulk]

@fqx since Apple removed the OpenSSL headers it is not possible to install cryptography without homebrew. With homebrew you can install it using the instructions we have on https://cryptography.io

[mrjefftang]

Is it possible to publish statically linked wheels for OS X?

[reaperhulk]

@mrjefftang we'll be discussing that more seriously as El Capitan gets closer to release, but yes, we can.

[glyph]

Remember that the statically linked wheels will need to come with a pile of trust roots, and then get updated for trust root changes :-).

[glyph]

@reaperhulk - one of the problems with following those instructions, I believe, is that you'll still get a wheel dynamically linked against homebrew. In other words, a cryptography wheel that only functions on a system with homebrew already installed. Perhaps an interim step would be to document how one could at least build one's own static cryptography wheel, before automating it for everyone?

[glyph]

@fqx @reaperhulk also, to be clear, you don't _need_ homebrew. You can just download the OpenSSL sources and build them yourself. It's just that homebrew is generally easier.

[reaperhulk]

Building your own static linking on OS X in a way that isn't super horrible is not a thing I know how to do. Right now the only way I know how to do it is to explicitly _remove_ the dylibs from the path, which forces it to use the .a file.

Hopefully there is a better way that I am not familiar with.

[glyph]

I have seen some references to this but they all seem to mention gcc and not clang; specify the ".a" on the command line; possibly including the full path.

I think the thing to try would be modifying _get_openssl_libraries to suffix things with '.a'?

[glyph]

Nope. It just goes on the command line as a ... thing, not as a -l. I did this:

diff --git a/src/_cffi_src/build_openssl.py b/src/_cffi_src/build_openssl.py
index 6a5bf2d..b88bb6f 100644
--- a/src/_cffi_src/build_openssl.py
+++ b/src/_cffi_src/build_openssl.py
@@ -16,7 +16,7 @@ def _get_openssl_libraries(platform):
         # specified on the linker command-line is significant;
         # libssl must come before libcrypto
         # (http://marc.info/?l=openssl-users&m=135361825921871)
-        return ["ssl", "crypto"]
+        return []
     else:
         return ["libeay32", "ssleay32", "advapi32",
                 "crypt32", "gdi32", "user32", "ws2_32"]
@@ -79,5 +79,8 @@ ffi = build_ffi_for_binding(
     pre_include=_OSX_PRE_INCLUDE,
     post_include=_OSX_POST_INCLUDE,
     libraries=_get_openssl_libraries(sys.platform),
-    extra_link_args=extra_link_args(sys.platform),
+    extra_link_args=extra_link_args(sys.platform) + [
+        "/usr/local/opt/openssl/lib/libcrypto.a",
+        "/usr/local/opt/openssl/lib/libssl.a",
+    ]
 )

and it produced a vaguely usable library:

$ python -c 'from cryptography.hazmat.bindings.openssl.binding import Binding; print(Binding())'
<cryptography.hazmat.bindings.openssl.binding.Binding object at 0x10d4ab5d0>

[glyph]

and it doesn't dynamically link against homebrew:

$ otool -L "${VIRTUAL_ENV}"/lib/*/site-packages/cryptography/hazmat/bindings/*.so
/Users/glyph/.virtualenvs/tmp-94fa8b4ef8dbca54/lib/python2.7/site-packages/cryptography/hazmat/bindings/_commoncrypto.so:
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57031.30.12)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1153.18.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/Users/glyph/.virtualenvs/tmp-94fa8b4ef8dbca54/lib/python2.7/site-packages/cryptography/hazmat/bindings/_constant_time.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/Users/glyph/.virtualenvs/tmp-94fa8b4ef8dbca54/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/Users/glyph/.virtualenvs/tmp-94fa8b4ef8dbca54/lib/python2.7/site-packages/cryptography/hazmat/bindings/_padding.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

[glyph]

Since @tdsmith asked:

$ python -c 'import OpenSSL.SSL as SSL; print SSL.SSLeay_version(SSL.SSLEAY_VERSION)'
OpenSSL 1.0.2d 9 Jul 2015

[glyph]

One area of concern here is that all of the engines appear to be built _only_ as dylibs:

$ ls /usr/local/opt/openssl/lib/engines/
lib4758cca.dylib  libcapi.dylib     libgmp.dylib      libpadlock.dylib
libaep.dylib      libchil.dylib     libgost.dylib     libsureware.dylib
libatalla.dylib   libcswift.dylib   libnuron.dylib    libubsec.dylib

so I don't know what functionality will be missing on a system that doesn't have those files.

[reaperhulk]

Closing this since we now ship static wheels on OS X to handle most of this issue.

[miohtama]

Looks like wheel for OSX El Capitan (10.11) is missing?

I am trying to do pip install cryptography and it tries to compile it and fails.

Python 2.7.6 (default, Apr 10 2014, 22:40:17) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Then:

pip install cryptography
Downloading/unpacking cryptography
  Downloading cryptography-1.1.tar.gz (348kB): 348kB downloaded
  Running setup.py egg_info for package cryptography

....

writing manifest file 'src/cryptography.egg-info/SOURCES.txt'

running build_ext

generating cffi module 'build/temp.macosx-10.9-x86_64-2.7/_commoncrypto.c'

creating build/temp.macosx-10.9-x86_64-2.7

generating cffi module 'build/temp.macosx-10.9-x86_64-2.7/_padding.c'

generating cffi module 'build/temp.macosx-10.9-x86_64-2.7/_constant_time.c'

generating cffi module 'build/temp.macosx-10.9-x86_64-2.7/_openssl.c'

building '_openssl' extension

creating build/temp.macosx-10.9-x86_64-2.7/build

creating build/temp.macosx-10.9-x86_64-2.7/build/temp.macosx-10.9-x86_64-2.7

clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c build/temp.macosx-10.9-x86_64-2.7/_openssl.c -o build/temp.macosx-10.9-x86_64-2.7/build/temp.macosx-10.9-x86_64-2.7/_openssl.o

build/temp.macosx-10.9-x86_64-2.7/_openssl.c:408:10: fatal error: 'openssl/aes.h' file not found

#include <openssl/aes.h>

         ^

[reaperhulk]

Your Python is built against the 10.9 SDK and therefore pip doesn't think the wheel (which is built against 10.10) is compatible. In reality it is, but pip has no way to know that.

[miohtama]

@reaperhulk: Any way to force it to use the wheel?

[reaperhulk]

You can download the wheel for your python directly from pypi (https://pypi.python.org/pypi/cryptography), rename it from 10_10 to 10_9 and then pip install the wheel file directly and it will work.

[miohtama]

Perfect. Thank you!

[legovaer]

I got exactly the same issue in OSX El Capitan (10.11) but I'm using the 10.10 SDK.

I am trying to execute pip install cryptography and it tries to compile it and fails.

$ python
Python 2.7.9 (default, Dec 19 2014, 06:00:59) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

...
writing manifest file 'src/cryptography.egg-info/SOURCES.txt'

running build_ext

generating cffi module 'build/temp.macosx-10.10-x86_64-2.7/_commoncrypto.c'

creating build/temp.macosx-10.10-x86_64-2.7

generating cffi module 'build/temp.macosx-10.10-x86_64-2.7/_padding.c'

generating cffi module 'build/temp.macosx-10.10-x86_64-2.7/_constant_time.c'

generating cffi module 'build/temp.macosx-10.10-x86_64-2.7/_openssl.c'

building '_openssl' extension

creating build/temp.macosx-10.10-x86_64-2.7/build

creating build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7

clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c build/temp.macosx-10.10-x86_64-2.7/_openssl.c -o build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7/_openssl.o

build/temp.macosx-10.10-x86_64-2.7/_openssl.c:408:10: fatal error: 'openssl/aes.h' file not found
#include <openssl/aes.h>

         ^

Executing the steps in pyca/cryptography#2138 seems to work.

[reaperhulk]

@legovaer you should get a wheel there -- are you on an old pip? Upgrade pip to latest and try to install again and I would expect it to download a prebuilt binary.