Cryptography: OpenSSL binaries shipped with 2.9.1 wheels fail to load on macOS High Sierra

[colatkinson]

Installation and importing work as expected. But once the OpenSSL backend is actually loaded, an exception is raised due to a missing symbol ____chkstk_darwin. Based on some similar issues I found, this seems like an ABI compatibility issue [1][2][3].

During packaging, if I understand correctly seems the shipped version of OpenSSL is pulled via brew [4]. This seems to pull in a Catalina-specific variant of the library. A possible solution then could be to force usage of a bottle for the earliest-supported macOS version, but I'm unsure (a) how to do this and (b) if there are issues that would arise from doing so.

Not sure what the best approach for testing such a fix is. Since the pipelines all run on Catalina, manual testing is required (which I can do easily enough). But I'm not sure how to run a pipeline that generates a wheel artifact to test.

Software versions:

• python: 3.6.8
• cryptography: 2.9.1
• cffi: 1.14.0
• pip: 18.1
• setuptools: 40.6.2

macOS

ProductName:    Mac OS X
ProductVersion: 10.13.6
BuildVersion:   17G12034

Installation method

Standard pip install. Pulls in cryptography-2.9.1-cp35-abi3-macosx_10_9_x86_64.whl.

Steps to reproduce

Take this script as an example:

from cryptography.hazmat.backends import default_backend

default_backend()

2.9.0 release:

$ pip install cryptography==2.9.0
$ python repro.py  # status code 0

2.9.1 release:

$ pip install cryptography==2.9.1
$ python repro.py
Traceback (most recent call last):
  File "repro.py", line 3, in <module>
    default_backend()
  File "/Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 75, in <module>
    from cryptography.hazmat.bindings.openssl import binding
  File "/Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 16, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/vida/tmp-venv/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so

[studioj]

similar on 2.7.16 32bit (yes we're migrating as fast as possible :-))

ProductName: Mac OS X
ProductVersion: 10.13.6

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
    from cryptography.hazmat.backends.openssl.backend import backend
  File "/Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 75, in <module>
    from cryptography.hazmat.bindings.openssl import binding
  File "/Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 16, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: dlopen(/Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/franky/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

had some issues with the windows install too... but that magically disappeared when i did a straight install from console (maybe a different package selected by pip??)

  generating cffi module 'build\\temp.win32-2.7\\Release\\_padding.c'
  creating build\temp.win32-2.7
  creating build\temp.win32-2.7\Release
  generating cffi module 'build\\temp.win32-2.7\\Release\\_constant_time.c'
  generating cffi module 'build\\temp.win32-2.7\\Release\\_openssl.c'
  building '_openssl' extension
  creating build\temp.win32-2.7\Release\build
  creating build\temp.win32-2.7\Release\build\temp.win32-2.7
  creating build\temp.win32-2.7\Release\build\temp.win32-2.7\Release
  C:\Users\franky\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\python27\include -Ic:\jenkins\workspace\my_own_package_single_pipeline_git\my_own_packageinstallcheckpy2\PC /Tcbuild\temp.win32-2.7\Release\_openssl.c /Fobuild\temp.win32-2.7\Release\build\temp.win32-2.7\Release\_openssl.obj
  _openssl.c
  build\temp.win32-2.7\Release\_openssl.c(546) : fatal error C1083: Cannot open include file: 'openssl/opensslv.h': No such file or directory
  error: command 'C:\\Users\\franky\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\cl.exe' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for cryptography
Successfully built my_own_package
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly

[reaperhulk]

It's interesting that 2.9 worked while 2.9.1 does not. We built 2.9 on a 10.14 builder (on a different CI system so we can't switch back, sorry) so maybe that's what caused this... We do specify -mmacosx-version-min=10.9, which as I understand it should have prevented this.

[reaperhulk]

Okay we've figured out the bug. We need to alter our build infrastructure to handle it, but we'll ship a 2.9.2 to resolve this soon (hopefully today).

[reaperhulk]

https://github.com/pyca/infra/pull/275 is the beginning of what we're going to need to do here.

[felixfontein]

I just experienced the same problem on macOS 10.11 with Python 2.7:

[...]
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: ____chkstk_darwin
  Referenced from: /usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
  Expected in: /usr/lib/libSystem.B.dylib
  in /usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

(Full stack trace can be found here if of interest.)

[reaperhulk]

This will affect any user installing a wheel on macOS < 10.15 right now. I'll have a test wheel soon-ish (thanks slow CI) if someone is willing to check that our updated builds work on older macOS.

[studioj]

This will affect any user installing a wheel on macOS < 10.15 right now. I'll have a test wheel soon-ish (thanks slow CI) if someone is willing to check that our updated builds work on older macOS.

if there are no volunteers by Friday morning UTC I'll gladly validate then

[colatkinson]

@reaperhulk I can test whenever the wheel is available.

[reaperhulk]

@colatkinson Here's a wheel to check! https://github.com/pyca/cryptography/suites/620232032/artifacts/4882584

[colatkinson]

@reaperhulk Works on my machine!

[reaperhulk]

Perfect, okay. We're going to run a 2.9.2 shortly then.

[reaperhulk]

2.9.2 is now released and this should resolve the issue! I'll go ahead and close but please comment if 2.9.2 does not fix it.

[felixfontein]

I can confirm that 2.9.2 again works for macOS 10.11. Thanks for fixing this!

[studioj]

confirmed! thanks @reaperhulk