Cryptography: Unable to install on macOS Big Sur (release version) and Xcode 12.2

[Fallen-Zen]

Using command sudo pip3 install cryptography to install the package. Tried to install dependency manager conan first.
Getting this errors in the console (below). What can I do? I'm on an Intel Mac, not ARM.

Collecting cryptography
  Downloading cryptography-3.2.1.tar.gz (540 kB)
     |████████████████████████████████| 540 kB 2.2 MB/s 
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Library/Python/3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/tmp/pip-build-env-8df8aq_8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
       cwd: None
  Complete output (107 lines):
  WARNING: The directory '/Users/fallen/Library/Caches/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
  Collecting setuptools>=40.6.0
    Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
  Collecting wheel
    Downloading wheel-0.35.1-py2.py3-none-any.whl (33 kB)
  Collecting cffi!=1.11.3,>=1.8
    Downloading cffi-1.14.3.tar.gz (470 kB)
  Collecting pycparser
    Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
  Building wheels for collected packages: cffi
    Building wheel for cffi (setup.py): started
    Building wheel for cffi (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-z6nn4o_z/cffi/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-z6nn4o_z/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/tmp/pip-wheel-i34xbxgo
         cwd: /private/tmp/pip-install-z6nn4o_z/cffi/
    Complete output (39 lines):
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14.6-x86_64-3.8
    creating build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/backend_ctypes.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/error.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/setuptools_ext.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/cffi_opcode.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/vengine_gen.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/pkgconfig.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/model.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/ffiplatform.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/api.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/vengine_cpy.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/commontypes.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/lock.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/recompiler.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/cparser.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/verifier.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/_cffi_include.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/parse_c_type.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/_embedding.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    copying cffi/_cffi_errors.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.macosx-10.14.6-x86_64-3.8
    creating build/temp.macosx-10.14.6-x86_64-3.8/c
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ffi -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c c/_cffi_backend.c -o build/temp.macosx-10.14.6-x86_64-3.8/c/_cffi_backend.o
    c/_cffi_backend.c:5854:2: error: Apple Arm64 ABI requires ffi_prep_cif_var
    #error Apple Arm64 ABI requires ffi_prep_cif_var
     ^
    c/_cffi_backend.c:6304:9: error: implicit declaration of function 'ffi_prep_closure' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        if (ffi_prep_closure(closure, &cif_descr->cif,
            ^
    2 errors generated.
    error: command 'clang' failed with exit status 1
    ----------------------------------------
    ERROR: Failed building wheel for cffi
    Running setup.py clean for cffi
  Failed to build cffi
  Installing collected packages: setuptools, wheel, pycparser, cffi
      Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-z6nn4o_z/cffi/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-z6nn4o_z/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-s68cz8tk/install-record.txt --single-version-externally-managed --prefix /private/tmp/pip-build-env-8df8aq_8/overlay --compile --install-headers /private/tmp/pip-build-env-8df8aq_8/overlay/include/python3.8/cffi
           cwd: /private/tmp/pip-install-z6nn4o_z/cffi/
      Complete output (39 lines):
      running install
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.14.6-x86_64-3.8
      creating build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/backend_ctypes.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/error.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/setuptools_ext.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/cffi_opcode.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/vengine_gen.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/pkgconfig.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/model.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/ffiplatform.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/api.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/vengine_cpy.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/commontypes.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/lock.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/recompiler.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/cparser.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/verifier.py -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/_cffi_include.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/parse_c_type.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/_embedding.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      copying cffi/_cffi_errors.h -> build/lib.macosx-10.14.6-x86_64-3.8/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.macosx-10.14.6-x86_64-3.8
      creating build/temp.macosx-10.14.6-x86_64-3.8/c
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ffi -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c c/_cffi_backend.c -o build/temp.macosx-10.14.6-x86_64-3.8/c/_cffi_backend.o
      c/_cffi_backend.c:5854:2: error: Apple Arm64 ABI requires ffi_prep_cif_var
      #error Apple Arm64 ABI requires ffi_prep_cif_var
       ^
      c/_cffi_backend.c:6304:9: error: implicit declaration of function 'ffi_prep_closure' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
          if (ffi_prep_closure(closure, &cif_descr->cif,
              ^
      2 errors generated.
      error: command 'clang' failed with exit status 1
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-z6nn4o_z/cffi/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-z6nn4o_z/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-s68cz8tk/install-record.txt --single-version-externally-managed --prefix /private/tmp/pip-build-env-8df8aq_8/overlay --compile --install-headers /private/tmp/pip-build-env-8df8aq_8/overlay/include/python3.8/cffi Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Library/Python/3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/tmp/pip-build-env-8df8aq_8/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.

[alex]

You're hitting this error in compiling cffi, one of cryptography's dependencies. You'll need to talk to the cffi folks about this issue.

[reaperhulk]

It is odd that pip isn't grabbing a wheel here unless you're on Apple Silicon though. If you're on x86_64 you may want to investigate whether you get binary wheels properly and file a bug with pip if not.

[hynek]

See also https://github.com/hynek/argon2-cffi/issues/68 and https://foss.heptapod.net/pypy/cffi/-/issues/477

My bug reporter went silent unfortunately but there's already a bug on cffi that people know about.

[charklewis]

Are there any workarounds for this? Python is used in a few of my work projects, and I normally don't interact with python so unsure how to progress.

[dustinrb]

@chark I had the same issue with CFFI when compiling on macOS 11 with the included version of Python 3.8. Switching to an alternative distribution fixed this for now. I'm using [email protected] from Homebrew.

[charklewis]

@dustinrb I am not to sure what you mean by this. Would this by to run brew install python? Will this impact any of the python setup/config/envs on my computer?

[reaperhulk]

For Big Sur Intel this appears to be https://github.com/pypa/pip/issues/9138. cryptography and cffi wheels should work fine, but they're not being detected as compatible by pip. There's nothing to be done here on our part, closing.

(Note that the pip issue includes some ugly workarounds users can use)

[sdebnath]

For those who have run into this and land on this issue from google searching, creating an environment via /usr/local/bin/python3 -m venv env after installing python3.9 via homebrew seems to have produced a working and usable environment. Good luck!

[charklewis]

Thanks @sdebnath I have done this and it has worked.

[dKemen]

Worked, thanks a lot!

[Greyvend]

For Big Sur Intel this appears to be pypa/pip#9138. cryptography and cffi wheels should work fine, but they're not being detected as compatible by pip. There's nothing to be done here on our part, closing.

(Note that the pip issue includes some ugly workarounds users can use)

cffi has made new release addressing this, now the compiling fails on cryptography due to OpenSSL. It's similar to what you seem to have addressed here. Are there any non-hacky solutions for that now?

[reaperhulk]

Now that cffi compiles against Apple Silicon you can compile cryptography using the instructions here: https://cryptography.io/en/latest/installation.html#building-cryptography-on-macos

Until we have access to M1 resources in our CI infrastructure (or I get an M1 Mac) we won't be able to ship wheels so this will be the only way to build cryptography using Apple Silicon for the foreseeable future.

[reaperhulk]

Please note that due to that pip/packaging bug you won't automatically get the wheel on x86_64 at this time either. You can either build it yourself (with those same instructions), manually download the wheel and install it (which will work just fine), or wait for pip to ship a fix.

[EVINK]

For those who have run into this and land on this issue from google searching, creating an environment via /usr/local/bin/python3 -m venv env after installing python3.9 via homebrew seems to have produced a working and usable environment. Good luck!

this is working for me on M1 mac

[bengolds]

Now that cffi compiles against Apple Silicon you can compile cryptography using the instructions here: https://cryptography.io/en/latest/installation.html#building-cryptography-on-macos

Until we have access to M1 resources in our CI infrastructure (or I get an M1 Mac) we won't be able to ship wheels so this will be the only way to build cryptography using Apple Silicon for the foreseeable future.

Looking forward to that! In the meantime, have people been able to compile cryptography on Apple Silicon using the instructions in the link? The command outlined on the site seems to crash with the same error as just plain running pip install cryptography.

ImportError: dlopen(/private/var/folders/mw/r8p01knj6lgg53x2x0h7yvvw0000gn/T/pip-build-env-kxj5899j/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure

If I can be of help, let me know! I get the same error when I try to run pip install -r dev-requirements.txt on the downloaded repo as well.

[my understanding of the approach of using /usr/local/bin/python3 -m venv env is that it's using an x86 version of python. ARM Homebrew uses /opt/homebrew/bin/python3 by default.]

[reaperhulk]

/usr/local/bin is locally installed software and not supplied by Apple. Users may have installed an x86_64 or arm64 python in that path. For example, if you install Python 3.9 from python.org it will symlink /usr/local/bin/python3 to /Library/Frameworks/Python.framework/Versions/3.9/bin/python3, which will be a fat binary with both x86_64 and arm64 code.

I have no trouble building cffi and cryptography locally on an M1 mac, but apparently my environment is somehow unusual.

[bengolds]

Ah, fair point -- they did mention "installing python 3.9 via homebrew", which was why I made the distinction.

I'm still early enough in my setup process that I'm not too worried about blowing away my environment. Can I ask how you set up your environment? Did you install Python 3.9 via homebrew or from python.org? Did you install libffi and openssl from homebrew? Did you hand-compile cffi?