Cryptography: Installation on an M1 Mac - Not in Rosetta Mode

[joefedorowicz]

Hello, I'm attempting to install Cryptography but not with terminal on macOS running in Rosetta Mode

• macOS Big Sur 11.2
• running Python3.9.1
• Trying to install cryptography
• cffi==1.14.4
• pip 21.0.1

My Output:

`ERROR: Command errored out with exit status 1:
command: /opt/homebrew/opt/[email protected]/bin/python3.9 /opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/tmpyz87x11u
cwd: /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-req-build-wlcjk9ou
Complete output (55 lines):

    =============================DEBUG ASSISTANCE=============================
    If you are seeing a compilation error please try the following steps to
    successfully install cryptography:
    1) Upgrade to the latest pip and try again. This will fix errors for most
       users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
    2) Read https://cryptography.io/en/latest/installation.html for specific
       instructions for your platform.
    3) Check our frequently asked questions for more information:
       https://cryptography.io/en/latest/faq.html
    =============================DEBUG ASSISTANCE=============================

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
    main()
  File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/opt/homebrew/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 161, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 145, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 44, in <module>
    setup(
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/opt/homebrew/Cellar/[email protected]/3.9.1_7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 432, in __init__
    _Distribution.__init__(self, {
  File "/opt/homebrew/Cellar/[email protected]/3.9.1_7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 708, in finalize_options
    ep(self)
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 715, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
    add_cffi_module(dist, cffi_module)
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
    execfile(build_file_name, mod_vars)
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/cffi/setuptools_ext.py", line 25, in execfile
    exec(code, glob, glob)
  File "src/_cffi_src/build_openssl.py", line 77, in <module>
    ffi = build_ffi_for_binding(
  File "src/_cffi_src/utils.py", line 54, in build_ffi_for_binding
    ffi = build_ffi(
  File "src/_cffi_src/utils.py", line 74, in build_ffi
    ffi = FFI()
  File "/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/cffi/api.py", line 48, in __init__
    import _cffi_backend as backend
ImportError: dlopen(/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure
  Referenced from: /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so
  Expected in: flat namespace
 in /private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so

`

Tried everything I could find on the internet. What am I missing?

[reaperhulk]

This issue keeps cropping up and it appears to be an issue where the Homebrew libffi include path is being passed as part of the compilation args, but the linker is using the system libffi. This results in this error at runtime. You can work around this if you understand how to pass the correct linker paths, but that's not the right long term solution. I'm not sure what (if anything) Homebrew has changed around this, but it will continue to be a problem until the root issue (which is unrelated to cffi or cryptography) is resolved.

[joefedorowicz]

Thanks. I was screwing around with things like this:

sudo LDFLAGS="-L$(mbrew --prefix [email protected])/lib" CFLAGS="-I$(mbrew --prefix [email protected])/include" pip install cryptography

which I saw in some comments...(mbrew is my m1 prefix for home-brew)

that didn't work. What else should I be doing? Where should I look for these paths?

[reaperhulk]

You'll need -L$(brew --prefix libffi)/lib in your LDFLAGS to let the linker find the shared lib matching the libffi header the compiler is finding. I emphasize that this is a workaround for other parts of the stack doing something incorrect, so it's not the sort of thing you want to be doing consistently, but it won't cause problems elsewhere to do this.

[joefedorowicz]

Thanks. Will try it today.

JOE FEDOROWICZ
HARMELIN MEDIA | SR. SYSTEMS ADMINISTRATOR
t: 484.270.9541 c: 215.738.1489

---

From: Paul Kehrer [e-mail]
Sent: Wednesday, February 3, 2021 11:59:50 PM
To: pyca/cryptography [e-mail]
Cc: Joe Fedorowicz [e-mail]; Author [e-mail]
Subject: Re: [pyca/cryptography] Installation on an M1 Mac - Not in Rosetta Mode (#5742)

You'll need -L$(brew --prefix libffi)/lib in your LDFLAGS to let the linker find the shared lib matching the libffi header the compiler is finding. I emphasize that this is a workaround for other parts of the stack doing something incorrect, so it's not the sort of thing you want to be doing consistently, but it won't cause problems elsewhere to do this.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpyca%2Fcryptography%2Fissues%2F5742%23issuecomment-773029236&data=04%7C01%7Cjfedorowicz%40harmelin.com%7Cd9b8efc10d2f419a60c008d8c8c9b4ad%7C19fedd5406f847d89f49e755fd420e7e%7C1%7C0%7C637480115948784652%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=A8b8AAQpiSZ8izIJiUHKwtAsyfohctjhfun2DjbEHXI%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAH3PIMQYKBCFS4ITUCSDYJLS5ISUNANCNFSM4XBRVXVQ&data=04%7C01%7Cjfedorowicz%40harmelin.com%7Cd9b8efc10d2f419a60c008d8c8c9b4ad%7C19fedd5406f847d89f49e755fd420e7e%7C1%7C0%7C637480115948784652%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DpU%2BcUXiTnoquqvnUDesOLi0jgKKUmgdD4WFaAXl2xA%3D&reserved=0.

[joefedorowicz]

So I found a minute to give it a shot and got the same error. That error doesn't include any mention of libffi, but a lot of mention of pep517. Thanks for trying.

[reaperhulk]

Your error is actually

ImportError: dlopen(/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure

This is from cffi. It is a runtime error and not a compile error so it only exhibits when the installation for cryptography triggers it. You need to pip uninstall cffi and then LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:. That should cause it to build a valid version of cffi (although it will not replace the broken copy in your pip cache, so future installs will pull the wrong binary unless you blow away that cache or always use the invocation above).

At that point you can install cryptography with the command you're using. Although, to be clear, sudo pip is always a bad idea and you should install things in a virtual environment.

[bengolds]

I have an M1 machine I can run tests on locally so we _could_ upload arm64 wheels for macOS. However, many users are being stymied by issues at the cffi level before they ever run into compilation problems with cryptography. I wrote up a Stack Overflow answer on resolving that problem, but we're likely to be in this situation for a while longer.

Thank you for taking the time to find a workaround for this! I've given the workaround you posted here (and on Stack Overflow) a try, to no avail. I've deleted the pip cache and reinstalled cffi using the commands you posted, and the same dogged error keeps showing up:

ImportError: dlopen(/private/var/folders/mw/r8p01knj6lgg53x2x0h7yvvw0000gn/T/pip-build-env-7uviy6hh/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure

If there's any other info I can post to be of help, let me know. Otherwise, if the solution is simply to wait until cffi is fixed, I'll do that!

[reaperhulk]

@bengolds it's definitely still linking against the wrong libffi in your case. Can you post the exact commands you tried? I would expect this to work:

python -m venv venv
LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include venv/bin/pip install cffi --no-binary :all:
LDFLAGS="-L$(brew --prefix [email protected])/lib" CFLAGS="-I$(brew --prefix [email protected])/include" venv/bin/pip install cryptography

(Note that I'm being lazy about activating the venv and just directly invoking it. Also note that as of 3.4 you'll need to brew install rust or have a sufficiently new version of the rust compiler in your PATH for cryptography to successfully compile)

[bengolds]

I tried those exact commands in a separate folder, to no avail -- same error as before. I've got Rust 1.4.9 installed through Homebrew.

The commands I was running before were the ones from your Stack Overflow comment; seems like it's the same commands as you just posted, with the exception of the virtual environment.

In both cases, I'm using the Homebrew, arm64 version of Python 3.9.1. I've also tried reinstalling that version; no luck.

[bschollnick]

Okay. I just realized that not only is one pre-req openssl, but rust as well. I'm now able to install Cryptography in M1 Native...

[Aloisius]

I'm not seeing the libffi issue after installing homebrew 3.0.0 on a clean m1. brew install python rust and I was able to pip3 install cryptography without issue. The [email protected] package uses macos' libffi and the libffi package is Cellar-only. The only way I can imagine having libffi's headers included is maybe if the the llvm clang package is installed and is being used for some reason instead of xcode's clang.

[bengolds]

I have it working now! I reinstalled homebrew (removed the /opt/homebrew folder). From there:

brew install python rust openssl
pip3 install --upgrade pip
env LDFLAGS="-L$(brew --prefix [email protected])/lib" CFLAGS="-I$(brew --prefix [email protected])/include" pip3 install cryptography

Simply running pip3 install cryptography didn't go so hot, but the build instructions in the FAQ worked. Thanks, all!

[Aloisius]

Simply running pip3 install cryptography didn't go so hot

Hmm. It really should have. Homebrew patches its python 3.9 package so that ctypes explicitly knows where [email protected] is since it python itself depends on openssl 1.1:

$ python3 -c "import ctypes.util; print(ctypes.util.find_library('ssl'))"                         
/opt/homebrew/opt/[email protected]/lib/libssl.dylib

$ python3 -c "import ctypes.macholib.dyld; print(ctypes.macholib.dyld.DEFAULT_LIBRARY_FALLBACK)"
['/opt/homebrew/lib', '/opt/homebrew/opt/[email protected]/lib', '/Users/me/lib', '/usr/local/lib', '/lib', '/usr/lib']

$ which python3
/opt/homebrew/bin/python3

$ python3 -V
Python 3.9.1

cffi uses ctypes.util.find_library to find the library to get symbols from.

For people who are having problems with libffi on M1 macs, if /usr/local/bin/brew exists from an old x86 homebrew installation, cffi will add $(brew --prefix libffi)/lib/pkgconfig to the pkg-config search list allowing it to pick up a homebrew installed libffi (even in the arm installs). It does not do this if you only have the arm version installed in /opt/homebrew.

By default, homebrew installs a libffi.pc file for pkg-config, but it points to the system libffi:

$ pkg-config libffi --variable=pcfiledir                   
/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11

$ pkg-config libffi --cflags
-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ffi

$ pkg-config --variable pc_path pkg-config
/opt/homebrew/lib/pkgconfig:/opt/homebrew/share/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib/pkgconfig:/opt/homebrew/Library/Homebrew/os/mac/pkgconfig/11

Brew isn't great at uninstalling itself, so if you had an old x86 install in /usr/local, be sure all the directories are completely empty - especially /usr/local/lib/pkgconfig, /usr/local/bin, /usr/local/include, /usr/local/lib and /usr/local/Frameworks.

[dvcrn]

I have both installations, one in /opt/homebrew for arm64 stuff, and one in /usr/local for x86_64 stuff that doesn't work properly on m1 yet.

None of the workarounds in this thread worked for me, any other ideas?

[judescripts]

Your error is actually

ImportError: dlopen(/private/var/folders/hj/5zfkv68d7lqgrfqt046bn23c0000gn/T/pip-build-env-e5wrkudv/overlay/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure

This is from cffi. It is a runtime error and not a compile error so it only exhibits when the installation for cryptography triggers it. You need to pip uninstall cffi and then LDFLAGS=-L$(brew --prefix libffi)/lib CFLAGS=-I$(brew --prefix libffi)/include pip install cffi --no-binary :all:. That should cause it to build a valid version of cffi (although it will not replace the broken copy in your pip cache, so future installs will pull the wrong binary unless you blow away that cache or always use the invocation above).

At that point you can install cryptography with the command you're using. Although, to be clear, sudo pip is _always_ a bad idea and you should install things in a virtual environment.

This is golden, thank you

[reaperhulk]

We've uploaded a cryptography wheel for arm64 and will do so on a best effort basis going forward. Since we have no M1 in our CI this process may not always be smooth, but for now 3.4.6 is covered.

Those seeing Symbol not found errors are having problems with cffi and should follow the instructions above to resolve that before installing cryptography itself.

[joefedorowicz]

Awesome thanks

JOE FEDOROWICZ
HARMELIN MEDIA | SR. SYSTEMS ADMINISTRATOR
t: 484.270.9541 c: 215.738.1489

---

From: Paul Kehrer [e-mail]
Sent: Sunday, February 21, 2021 12:18:38 PM
To: pyca/cryptography [e-mail]
Cc: Joe Fedorowicz [e-mail]; Author [e-mail]
Subject: Re: [pyca/cryptography] Installation on an M1 Mac - Not in Rosetta Mode (#5742)

We've uploaded a cryptography wheel for arm64 and will do so on a best effort basis going forward. Since we have no M1 in our CI this process may not always be smooth, but for now 3.4.6 is covered.

Those seeing Symbol not found errors are having problems with cffi and should follow the instructions above to resolve that before installing cryptography itself.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpyca%2Fcryptography%2Fissues%2F5742%23issuecomment-782891802&data=04%7C01%7Cjfedorowicz%40harmelin.com%7C4a78c1bced354ef272ae08d8d68cbb2c%7C19fedd5406f847d89f49e755fd420e7e%7C1%7C0%7C637495247233717461%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pMIUqY1tDzIHUtyx0veoAtoPwdF4ewTiGQGj%2F7hqgi0%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAH3PIMT2KTXR6NEMTTUJCJDTAE565ANCNFSM4XBRVXVQ&data=04%7C01%7Cjfedorowicz%40harmelin.com%7C4a78c1bced354ef272ae08d8d68cbb2c%7C19fedd5406f847d89f49e755fd420e7e%7C1%7C0%7C637495247233727457%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AGt2G1GnYHo0rEJto6DyNlIrWd9z4H5g2FVyOGBPpm4%3D&reserved=0.