Cryptography: Python 3.7 on Windows now dynamically links OpenSSL

[zooba]

This is not a bug (yet), but wanted to make sure you're aware that I've made a change to Python 3.7 on Windows to dynamically link OpenSSL rather than statically link it. This means that distributions on Windows will include libeay32.dll and ssleay32.dll with all the default exports that come from the normal build.

You can download (or nuget install) a zipped daily build from https://www.myget.org/feed/python/package/nuget/pythondaily with the changes, or the pre-built DLLs we are using come from https://github.com/python/cpython-bin-deps/tree/openssl-bin-1.0.2k

I have no idea how this affects the cryptography package, but I would not be surprised to see new/different conflicts from earlier versions, and also potential opportunities to better integrate cryptography with the built-in _ssl and _hashlib modules.

Officially, it is not supported in Python 3.7 to update OpenSSL by dropping in new builds, but it is likely to work if no public structures have been modified. Again, it seems you currently do not do that, but it is something to be aware of.

Happy to answer any questions that I can about the change. If it's totally irrelevant then feel free just to close this out.

[ofek]

Thanks for the heads up! Out of curiosity, can you please link to a PEP or other discussion describing the rationale behind this change?

[tiran]

@ofek The change brings Windows in line with Linux, BSD, and macOS. All platforms use dynamic builds by default. The macOS builds bring their own OpenSSL shared libraries, too.

[reaperhulk]

@zooba thanks for the heads up. We currently ship static wheels on Windows so unless we've done something funky with our symbol visibility that interacts poorly with the linker (which is possible? I don't know how that works on Windows) we should be fine. I'm going to leave this open until we have a chance to test it though.

[zooba]

The rationale is basically just to reduce our build time. By having a pre-built and dynamically linked binary, our CI system no longer needs to rebuild OpenSSL in its entirety over and over again (we did the same thing for Tcl/Tk, though those were already DLLs). The change to DLLs should be invisible to normal users, but potentially impactful to people who are trying to work around OpenSSL.

I suspect your wheels will continue to work fine, but would like to know sooner rather than later so we can make changes on our side if necessary. And if there are ways that you'd benefit from e.g. using ctypes against the same OpenSSL as ssl (which as Christian points out, should now be available across all OSs), you've got more time to make those work.

[njsmith]

Windows always looks up shared library symbols using a tuple like (library name, symbol identifier). So if you were shipping a DLL named libeay32.dll or ssleay32.dll, then you could get a conflict, but otherwise you should be fine.

[jmcgill298]

I can successfully install on 3.6, but not 3.7, and the issue seems to be locating openssl file(s):

Collecting cryptography
  Using cached cryptography-2.1.4.tar.gz
Requirement already satisfied: idna>=2.1 in c:\users\jacob\venv\py37\lib\site-packages (from cryptography)
Requirement already satisfied: asn1crypto>=0.21.0 in c:\users\jacob\venv\py37\lib\site-packages (from cryptography)
Requirement already satisfied: six>=1.4.1 in c:\users\jacob\venv\py37\lib\site-packages (from cryptography)
Requirement already satisfied: cffi>=1.7 in c:\users\jacob\venv\py37\lib\site-packages (from cryptography)
Requirement already satisfied: pycparser in c:\users\jacob\venv\py37\lib\site-packages (from cffi>=1.7->cryptography)
Building wheels for collected packages: cryptography
  Running setup.py bdist_wheel for cryptography ... error
  Complete output from command c:\users\jacob\venv\py37\scripts\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\Jacob\\AppData\\Local\\Temp\\pip-build-ykewsio3\\cryptography\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\Jacob\AppData\Local\Temp\tmpwyomoziupip-wheel- --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.7
  creating build\lib.win-amd64-3.7\cryptography
  copying src\cryptography\exceptions.py -> build\lib.win-amd64-3.7\cryptography
  copying src\cryptography\fernet.py -> build\lib.win-amd64-3.7\cryptography
  copying src\cryptography\utils.py -> build\lib.win-amd64-3.7\cryptography
  copying src\cryptography\__about__.py -> build\lib.win-amd64-3.7\cryptography
  copying src\cryptography\__init__.py -> build\lib.win-amd64-3.7\cryptography
  creating build\lib.win-amd64-3.7\cryptography\hazmat
  copying src\cryptography\hazmat\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat
  creating build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\base.py -> build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\certificate_transparency.py -> build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\extensions.py -> build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\general_name.py -> build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\name.py -> build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\oid.py -> build\lib.win-amd64-3.7\cryptography\x509
  copying src\cryptography\x509\__init__.py -> build\lib.win-amd64-3.7\cryptography\x509
  creating build\lib.win-amd64-3.7\cryptography\hazmat\backends
  copying src\cryptography\hazmat\backends\interfaces.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends
  copying src\cryptography\hazmat\backends\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends
  creating build\lib.win-amd64-3.7\cryptography\hazmat\bindings
  copying src\cryptography\hazmat\bindings\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\bindings
  creating build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\cmac.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\constant_time.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\hashes.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\hmac.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\keywrap.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\mac.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\padding.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\serialization.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  copying src\cryptography\hazmat\primitives\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives
  creating build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\aead.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\backend.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\ciphers.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\cmac.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\decode_asn1.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\dh.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\dsa.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\ec.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\encode_asn1.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\hashes.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\hmac.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\rsa.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\utils.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\x25519.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\x509.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  copying src\cryptography\hazmat\backends\openssl\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\backends\openssl
  creating build\lib.win-amd64-3.7\cryptography\hazmat\bindings\openssl
  copying src\cryptography\hazmat\bindings\openssl\binding.py -> build\lib.win-amd64-3.7\cryptography\hazmat\bindings\openssl
  copying src\cryptography\hazmat\bindings\openssl\_conditional.py -> build\lib.win-amd64-3.7\cryptography\hazmat\bindings\openssl
  copying src\cryptography\hazmat\bindings\openssl\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\bindings\openssl
  creating build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\dh.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\dsa.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\ec.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\padding.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\rsa.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\utils.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\x25519.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  copying src\cryptography\hazmat\primitives\asymmetric\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\asymmetric
  creating build\lib.win-amd64-3.7\cryptography\hazmat\primitives\ciphers
  copying src\cryptography\hazmat\primitives\ciphers\aead.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\ciphers
  copying src\cryptography\hazmat\primitives\ciphers\algorithms.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\ciphers
  copying src\cryptography\hazmat\primitives\ciphers\base.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\ciphers
  copying src\cryptography\hazmat\primitives\ciphers\modes.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\ciphers
  copying src\cryptography\hazmat\primitives\ciphers\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\ciphers
  creating build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\concatkdf.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\hkdf.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\kbkdf.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\pbkdf2.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\scrypt.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\x963kdf.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  copying src\cryptography\hazmat\primitives\kdf\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\kdf
  creating build\lib.win-amd64-3.7\cryptography\hazmat\primitives\twofactor
  copying src\cryptography\hazmat\primitives\twofactor\hotp.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\twofactor
  copying src\cryptography\hazmat\primitives\twofactor\totp.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\twofactor
  copying src\cryptography\hazmat\primitives\twofactor\utils.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\twofactor
  copying src\cryptography\hazmat\primitives\twofactor\__init__.py -> build\lib.win-amd64-3.7\cryptography\hazmat\primitives\twofactor
  running egg_info
  writing src\cryptography.egg-info\PKG-INFO
  writing dependency_links to src\cryptography.egg-info\dependency_links.txt
  writing requirements to src\cryptography.egg-info\requires.txt
  writing top-level names to src\cryptography.egg-info\top_level.txt
  reading manifest file 'src\cryptography.egg-info\SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'docs\_build'
  warning: no previously-included files matching '*' found under directory 'vectors'
  writing manifest file 'src\cryptography.egg-info\SOURCES.txt'
  running build_ext
  generating cffi module 'build\\temp.win-amd64-3.7\\Release\\_padding.c'
  creating build\temp.win-amd64-3.7
  creating build\temp.win-amd64-3.7\Release
  generating cffi module 'build\\temp.win-amd64-3.7\\Release\\_constant_time.c'
  generating cffi module 'build\\temp.win-amd64-3.7\\Release\\_openssl.c'
  building '_openssl' extension
  creating build\temp.win-amd64-3.7\Release\build
  creating build\temp.win-amd64-3.7\Release\build\temp.win-amd64-3.7
  creating build\temp.win-amd64-3.7\Release\build\temp.win-amd64-3.7\Release
  C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\jacob\appdata\local\programs\python\python37\include -Ic:\users\jacob\appdata\local\programs\python\python37\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\8.1\include\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\winrt" /Tcbuild\temp.win-amd64-3.7\Release\_openssl.c /Fobuild\temp.win-amd64-3.7\Release\build\temp.win-amd64-3.7\Release\_openssl.obj
  _openssl.c
  build\temp.win-amd64-3.7\Release\_openssl.c(483): fatal error C1083: Cannot open include file: 'openssl/opensslv.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit status 2

[alex]

That is almost certainly caused by the fact that for 3.6 you get one of our official wheels that bundles OpenSSL, but on 3.7 you're compiling it yourself, so you need your own OpenSSL.

That's unrelated to this issue.

[jmcgill298]

Do you mean:

1. You don't offer "official wheels" in 3.7 so I have to compile it myself
2. You do offer "official wheels" in 3.7, but I am not using them

If, the latter, then what would I need to do differently with 3.7 than pip install cryptography to use your official wheels?

If the former, then perhaps that should be documented

[reaperhulk]

We don't have wheels uploaded for 3.7 so you'll need to compile it yourself at this time. Our setup.py currently documents support up to 3.6, and we document the versions we test here https://cryptography.io/en/latest/installation/. In general you can assume that we'll upload a wheel for each CPython version shortly after it is officially released.

[jmcgill298]

thanks @reaperhulk

[alex]

Nothing to do here, everything is expected to work ok.

[has-taiar]

I am seeing this problem on python 3.7 and windows 10.
I am trying: pip install cryptography and it keeps failing with the following error:

build\temp.win32-3.7\Release\_openssl.c(493): fatal error C1083: Cannot open include file: 'openssl/opensslv.h': No such file or directory error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Enterprise\\VC\\Tools\\MSVC\\14.14.26428\\bin\\HostX86\\x86\\cl.exe' failed with exit status 2

[reaperhulk]

The issue you're seeing is because we don't ship a Python 3.7 wheel for Windows yet (and won't until the final version is released). It has nothing to do with this issue.

You can compile it yourself, but that is relatively challenging on Windows which is why we'll ship a wheel!

[Mattwmaster58]

@jmcgill298 FWIW, the installation wiki page has easy instructions on how to install on windows. (I just skipped the first of the four commands listed, ie C:\> \path\to\vcvarsall.bat x86_amd64)