Cryptography: Manylinux Wheel

[mhils]

As per #2913, it seems like the Cryptography_locking_cb() error on Python 3 with OpenSSL < 1.1.0 won't/can't be fixed reasonably (correct me if I'm wrong). While shipping precompiled wheels with OpenSSL 1.1.0 solves the issue on Windows and OSX, cryptography on Linux currently uses the system OpenSSL, which is usually older than that (e.g., Ubuntu 16.04 LTS ships with 1.0.1).

We are currently porting @mitmproxy to Python 3 with the intention to drop Python 2 support ASAP. (Un)fortunately we have a lot of Linux users, and the Cryptography_locking_cb() error kind of holds us back.

A great way to mitigate this would be to ship precompiled manylinux wheels, which would also allow us to support ALPN on more systems. I'd love to hear if there's anything that holds this back or if this is something we can hope for in the near future. I'd be happy to offer the mitmproxy userbase as a testbed or help with anything else, that being said I usually don't compile wheels for breakfast :wink: .

Thanks!
Max

[reaperhulk]

So there are two ways to fix the locking callback problem.

1) Compile against OpenSSL 1.1.0, which registers its own locking handlers (hooray)
2) Switch back to the older callback model in cffi. The drawback is that this doesn't play nice with SELinux, although I vaguely recall that a tweak to the way libffi did things could bypass that problem. It's possible that change made it into cffi as well. @tiran do you happen to know where that stands?

Shipping a manylinux1 wheel would give us a solution in the form of number 1, but does mean that Linux users on pip 8.1+ will no longer use system OpenSSL by default. The old behavior would only be obtainable by doing pip install cryptography --no-binary :all. Before we can ship a wheel like that we need OpenSSL 1.1.0 to actually be released, we need to ship a cffi manylinux1 wheel (we'll need to talk to the PyPy team about that), and I'd like to send an email to the cryptography-dev mailing list asking for feedback on the potential switch.

[alex]

@reaperhulk Technically manylinux1 wheels for cffi and cryptography are seperable. If we do pypy only, you still need a compiler and libffi-dev headers, but you don't need any of the OpenSSL stuff.

[mhils]

FWIW, this now breaks our Python 3 tests as the sheer amount of Cryptography_locking_cb error messages make the error log exceed 4MB, at which point travis fails ¯_(ツ)_/¯.
@Kriechi came up with this very temporary band-aid:

set -o pipefail; tox 2>&1 | grep -v Cryptography_locking_cb

Needless to say, it'd be great if we could help make this work again somehow. :blush:

[reaperhulk]

I think we probably need to revert the static callbacks and just kill this bug (at the cost of some nebulous selinux issues). Opinion @alex?

[alex]

@reaperhulk Can you remind me, are there particular platforms on which this locking_cb issues occurs?

[reaperhulk]

It appears to be Python 3.5 on any platform

[alex]

Can we disable static callbacks only on 3.5+ then? This gives us an SELinux friendly thing on other platforms (claim: people using SELinux are less likely to be using 3.5)

[reaperhulk]

We probably could disable static callbacks just on 3.5. We have the older codepath for old PyPy still, would just need to have 3.5 register that way instead of using def_extern.

Also, to keep this issue up to date, I built a manylinux1 builder for cryptography: https://github.com/reaperhulk/cryptography-manylinux1

Right now it will not be capable of finding trust roots for TLS by default though, so we need to resolve that issue before we can consider shipping it. njsmith has a proposal for how we might solve that (https://github.com/pypa/manylinux/issues/74#issuecomment-229823327), although it remains to be seen if we can truly reliably detect distro pathing from Python.

[mhils]

We probably could disable static callbacks just on 3.5. We have the older codepath for old PyPy still, would just need to have 3.5 register that way instead of using def_extern.

Would that just mean special-casing Python 3.5 here or is there more to that?

[alex]

Just there I think.

On Tue, Jul 26, 2016 at 8:05 PM, Maximilian Hils [email protected]
wrote:

We probably could disable static callbacks just on 3.5. We have the older
codepath for old PyPy still, would just need to have 3.5 register that way
instead of using def_extern.

Would that just mean special-casing Python 3.5 here
https://github.com/pyca/cryptography/blob/1a6628e55126ec1c98c98a46c04f777f77eff934/src/_cffi_src/openssl/callbacks.py#L47-L50
or is there more to that?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pyca/cryptography/issues/2970#issuecomment-235443417,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAADBGxYbdAt0Q4Ik39BoY6wk2mnoFu9ks5qZqC_gaJpZM4IvGKv
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[reaperhulk]

The specific issue mentioned in this issue (locking callbacks) are no longer a problem, but leaving this open as a generic manylinux1 tracking issue.

[dstufft]

What is needed to get cryptography shipping manylinux1 wheels? How can I help to make this a reality?

[alex]

How do you feel about fixing trust stores on Linux? ;-) (Not sure what the
more specific technical task is here)

On Tue, Dec 27, 2016 at 2:40 PM, Donald Stufft [e-mail]
wrote:

What is needed to get cryptography shipping manylinux1 wheels? How can I
help to make this a reality?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pyca/cryptography/issues/2970#issuecomment-269372995,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAADBK7XmzaqlHq2qTY-MjeDVaZQ0NBLks5rMWm7gaJpZM4IvGKv
.

--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[reaperhulk]

We need to confirm that there's a way to set verify paths at runtime and then we need a mechanism to detect the location of the trust roots for (your manylinux1 compatible linux distro here). This can fail closed, but needs to work for pretty much every distro that actually gets used or else we'll constantly be dealing with a stream of bug reports.

[stardust85]

Maybe you can find this article useful http://kb.kerio.com/product/kerio-connect/server-configuration/ssl-certificates/adding-trusted-root-certificates-to-the-server-1605.html

[OptiverTimAll]

The requests library already ignores the system trust store on Linux (and, I think, everywhere), so I think this battle has already been lost. If cryptography were to default to CA roots from certifi and respect $REQUESTS_CA_BUNDLE, then it it won't cause any new problems that people haven't already learned to work around.

[collisdigital]

quick heads-up; we're still having callback issues with Apache with the latest fixes; not with os_random but pem callbacks; will provide more details soon (https://github.com/ONSdigital/eq-survey-runner/issues/895)

[reaperhulk]

@collisdigital Could you open that as a separate issue when you have more details? Seems like we need to dig into this a bit more to solve it for the final callback.

[ajmaddaford]

@reaperhulk I've raised https://github.com/pyca/cryptography/issues/3377 for the issue that @collisdigital mentioned above

[ofek]

Is just using certifi an issue?

[alex]

It'd be a pretty significant compatibility break, which we shouldn't take lightly.

[reaperhulk]

At PyCon I believe we generally agreed that we want to do a manylinux1 wheel and that to maintain expected behavior we'll need to patch pyOpenSSL to set the certificate path whenever an SSL_CTX is instantiated (or whichever object it is). However, we need a released version of pyOpenSSL that does this before we can start shipping cryptography wheels.

[reaperhulk]

https://github.com/pyca/pyopenssl/issues/632

[ofek]

This is great news! If it helps to take a look at an existing project, see https://github.com/ofek/coincurve. I ship 64 & 32 bit manylinux1 wheels. In particular, check out https://github.com/ofek/coincurve/blob/master/.travis/deploy.sh and https://github.com/ofek/coincurve/blob/master/.travis/build-linux-wheels.sh.

[reaperhulk]

We have a bit of familiarity with manylinux1 :)

https://github.com/reaperhulk/cffi-manylinux1
https://github.com/reaperhulk/argon2cffi-manylinux1
https://github.com/reaperhulk/cryptography-manylinux1
https://github.com/pyca/bcrypt/tree/master/wheel-scripts

Thanks though!

[ofek]

Ahh, nice :)

[reaperhulk]

As an update, here's where we're at:

https://github.com/pyca/pyopenssl/pull/633 is in review for pyOpenSSL.
https://github.com/pyca/infra/pull/98 is a PR to build the docker container we'll need for compiling manylinux1 wheels

After those merge we have a few more steps:

• pyOpenSSL 17.1.0 release
• A PR to update the cryptography-wheel-builder to add manylinux1 wheel building.
• A PR to update the installation docs of cryptography to document the scenarios under which a manylinux1 wheel will be used and how to not get it if you don't want it (--no-binary cryptography).
• A PR to add links and information about manylinux1 to the changelog.
• cryptography 2.0 release with manylinux1 wheels!

[reaperhulk]

All these steps are now complete other than the 2.0 release itself. I'm going to go ahead and close this issue. 2.0 will ship a wheel (with OpenSSL 1.1.0f unless there is a 1.1.0g before 2.0 is released!)