Cryptography: Add dlopen-based OpenSSL bindings

[jsha]

Right now in order to install pycryptography, you need a full build chain installed on a machine. That in turn means that any dependencies, like certbot, cannot be installed without a full build chain. In practice that also means it can be hard to deploy in limited memory situations like RasPi's or very small (128-256MB) VPS instances, because the compilation steps run out of memory.

It would be very convenient to have an option to install a pycryptography library that doesn't have any build dependencies and dlopen's the OpenSSL library to access the necessary functions.

[jsha]

/cc @bmw

[reaperhulk]

To do this while retaining full compatibility with our existing bindings doesn't appear feasible due to the extensive use of macros within OpenSSL. Since those macros aren't real symbols we can't dlopen and call them.

For the case of x86 (and x86_64) machines we may start shipping a manylinux1 wheel soon, which would allow users on linux with those arches to download a precompiled wheel. This would potentially solve the issue in the most common case, but unfortunately won't help for alternate architectures like ARM or PPC.

It would be possible to implement a dlopen-based backend that performs some subset of cryptography's actions (how big that subset is would require research), but shipping that would require #1522 and #2214 to be resolved.

[jsha]

That makes sense. FWIW, these are the imports that certbot currently uses (spread across various files):

from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import hmac
from cryptography.hazmat.primitives import serialization
import cryptography.exceptions

[bmw]

Adding to that, Certbot also uses pyopenssl, primarily for:

• Generating RSA keys
• TLS handshake
• Serializing, parsing, creating, loading, saving, and coverting certs and CSRs

If you'd like more information on exactly what subset of features Certbot is using, please let me know what you need and I'm happy to write it up.

While we certainly would be interested in a dlopen approach, a manylinux1 wheel should provide some mitigation to the problem. One of the more common places we get compilation problems is people using a small VPS instance without swap and a number of services running. The issue about this on our repo is certbot/certbot#1081.

[alex]

I think manylinux1 is much more likely to be a short term solution to
this problem; a significant portion of the OpenSSL public API is
macro-based.

On Mon, Jun 20, 2016 at 8:06 PM, Brad Warren [email protected]
wrote:

Adding to that, the Certbot also uses pyopenssl, primarily for:

• Generating RSA keys
• TLS handshake
• Serializing, parsing, creating, loading, saving, and coverting certs
  and CSRs

If you'd like more information on exactly what subset of features Certbot
is using, please let me know what you need and I'm happy to write it up.

While we certainly would be interested in a dlopen approach, a manylinux1
wheel should provide some mitigation to the problem. One of the more common
places we get compilation problems is people using a small VPS instance
without swap and a number of services running. The issue about this on our
repo is certbot/certbot#1081
https://github.com/certbot/certbot/issues/1081.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pyca/cryptography/issues/2980#issuecomment-227305798,
or mute the thread
https://github.com/notifications/unsubscribe/AAADBGCD-i6PhfiqQt9ddJwzc1FbAtyZks5qNysPgaJpZM4I408Q
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[bmw]

Okay great. After talking about this with the rest of Certbot team, we definitely would be interested in the manylinux1 wheel. The largest group having compilation problems are small VPS instances and the wheel should resolve the issue for them.

[bmw]

Any update on this? A manylinux1 wheel would significantly benefit Certbot as it eliminates many of our non-Python dependencies, problems with OpenSSL changing out from underneath us, users running out of memory while compiling, and problems with people who have a misconfigured version of OpenSSL.

In general, I think it'd be a real benefit to any client side Python program that uses cryptography with little control of the environment the program is run in.

[reaperhulk]

There is an update! https://github.com/pyca/cryptography/issues/2970 is where we're tracking manylinux1, but essentially the goal is to ship a manylinux1 wheel with the next release (pending sorting out some pyopenssl questions).

[reaperhulk]

For certbot this will solve the problem on the manylinux1 compatible distributions, but be aware that alpine linux is not compatible due to their use of musl libc.

[reaperhulk]

The manylinux1 wheel shipped with cryptography 2.0 a few days ago!

[bmw]

I saw! That's great news. We'll definitely be making changes in Certbot to take advantage of this.

[alex]

Now that we've got linux wheels, I'm going to close this; dlopen based bindings don't appear to be happening.

If anyone disagrees, please speak up.