Cryptography: Dependencies to build on Alpine Linux?

[fourstepper]

As per the docs, I added cargo as a dependency in Alpine Linux, but the build is still failing:
https://gist.github.com/fourstepper/27db0fc4247ad59970a84e12ac70a938

here are the packages that get installed:
cargo python3-dev py3-pip gcc git curl build-base autoconf automake py3-cryptography linux-headers musl-dev libffi-dev openssl-dev openssh py-virtualenv

Am I missing something?

Didn't wanna populate the discussion here: https://github.com/pyca/cryptography/issues/5771 which is why this is a separate issue

[tiran]

You are using an old version of Alpine with an outdated Rust compiler. Please install latest Alpine with Rust 1.45 or newer.

[fourstepper]

Alpine 3.13 is a month old release though and 3.12 is still supported (and will be for over a year)

Running this all in containers is all chill and fine (like I do) but I can image this being quite problematic for people who run Alpine on virtual or physical servers. Have you considered introducing the change later to stay compatible with Alpine 3.12 until let's say Alpine 3.14 is out?

[tiran]

It's a chicken and egg problem. For a long time there will be platforms with too old Rust or pip versions. For example the Rust toolchain in CentOS 7 and RHEL 7 SCLs is too old, too.

Rust bindings are still optional. You can still disable Rust dependency by setting the environment variable CRYPTOGRAPHY_DONT_BUILD_RUST=1.

@alex @reaperhulk could you please add this error message to FAQ and explain that Rust >= 1.45 is required? It might also be a good idea to mention Rust >= 1.45, Cargo, and CRYPTOGRAPHY_DONT_BUILD_RUST=1 in the debug assistance message in setup.py. Linux distros package rustc and cargo separately. Users may end up with rustc but without cargo.

  error[E0658]: use of unstable library feature 'str_strip': newly added
     --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-macros-backend-0.13.1/src/method.rs:144:18
      |
  144 |                 .strip_prefix(prefix)
      |                  ^^^^^^^^^^^^
      |
      = note: see issue #67302 <https://github.com/rust-lang/rust/issues/67302> for more information

[fourstepper]

@tiran CentOS 8 and RHEL 8 is out since 2019. Alpine 3.13 is out since last month... I think there is a difference, even though I understand your statement.

Where should I ideally watch for a heads-up when CRYPTOGRAPHY_DONT_BUILD_RUST=1 will stop working?

[tiran]

For what it's worth, RHEL 8.2 won't be able to compile the project either. RHEL 8.2.1 has rust 1.43. Only RHEL 8.3 and latest CentOS have recent Rust.

The difference between Alpine and other major Linux distributions (Arch, CentOS, Debian, Fedora, Gentoo, RHEL, SUSE, Ubuntu, ...) is its libc. The majority of Linux distributions is using glibc, which is natively supported by manylinux binary wheels. Alpine uses musl libc. It's not ABI compatible to glibc and not supported by standard binary Python packages.

AFAIK nobody has volunteered to define a manymusl standard PEP for binary wheels that are compatible with Alpine.

[fourstepper]

It seems that Debian 10.8 is also incompatible due to the Rust version...? Or am I wrong?

I know that there is a difference in the C library used in Alpine and frankly I haven't done any work in this area so I don't feel entitled to complain.

[tiran]

It seems that Debian 10.8 is also incompatible due to the Rust version...? Or am I wrong?

Rust on Debian's current stable is too old, too. Debian testing and upcoming stable have are fine.

A recent pip version will automatically pick the binary wheels. The pip package on Debian might be too old to recognize abi3 manylinux wheels.

[alex]

Yes, we will look for ways to improve our documentation -- we recommend following upstream Rust's installation docs: https://www.rust-lang.org/tools/install

[manuelmarquezs]

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. The build of the cryptography package that used to take a few seconds now takes minutes.

We have updated alpine, installed all the dependencies listed in the docs: sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo and we are using Rust>1.45.0.

Any idea on why the compilation with Rust is that slow??
How could we speed it up??

Building wheels for collected packages: cryptography
  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): finished with status 'done'

[reaperhulk]

It is definitely slower to build, but needing to build it every CI run (when you're not us!) isn't ideal. I'd suggest caching the wheel it creates and installing from that since that will work between runs. Ideally we'd be able to upload alpine compatible wheels, but that will require work upstream with pip/wheel/packaging alongside a PEP (a discussion of which @webknjaz just started here: https://discuss.python.org/t/wheels-for-musl-alpine/7084)

We've also updated our docs to more clearly call out the need to get a newer Rust compiler if you are on anything less than the latest Alpine: https://cryptography.io/en/latest/installation.html#alpine

I'll close this for now since these deps are now reasonably well documented, but if people have additional suggested improvements please feel free to open another issue or submit a PR!

[rjnienaber]

I'm using a Docker image that's based on Alpine Linux 3.10.3 and upgrading wasn't an option. Adding the following to my Dockerfile gets cryptography to build and install:

ENV PATH="/root/.cargo/bin:${PATH}"

RUN curl https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-musl/rustup-init --output /tmp/rustup-init \
    && chmod +x /tmp/rustup-init \
    && /tmp/rustup-init -y \
    && pip install --no-cache-dir cryptography \
    && rustup self uninstall -y

[garloff]

curl https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-musl/rustup-init --output /tmp/rustup-init

Downloading and running a 12MB x86_64 binary (even via https) does not look like the ultimate portable and secure option ...

[tiran]

Downloading and running a 12MB x86_64 binary (even via https) does not look like the ultimate portable and secure option ...

I agree. Better options are:

• Update base image to Alpine 3.12 or newer
• finalize and implement manymusl wheel standard, https://discuss.python.org/t/wheels-for-musl-alpine/7084

Release 3.4.0 required 3.13. Since then MSRV was lowered to Rust >= 1.41. Starting with release 3.4.5, Alpine 3.12 is sufficient to build cryptography.

[wayne-li2]

Quick question for my own understanding:

Once https://www.python.org/dev/peps/pep-0656/ is implemented, cryptography maintainers can ship a binary wheel, and after that, end-users no longer need a rust compiler in order to use this package?

[alex]

That's correct.

On Tue, Jun 1, 2021 at 6:23 PM Wayne Li @.*> wrote:

Quick question for my own understanding:

Once https://www.python.org/dev/peps/pep-0656/ is implemented,
cryptography maintainers can ship a binary wheel, and after that, end-users
no longer need a rust compiler in order to use this package?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pyca/cryptography/issues/5776#issuecomment-852520543,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAAGBBUGPAMK4AULGYNO6LTQVMUPANCNFSM4XI3HHFQ
.

--
All that is necessary for evil to succeed is for good people to do nothing.

[reaperhulk]

One additional requirement: a new pip will need to be released that understands the new tag. (If the tag has already been determined then that pip work could start now and I encourage any Alpine users to help out there!)