Cryptography: Dependency on rust removes support for a number of platforms

[mgorny]

I would like to report that the newly added dependency on Rust has made it impossible to package cryptography for a number of supported Gentoo architectures (and these are architectures where people actually use Python packages that depend on cryptography).

Please see Platform Support. Besides degrading a few of our targets to 'Tier 3', Rust completely does not support at least alpha, hppa, ia64, m68k, s390 (not -x). The cost of adjusting cryptography's C code to these platforms (even if we assumed they wouldn't work out of the box as they do so far) is much less than the cost of porting the whole Rust. You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

Right now it's a really hard struggle to even get Rust packaged for non-Tier 1 architectures (i.e. where upstream does not provide bootstrap binaries).

[LindezaGrey]

destroyed my CI/CD Pipeline this morning :laughing:

[clanzett]

It is also impossible to build on ANY alpine linux version/architecture. Please remove this dependency as soon as possible.

[tiran]

It is also impossible to build on ANY alpine linux version/architecture.

Update to most recent Alpine with Rust >= 1.45. Alpine latest has rust (1.47.0-r2).

Please remove this dependency as soon as possible.

Not going to happen. Rust dependency will stay to replace C code with an actual safe language.

You can disable Rust integration in 3.4.x (see FAQ). Starting with 3.5 cryptography will have mandatory Rust code.

[clanzett]

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...
But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

[mgorny]

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

[tiran]

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.

[silverfisk]

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

[clanzett]

Memory safety is of course a compelling reason, but you should also keep in mind that may developers out there are using your package, which is great by the way, and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on. A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

[clanzett]

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

[Smirl]

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time. From the changelog:

BACKWARDS INCOMPATIBLE: Support for Python 2 has been removed.
We now ship manylinux2014 wheels and no longer ship manylinux1 wheels. Users should upgrade to the latest pip to ensure this doesn’t cause issues downloading wheels on their platform.
cryptography now incorporates Rust code. Users building cryptography themselves will need to have the Rust toolchain installed. Users who use an officially produced wheel will not need to make any changes. The minimum supported Rust version is 1.45.0.
cryptography now has PEP 484 type hints on nearly all of of its public APIs. Users can begin using them to type check their code with mypy.

This should probably be a 4.0 release it would cause a lot less pain for the community. I suppose many have cryptography>=3.0,<4 in setup.py or requirements.txt.

Update
I was unaware that you don't use semantic versioning. I have found in the docs https://cryptography.io/en/latest/api-stability.html#versioning. However it was a little confusing given the style. As many of our dependencies require this we will try not to pin a version and install rust/use manywheel packages.

[tiran]

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

PS: I've been preparing Fedora packaging for almost three months. My Fedora Rawhide packages of python-cryptography 3.4 landed about 12 hours after the release -- and I had to jump through several extra hoops because Fedora does not permit vendoring of Rust crates.

[mgorny]

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

[tiran]

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

[public]

@tiran

I wonder if there's a way to produce a more helpful error message, perhaps with a link to some documentation about the new required build deps. The current message most people will hit is

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed.
      =============================DEBUG ASSISTANCE=============================

Which does sort of attempt to guide people the right way but could be more specific for common cases like Alpine.

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

[mgorny]

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

That's a very roundabout way of showing the middle finger to users of 'other hardware platforms' that are not supported by Rust (or LLVM) at all.

[fourstepper]

CI testing went down the drain with this in GitLab CI with docker:stable-dind, as it's based off of Alpine. I can imagine this happening all over the place.

[tiran]

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

The documentation for Alpine builds has been improved https://github.com/pyca/cryptography/pull/5763 already. I have a pending PR for Fedora, CentOS, and Debian at https://github.com/pyca/cryptography/pull/5764 . If you have any suggestions for better build instructions, please help by opening a PR.

[thesamesam]

For myself, I'd like to note that a heads-up in the CHANGELOG / Release Notes (I may have missed it) would've helped:
1) when considering the move ("we're considering doing this, please give feedback")
2) when it was decided it should actually happen ("release X will optionally contain Rust, release Y will have mandatory Rust")

I don't see any mention before the release in the CHANGELOG or in the emails to python-announce-list.

That's the one thing which distro maintainers or packagers really should be reading on every new release. Including it as a warning in previous releases would've helped us out here to both object/raise concerns and begin preparations.

[cdesousa]

To be fair the developers did ask for feedback (in a not very inviting way):

_We are interested in feedback in things we can do to make this a smooth process, particularly from folks who are re-distributing cryptography. Please note: feedback of the form “don’t use Rust” is not productive and will be disregarded_

I highly suspect there will be many more complaints throughout the day about stuff breaking. I had stuff breaking on amazon linux and was confused as to why pip install ansible was asking for rust, seems that one is tagged to latest cryptography in requirements.txt (which is of course not directly the fault of the devs here).

[MrMino]

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about first impressions :smile:. That's a good reminder to update the alpine images though.

Hands up: who here watches every update of every transitive dependency in each app/environment they have to maintain?

BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?

[clanzett]

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.
I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about reminders to update to the newest alpine 😄.

I totally agree. I really think we did not even scratch the tip of the iceberg. There might be some "interesting" problems ahead.

And don't get me wrong @tiran, I also agree with you regarding your security concerns, but this switch could have really done some serious damage. We will see...

[xgqt]

Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

other hardware platforms

lol

[ambv]

It sounds like y'all are getting an unexpected lesson about the flip-side of open source software. I would like to push back on some of the entitlement I'm sensing reading in this thread.

Before I begin, I'd like to remind you that security is a numbers game. If the cryptography maintainers can help 90% of their users by switching to a modern memory-safe language, then they'd be irresponsible holding back just because among the remaining 10% there exist fringe platforms which can't even run a Rust compiler.

You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer is worth more than the time of cryptography maintainers. They are volunteers as well, y' know. You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.

In fact, your only real recourse long-term is exactly to band up volunteers to make Rust work on your platform. Or pay some hired guns to do the initial work for you. After all, you will be seeing more and more projects depending on this programming language as time goes on. Yelling at creators of those projects for choosing a language that is inconvenient for you is both ineffective and laughable.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

I invite you to do that. Please put your money and time where your mouth is. Report back in a year's time how it went.

[xgqt]

Let's depend on Rust because everything does!
Let's depend on glibc because everything does!
Let's depend on GNU coreutils because everything does!
Let's depend on systemd because everything does!
Let's depend on Linux because everything does!

Afternote: should be "HARD depend" - without the ability to toggle building with the given component.

[manuelmarquezs]

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. The build of the cryptography package that used to take a few seconds now takes minutes.

We have updated alpine, installed all the dependencies listed in the docs: sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo and we are using Rust>1.45.0.

Any idea on why the compilation with Rust is that slow??
How could we speed it up??

Building wheels for collected packages: cryptography
  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): finished with status 'done'

[wvxvw]

It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.

cryptography is a transitive dependency in our code: through jira, which requires oauthlib. Of course, in part, the blame is on pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version. But you, as a maintainer, also did a very bad job at signaling this to pip.

If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.

I'd be totally fine with there being a cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.

[felixfontein]

@wvxvw creating a new package would actually make the situation a lot worse. Except if the Python module name is also renamed, which would require everyone to adjust there code. Which would also be worse IMO.

[tiran]

It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.

Ubuntu Bionic LTS has recent Rust and Cargo version. pip is the problem here. Ubuntu 18.04 comes with an old, unsupported pip version. I recommend that you install a newer pip version in a virtual env. It might also be worth the trouble to file an update request with your vendor.

# echo $(source /etc/os-release; echo $VERSION)
18.04.5 LTS (Bionic Beaver)
# apt update
# apt install -y rustc python3-pip
# rustc -V
rustc 1.47.0
# pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

[MrMino]

@ambv I'd agree with everything you've said, if it wasn't for this line:

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

With all due respect, it seems to me like this is a strawman. We're not talking about a security update here, are we?

I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.

Currently the lockfile in one of repositories I maintain shows 354 dependencies. The main dependency list includes around 50. The app that is broken for me is a staging environment for a distributed hardware testing platform. This is as far from security concerns as you could get, there's no way I'm able to inspect every single transitive dependency for updates, and there's even less probability that management permits me to spend my time on it.

Please have some empathy. I'm really not blaming any cryptography maintainer, I don't have the time and energy to - I'm fighting an NP-complete problem and this update is making my life significantly harder currently.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer

We have been able to fix our alpine Pipelines [...] but they are now extremely slow. We have gone from 30s to 4min

It's really not just "some other platforms". Ubuntu 18 is also affected...

Also, now It's not _just_ about Gentoo, is it? Don't you think the 90% figure is a bit exaggerated?

[wvxvw]

@felixfontein absolutely false... I'd be able to live forever with cryptography instead of cryptography2. And, in case I really wanted the update, I'd be able to update. pip integration is the major problem here, as I just wrote. There's no good way to tell pip that new version of the package is incompatible, beside renaming the package. I believe that the change made to cryptography warrants such a move.

@tiran Sorry, not going to happen. I'm neither updating pip, nor am I installing Rust in those images. Our product is also written in Rust, and your dependency on Rust is completely unnecessary for us to consider in our CI. pip is the version it is for a lot of reasons too.

For me, a solution that would involve embedding the entire jira package with its transitive dependencies in our own deployment would be easier than trying to deal with your demands to our CI environment...

[felixfontein]

@wvxvw have you ever dealt with cases where pip packages were renamed, but the corresponding Python modules weren't? If you didn't, believe me: it's not fun.

[MrNaif2018]

@tiran Just my two cents here.
I would say, that any improvement that reduces the risk of security vulnerabilities is a good improvement. If rust really improves the memory management and helps avoids some future vulnerabilities-go for it. But there are only two things that concern me:

• Is rust itself stable and secure enough? May there be that there is/will be an vulnerability in rust, which will be impossible to fix inside the library code itself?
• And the main thing. There is nothing wrong with improving the code or making backwards-incompatible changes. But I think that what should have been done is a reverse of what it is done.
  Maybe instead of making rust required in this release, and making it optional by setting an env variable, it could be made optional in this release, that could be enabled by an env variable, and a notice to the changelog could be added:

 "Added an improved version of the library using Rust. Please help us test that by setting `CRYPTOGRAPHY_USE_RUST=1` at the installation step. **It will be required in the future release.**"
That way, nothing would break, but everyone would have been warned and way more people could adapt their workflows and help test the new implementation.

Of course, probably nothing can be done right now after the release is done. Or maybe could it be implemented that way?

I am also using cryptography as a non-direct dependency in docker image builds (alpine linux, via electrum)

[wvxvw]

@felixfontein correct me if I'm wrong, concurrent.futures and pathlib would be the examples, maybe urllib too.

Of course, it's not nice, but it's a problem that is fixable entirely on my side. I cannot fix the dependency on cryptography for others though, not on my side at least, w/o completely taking over another package.

[felixfontein]

@wvxvw for pathlib / pathlib2, the name of the Python module changed (from pathlib to pathlib2). In that case, a rename makes sense, because the semantics change. (I haven't looked at the others. I guess they will have similar reasons.) In any cas,e that's not the case for cryptography, using the library does not change (except possible removals of deprecated features). Just renaming everything because a new dependency was added is pretty extreme and very unfriendly to most users of this library.

Different topic: doesn't a simple constraints file with cryptography < 3.4 help in your case?

[alex]

👋 good morning everybody. This ticket contains a lot of different complaints about the 3.4 release, which incorporates Rust. First: I'm sorry this transition has not been as smooth as we'd all like.

Some context: For several years, @reaperhulk and I have been doing research and publishing on the security threat posed by memory unsafe languages like C. Just last week I spoke at Enigma on this. We believe long term adoption of memory safe languages is nothing short of a moral imperative.

I'm not going to rehash everything that's been said in this thread, but to summarize my views: We made substantial efforts to communicate this change in advance, it's possible to disable this change with a simple environment variable (for this release only), we invested a lot of our time and energy is improving the Rust integration with python packaging (including contributing PEP384 support to pyo3).

And finally: No, I don't care that Rust is not well supported on HP-UX, AIX, and similar. You'll notice I'm being cavalier and not at all empathetic on this particular point. That's because these are _incredibly_ minority platforms, and we (speaking more broadly) can't hold up security improvements for platforms stuck in the past.

On a philosophical note, I'm surprised to discover that so few users of Python packaging employ lockfiles/exact version pinning.

With all that said, let's get constructive!

I think there's a few categories of issues going on here, let's talk through them -- I'll sketch out my views and would appreciate feedback:

1) There's lots of users who could be consuming wheels (in which case they have no need for Rust to be installed), but for the fact that their pip is out of date. Here, I'm not sure there's anything to be done but encourage people to update their pips -- we already do this in every piece of documentation we have.

2) Users with pips that predate PEP517 get an ImportError. This is another case where updating pip resolves it, and we've made a change to have this print out additional debug information.

3) Users with older versions of rust installed experience nasty compilation errors. Here I think we can a) improve our docs to clearly indicate the version of rust required, b) more clearly suggest installing rust with the docs from upstream, c) think about how we might implement the version check directly and thereby give a clear error message (https://github.com/rust-lang/rust/issues/65262#issuecomment-763765009 would be best, but it's not yet stable)

4) It's clear that a lot of alpine users would benefit from having wheels. Someone (not me!) ought to do the work to write a PEP for an alpine build tag, then we'll ship wheels for it and reduce the burden on all those users.

5) It's clear that, despite our best efforts to communicate this, folks were taken by surprise. Where would you have liked us to post this circa 6 months ago?

[webknjaz]

@clanzett

and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on

I see how this may make you feel frustrated I don't think anybody is forcing you to do anything. It's completely your choice like you chose to use the library out of your own free will (instead of implementing your own, I guess). Blaming things on others is rather destructive thing to do. Besides, did you pay the maintainers for any of the great work they've been doing all these years? Did you pay them for not introducing a safer technology? This comes off as thankless.

@Smirl

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time.

Nobody promised semver and it's publicly documented at https://cryptography.io/en/latest/api-stability.html.

@MrMino

BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?

I think a combo of pinned envs + dependabot (or similar) + CI can catch this just fine. Pinned envs are reproducible, and any incompatibilities are clearly visible in automated bot bumps when they explode in CI w/o automatically breaking the main branch.

@manuelmarquezs

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. [...]

Any idea on why the compilation with Rust is that slow??
How could we speed it up??

How about:

1. Cache base docker layers/images (as in all the static deps are built ones and so install+compile is not invoked in runtime)
2. Cache pip wheels — pip builds wheels out of sdists when they are installed for the first time, any subsequent installs that select the same version will pick up those wheels and just unpack them (platform-specific wheels are zip archives with pre-compiled C-extensions)

@wvxvw

[...] Of course, in part, the blame is on pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version.

Hey, let's not go into assigning blame here. Plus there's absolutely no fault of pip here — it resolves the deps correctly per declared metadata. Besides, only people involved in the development can dictate what should be done and how, consumers who just come in to do the blame and expect to benefit from free labor are in no position to demand how the volunteer efforts are being spent. The maintainers don't owe you anything — by using this software you agreed with its license that says that it's provided "as is" (the license files are in the repo, I encourage you to read them). Also, you are free to fork the project and set your own rules there (for as long as they comply with the licenses, of course).

But you, as a maintainer, also did a very bad job at signaling this to pip.

This is not true, the maintainers did a good job within what the current state of the Python packaging allows. I do agree, though, that the packaging ecosystem could be improved. But again, it's run by volunteers meaning that you could influence it by making a volunteer effort. It would be a good idea to even just participate in discussions around improving the distribution package metadata and contribute your use-cases and ideas. Those discussions can be found at https://discuss.python.org/c/packaging/14.

Some topics for improvements are tags for Alpine, as @alex rightfully mentioned, and less widespread things like BSD and others. I believe this would require somebody working on a PEP similar to ones introducing manylinux.

If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.

I'd be totally fine with there being a cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.

It would be a good suggestion at the time when the maintainers were collecting the feedback, not post-mortem.

@MrMino

I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.

It _is_ covered, see: https://cryptography.io/en/latest/api-stability.html. In particular, it says:

X.Y is a decimal number that is incremented for potentially-backwards-incompatible releases.

@wvxvw

Our product is also written in Rust, and your dependency on Rust is completely unnecessary for us to consider in our CI. pip is the version it is for a lot of reasons too.

You seem to expect deprecated software to be supported virtually forever. I believe it's absolutely possible, you'll just need to pay somebody to perform such a maintanence.

@felixfontein correct me if I'm wrong, concurrent.futures and pathlib would be the examples, maybe urllib too.

I believe Felix is talking about https://github.com/pypa/pip/issues/8509 — TL;DR because pip doesn't yet have a concept of transactions, files claimed by two different packages get wiped out on upgrade.

[mcejp]

Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Presumably I'm doing something wrong, but this hit me on Debian 10

[webknjaz]

@mcejp

Presumably I'm doing something wrong, but this hit me on Debian 10

Sounds like you've hit an old version of pip as mentioned before. Be aware that deb-like OSs have policies in place that hugely change the whole Python experience in those envs: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e.

[MrMino]

@alex I've created https://discuss.python.org/t/wheels-for-musl-alpine/7084 for Alpine.

Where would you have liked us to post this circa 6 months ago?

There's no one place that could've fulfilled this purpose. To me the issue is that we _as a community_ don't have standardized channels for such communication. I think this issue is bigger than the project in question.

@webknjaz

It is covered, see: https://cryptography.io/en/latest/api-stability.html.

Yup. And that scheme failed us today, when lots of maintainers realized that some of the dependencies they use are made with total disregard for this. People assume it's semantic. We either start acknowledging that fact, or we'll be introducing a major breakage in the ecosystem every other quarter.

_Assumptions are the mother of all f*ck ups_

[webknjaz]

Yup. And that scheme failed us today
[...]
_Assumptions are the mother of all f*ck ups_

TBH there are a lot of packages in Python ecosystem that use this scheme starting with ansible-core, ending all the packages that use CalVer, for example. Going further, among OSs that many in this thread use, there's Ubuntu that does CalVer too. So it's not like there's an assumption in the community that every project would use SemVer automatically. Reading the docs about the versioning expectations is on those who specify the dependencies in their envs, many libs don't have this properly documented but this one does and it is very clear about what to expect...

There's no one place that could've fulfilled this purpose. To me the issue is that we _as a community_ don't have standardized channels for such communication. I think this issue is bigger than the project in question.

The docs at https://cryptography.io/en/latest/community.html point at an ML and an IRC channel. This should be enough honestly. More places make it harder to track things... OTOH I'd probably encourage the maintainers to enable Discussions in the repo settings as this provides a great platform for Q&A style posts.

[MrMino]

So it's not like there's an assumption in the community that every project would use SemVer automatically

I'm not talking about end-user apps and huge behemoths such as Ubuntu. I'm talking packaged APIs. I haven't checked, but I think that if I do, it will turn out that the vast majority of packages on PyPI use SemVer. Or any scheme that uses "if the first digit changes, you'll have to open your IDE, sorry".

More places make it harder to track things...

No, not forums to discuss it - a functional scheme (e.g. in PyPI), that would allow a maintainer to notify their users about a breaking change in advance. We all do this using a weird amalgamation of emails, forum posts, pinned issues and warning messages. Why?

The signal to noise ratio for this is so small that it becomes tedious to stay alert.

I don't want to subscribe to every possible message on the topic there is, I'd like to have something that gives me a clear signal that says "dude, if you don't fix your crap in 1 week, it will end up in flames".

[alex]

FWIW, our version/API compatibility scheme comes from Django.

A PyPI feature to let us pro-actively notify our users seems like a good idea, I'd encourage folks to discuss that on the PyPI issue tracker!

[clanzett]

@webknjaz :

I did never blame anybody. I just said, that this change is forcing developers to adapt their CID/CDs a.s.o. and asked to take back this change. Nothing else....

The problem is, that this great library is used as a dependency in a lot of other packages and I guess many users are not aware of that. So just using another library won't fix anything - at least not for me - because that would mean that I also would need to change many others.

In fact I am very thankful for all the work, but I think open source does not mean, that I can't speak about my concerns!
So please read my comments or try to understand them instead of accusing my of being not thankful!!

[MrMino]

@alex

FWIW, our version/API compatibility scheme comes from Django.

Which, honestly, might not be a good idea. Django assumes that _it_ runs _your_ code (i.e. inversion of control), not the other way around, so breaking changes are unlikely to have that much of a fallout there.

Simply put, Django, Ubuntu, Chrome, Pip, & others that use non-semantic versioning are almost never a transitive dependency. That's why using CalVer there doesn't come with a high cost for people that use it. In the case of Cryptography however...

[webknjaz]

@MrMino

No, not forums to discuss it - a functional scheme (e.g. in PyPI), that would allow a maintainer to notify their users about a breaking change in advance.

This is something that should go to https://github.com/pypa/warehouse or https://discuss.python.org to see if people would want to discuss implementing it.

We all do this using a weird amalgamation of emails, forum posts, pinned issues and warning messages. Why?

The signal to noise ratio for this is so small that it becomes tedious to stay alert.

I don't want to subscribe to every possible message on the topic there is, I'd like to have something that gives me a clear signal that says "dude, if you don't fix your crap in 1 week, it will end up in flames".

I think that the deprecation warnings serve this purpose rather well. But in this case, instead of a deprecation warning that would be happening in runtime, there was a build-time error that I'd consider a sort of a warning too: there's no real functional Rust code on the main branch right now — it only has a no-op with the main purpose of battle-testing the build toolchain setup at https://github.com/pyca/cryptography/blob/master/src/rust/src/lib.rs. I believe that this can be considered a "soft bump" of the build deps, it's easy to disable for one version with an env var, as described in the FAQ. It seems to be way smoother than merging lots of Rust and using it right away. The mechanism with the env vars is almost the same as with any previous deprecation.

@clanzett

In fact I am very thankful for all the work, but I think open source does not mean, that I can't speak about my concerns!
So please read my comments or try to understand them instead of accusing my of being not thankful!!

It's good that you clarify this now: the problem is that in the text medium many language constructs can carry some sort of emotion and we need to care not only about what we write but also how we do it because it's somebody else who is going to experience the reading and can perceive the same words differently. Especially with the surrounding context saying that they "did a very bad job", or "in part, the blame is on pip" — this explicitly says "blame", I didn't interpret it in an arbitrary, unless there's another meaning of the word that I don't know...

[sabueso]

FIRE IN THE HOLE

[MrMino]

@webknjaz

I think that the deprecation warnings serve this purpose rather well.

You could've put a large ASCII art of a kitty in the warning message and I still wouldn't notice. The reason we have automated pipelines is so that we don't have to look at them _until they give us a signal_.

[alex]

Don't think I won't start including giant ASCII doggos...

I agree wholeheartedly about the merits of automation, and it's clear there's gaps here where there's no infrastructure available for some of that.

That said, I think the folks most fully embracing automation (pinned dependencies + dependabot for updates) would have experienced this relatively smoothly.

[ambv]

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

With all due respect, it seems to me like this is a strawman. We're not talking about a security update here, are we?

@MrMino, the OP here is a Gentoo package maintainer, not an end user. You'd expect repackaging security-related libraries is done with more care than a simple fire-and-forget script in CI.

I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.

While this particular project's versioning scheme is different, I agree that a 4.0 version number would signal the dependency change in a clearer way. Maybe they'll make 3.5 into a 4.0 because there you won't have a magic "get out of jail free" environment variable.

Ultimately though the issue we're commenting on would still be created by someone and we would still be having the same discussion. The developers did communicate their change through their available channels and it was missed/ignored. And most people have this dependency as a transitive dependency.

Currently the lockfile in one of repositories I maintain shows 354 dependencies. The main dependency list includes around 50. The app that is broken for me is a staging environment for a distributed hardware testing platform. This is as far from security concerns as you could get, there's no way I'm able to inspect every single transitive dependency for updates, and there's even less probability that management permits me to spend my time on it.

If Nokia's management is fine running random code from the Internet on your infrastructure then that's your call but ultimately large organizations ought to do some sort of dependency gatekeeping. I can tell you this from first-hand experience as this used to be one of my responsibilities. You can find other FAANG employees telling you the same.

In any case guilting unpaid volunteers by mentioning their actions will inconvenience your paid job isn't a good view. If the maintainers are unable to maintain an LTS version with C code alongside the current version based on Rust, nothing you can say will change that.

Also, now it's not _just_ about Gentoo, is it? Don't you think the 90% figure is a bit exaggerated?

Note that Rust is available on Gentoo, just not on fringe architectures like alpha, hppa, ia64, m68k, and s390. None of those are even actively supported by CPython. s390x and ppc64le are "best effort" and at least we have stable buildbots for them.

90% was obviously a rhetorical figure but we can roughly estimate it for instance by looking at the PyPI download stats for the most popular dependency for Python. On February 6th, it has seen 3.17 million downloads on Linux. That's 96% of the downloads. So Alex's claim that non-Linux platforms are "incredibly minority platforms" is well supported by those numbers, especially if you take into account that another 2.5% is Windows.

We can also look at Python version proportions and we'll see that on that same day 60% of downloads were from Pythons 3.7 - 3.9. That means they're newer than Ubuntu 18.4 LTS which ships with Python 3.6, which also has been established can be fixed by upgrading pip. In fact, 19.9% downloads were from EOL versions of Python, including Python 2.7 - 3.5 which are irrelevant in this discussion because cryptography 3.4 does not run on any of them. So excluding those from the proportions, roughly 75% of global downloads are from Linux running Python 3.7+.

It would be cool to see those downloads per distro, per CPU architecture, or per pip version but it is what it is. So yeah, somewhere between 75% through 95%. We'll see this number move up as Debian 11 gets released this year, and so on. For others, upgrading pip or using rustup is a one-time change that the user needs to make to carry on. This includes Gentoo, Ubuntu 18.4 LTS, and Alpine Linux. Unless you're on m68k...

Finally, this is somewhat off-topic but Alpine Linux itself is a weird choice for a Python container given that it's slow and CPython does not run any stable buildbots on either Alpine or any other musl-based distro.

[tiran]

Note that Rust is available on Gentoo, just not on fringe architectures like alpha, hppa, ia64, m68k, and s390. None of those are even actively supported by CPython. s390x and ppc64le are "best effort" and at least we have stable buildbots for them.

Nit pick: s390 and s390x are different platforms. According to Wikipedia, s390 was a mainframe system released in 1990 and discontinued in 1998.

Cryptography's Rust extension builds fine on s390x. https://koji.fedoraproject.org/koji/buildinfo?buildID=1704720

[thesamesam]

@ambv @alex Please note that this was not mentioned in the CHANGELOG or release announcement emails beforehand as far as I can tell (I may have missed it, but if I did, that might mean shouting louder would've been a good idea -- i.e. more than once) which is what would've been needed to ensure any sensible packagers or consumers are picking it up.

I don't think it's doable to expect people to read the entirety of development mailing list archives to find what is needed...? I try to do this sometimes but it's not feasible to do it all of the time. Anything critical or important should be mentioned in release notes, and in this case, IMO ahead of time.

Keep in mind that some of us maintain hundreds of packages and I think we give better care to things by not becoming entirely consumed with the minutiae of development of each of them. In this case, yes, it would've been great to see, but an RFC on a random patch? Probably not. Other people are way better placed to give their views on those.

You might also say we should read all commits (or at least summaries) and I do actually try to do this and review diffs to ensure they look sane, but I'm not sure that everybody can or does do this in reality.

[alex]

By "beforehand" do you mean "at the time of the 3.3" release? Because that's correct, it has been included in the changelog for master for a while though.

[clanzett]

@webknjaz
"did a very bad job" or "...blame is on pip" was no statement from me. It was from @wvxvw ;-) just to clarify...

[omersi]

My version is pinned down to 3.3.1 and still facing the problem over alpine image.
What's the point of pinning down a version if I suffer from such upgrades?

cryptography==3.3.1

[thesamesam]

@alex Right, that's what I meant. It would've helped me out (and some others out, based on the upvotes) at least to know beforehand before any Rust was written so I could share any views or useful perspectives, but also a "get your engines ready" a release or two in advance.

I understand in this case that I wouldn't have been able to dissuade you from using Rust (and I like Rust!) but I could've possibly raised notes re other platforms, and that may have influenced when Rust was e.g. made mandatory. I don't know - that's just speculation, I'm just saying that opportunity to share views at a critical juncture would've been neat.

Separately, a "hey guys, stuff is going to break in a few releases" would've definitely been helpful to give a kick-up-the-backside to get some bootstrap binaries built and other work needed to package Rust where we have non-Tier 1 Rust support.

I try to keep an eye on development in packages I maintain or have an interest in, but it's not always doable. Especially because some packages don't update the CHANGELOG until the time of release (this isn't your fault at all, I just mean it's not like a workflow I can universally apply).

There's enough to do with the changelogs, deluge of emails daily, and release notes of packages which need updating and bugs fixing without having to go _searching_ for issues which may not exist. But yes, obviously if it's a core/critical/security-relevant package, I look harder.

[MrMino]

@ambv

the OP here is a Gentoo package maintainer, not an end user.

Forgive me if I misread your comment, but I'm pretty sure that you haven't specified who are you responding to. Many other users are now piggybacking this thread to moan about their particular cases, so I assumed you're responding to all of us.

If Nokia's management is fine running random code from the Internet on your infrastructure then that's your call

Sorry, no. You're reframing my comment into something I have not said, and putting me into a very uncomfortable position by mentioning my employer. I don't like this tactic at all.

• This is not about security.
• This is not random code.
• We don't run random code on random infrastructure.
• The update has not been flagged by anything that would alert me to it.
• Running random code is what staging environments are for.
• I'm responsible for staging environments.

---

In any case guilting unpaid volunteers by mentioning their actions will inconvenience your paid job isn't a good view.

Just so we're clear: I'm not. And I disapprove of anyone who's doing that.

I'm rather grumpy about the fact that it had to happen and I try to find a constructive way out of it, but that's it. I am incredibly grateful for all the work the maintainers have put into this project. If any of my comments read as if it's otherwise - tell me which one, I'll edit or remove it.

[webknjaz]

@MrMino

You could've put a large ASCII art of a kitty in the warning message and I still wouldn't notice. The reason we have automated pipelines is so that we don't have to look at them _until they give us a signal_.

They give you a signal. But by default, it's ignored. The best practice is to use python -W error in CI and filterwarnings = error in pytest that will notify you about such things immediately (this allows you to ignore specific warnings temporarily until you fix your compatibility) — the first entry in FAQ mentions this (https://cryptography.io/en/latest/faq.html#i-cannot-suppress-the-deprecation-warning-that-cryptography-emits-on-import) and here's pytest docs showcasing the use of it: https://docs.pytest.org/en/stable/reference.html#confval-filterwarnings

@clanzett

was no statement from me. It was from @wvxvw ;-) just to clarify...

I'm sorry, I must've confused the quotes with this much hate happening here. Still, your comment about being forced to do things can also be interpreted this way.

@omersi your problem doesn't look related to this discussion. But I guess you're seeing the need to compile things because Alpine doesn't match manylinux tags (that's expected) and there's no standard for pre-compiled Alpine wheels (see https://discuss.python.org/t/wheels-for-musl-alpine/7084).

[MrMino]

@webknjaz Nice. I'll need to read up on that and I'll make sure that the pipelines do alert me in the future then. Thanks.

[alex]

Adding to the CHANGELOG in advance of future releases is a good idea, and one we'll try to incorporate.

[thesamesam]

@Alex Thanks! A shout out in the stuff beforehand so I know when I'm doing a release to keep something in mind for future is a big help.

[MrNaif2018]

Actually, I love such updates because they usually move the industry forward, showing the need in new features. I would love to see alpine linux wheels standardized, but actually this issue and some pieces of advice not to use alpine have urged me to compare docker images (a little bit offtopic, but still).
My current alpine image is using lots of compiled packages, including cryptography (even before that release it was quite a lot, but now with rust leftovers it will probably be even more in size).
I have tried to do the same with an python:3.6-slim-buster image instead of python3.6:3.6-alpine.
And to my surprise, the end image was 232 MB instead of 315 MB (26 % less) of the alpine image, plus that image uses wheels (so such changes like this would do nothing), which speeds up the build like 10 times, and makes it possible to even add caching.

So, what can I say, maybe this update is an call for action to check your apps and upgrade to something better (:

[mgorny]

First of all, I find it really sad (or even distressing) that my original argument that Rust doesn't support modern Linux systems on minor architectures immediately got twisted into 'HP-UX, AIX, and similar'. No, this is really about up-to-date Linux systems. I can't say we managed to provide superior support to them so far but we haven't been put into the position of having to discontinue their support entirely either.

Secondly, I filed this bug because I thought that maybe upstream hadn't realized how non-portable Rust is and how much problems this causes. However, reading this thread -- and I'm sorry if I'm wrong -- I get the feeling that upstream just doesn't care about all their users. In fact, I'm myself not interested in optimizing stuff for everyone. However, I don't believe this should go at a price of providing zero support for minorities. Slow, pure Python fallback would be at least a nice thing for those who can't use Rust extensions.

Finally, and this personal -- I feel really betrayed here. So far I was advocating cryptography as a really decent replacement for the dead and buried pycrypto library. And now it turns out that cryptography is no longer a replacement for it because it stops working on some platforms we do actively support.

[hynek]

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. The build of the cryptography package that used to take a few seconds now takes minutes.

Have a look at https://pythonspeed.com/articles/docker-cache-pip-downloads/

It is now possible to share pip caches between builds which includes the wheeling of binary extensions. This should improve your build times even beyond the 30s.

[alex]

I really don't think Alpha, s390, or hppa are any less obscure or a minority than AIX or HPUX. But in any event, feel free to substitute them in my comment, I stand by it.

While I'm sorry this is causing difficulties for you, I do not see how this was avoidable. Our users (indeed, all users) are on x86(-64) and ARM. We simply cannot abandon forward progress to platforms that haven't released in decades.

[omersi]

@webknjaz
It started failing today with rust exception.
What do you think caused it?

Alpine 3.10
cryptography==3.3.1

[alex]

FTR, if anyone from HP, IBM, or DEC cares about this problem, HMU. Happy to work with you to find ways to fund LLVM and Rust.

[webknjaz]

@omersi

It started failing today with rust exception.
What do you think caused it?

No idea, you didn't provide enough info to deduct that just from a few sentences w/o any details. But it is 100% certain that the version you're linking does not have any Rust references at all. I would guess that you may be installing a newer version of cryptography in some other place of your project without realizing it. FWIW this is not the right place to get support.

[lu-zero]

Alternatively helping on the rust-gcc front-end might help solving at least partially the problem.

[rrey]

I was impacted in my CI running in an alpine container on an ansible collection repo.
I added rust and cargo in my alpine install deps (apk add cargo rust) and got my CI back.

Not a big deal on my side.

[CounterPillow]

FTR, if anyone from HP, IBM, or DEC cares about this problem, HMU. Happy to work with you to find ways to fund LLVM and Rust.

DEC went out of business long ago, HP absolutely does not care about their RISC ISA from 1996 that they've officially stopped support for in 2013 (8 years ago), and IBM is content huffing its own mainframe farts consisting of 60 year old COBOL.

[Xyrodileas]

Like everyone, pipelines went down because using Alpine for containers..

[alex]

Folks who are commenting that your pipelines broke: Do you pin your Python dependency versions (pip-tools compile, pipfile, etc.)?

[alfredodeza]

Although we are pinning dependencies, we saw breakage because Cryptography is a dependency of a direct dependency in our requirements.txt file. In particular it is Authlib==0.15.2 which in turn requires cryptography>=3.2,<4. So pinning all your dependencies _does not guarantee_ preventing a problem like this.

My highest respect for all the folks working on this project on their own time, sometimes it is tricky to avoid breaking changes, even if they are well communicated.

[alex]

Ah, so you pin direct dependencies, but not transitive ones. Is that what other folks are doing?

[fourstepper]

For me it was similar, ansible depends on cryptography

[abrookins]

@alex Right - same here in one of our builds. We'd pinned direct dependencies but not cryptography.

[alex]

Can I ask why? Is this an intentional design choice, a tooling gap, something else?

[Eric-Arellano]

In the pip ecosystem, a constraints file can be particularly useful for pinning transitive dependencies: https://pip.pypa.io/en/stable/user_guide/#constraints-files. It allows you to keep your requirements.txt to only include requirements you directly use, but still be able to pin transitive deps. With pip's old resolver, the constraints file will override all transitive deps to use your version; with the new resolver, it will error if your pin is not compatible with a direct dep.

Here's a simple script to generate that constraints file:

#!/usr/bin/env bash

set -euo pipefail

# You can change these constants.
PYTHON_BIN=python3
VIRTUALENV=build-support/.venv
PIP="${VIRTUALENV}/bin/pip"
REQUIREMENTS_FILE=requirements.txt
CONSTRAINTS_FILE=constraints.txt

"${PYTHON_BIN}" -m venv "${VIRTUALENV}"
"${PIP}" install pip --upgrade
"${PIP}" install  -r "${REQUIREMENTS_FILE}"
echo "# Generated by <relative_path_to_script.sh> on $(date)" > "${CONSTRAINTS_FILE}"
"${PIP}" freeze --all >> "${CONSTRAINTS_FILE}"

Note that you don't need to include all your transitive deps in a constraints file! You can have a constraints file with only one entry.

[abrookins]

@alex I suspect a lot of casual projects like mine (and not-so-casual ones apparently) only pin direct dependencies rather than pip freeze, or Pipenv, etc.

[jtrakk]

In case it's helpful to anybody, the principal tools for pinning all dependencies are:

• pip-compile from pip-tools which is easiest if you have an existing setup.py
• Poetry which is easiest for new projects.
• (Pipenv does this too but I would not recommend Pipenv.)

The workflow is to have two files: in the "constraints" file, you define your dependency constraints manually. Then use a locking tool like pip-compile or poetry lock to generate a separate "lockfile" which contains all of the pinned transitive dependencies and hashes.

[tj90241]

@alex I suspect a lot of casual projects like mine (and not-so-casual ones apparently) only pin direct dependencies rather than pip freeze, or Pipenv, etc.

There are likely large projects out there which do not - OpenStack being one example. Vendors whom package into their own python packages almost certainly don't fall into this category. However, if you pull an OpenStack package like keystone from pypi right now, it is part of the ripple effect here.
https://github.com/openstack/keystone/blob/master/requirements.txt#L9

[webknjaz]

I second the advices above: I recommend using pip-tools, it's the most native thing that pip can use directly. Other lockfile formats are usually specific to custom package managers but should also achieve the same goal.
If your project is on GitHub, I recommend enabling Dependabot — it supports pip-tools managed constraint pairs, so when it sends PRs updating separate dependencies, each such change gets tested and it's quite transparent when something in the dep tree breaks, even if it is transitive — you'll know which one right away.

[ntoll]

Sending big hugs to @tiran - you're doing a great job matey.

[omersi]

@omersi

It started failing today with rust exception.
What do you think caused it?

No idea, you didn't provide enough info to deduct that just from a few sentences w/o any details. But it is 100% certain that the version you're linking does not have any Rust references at all. I would guess that you may be installing a newer version of cryptography in some other place of your project without realizing it. FWIW this is not the right place to get support.

Found the root cause
other libs were using unpinned versions causing this failure.

[tidux]

Bounty for LLVM+clang+Rust on m68k Linux/NetBSD/AmigaOS when?

[AdilHindistan]

Can I ask why? Is this an intentional design choice, a tooling gap, something else?

I think it is tooling gap. pip freeze is not that helpful when you want some packages to be pinned but not all.
I spent whole day cleaning up the mess this change caused. I had ansible pinned but it was pulling the latest cryptography.
As people suggested above pinning cryptography as well is the solution.

[webknjaz]

@AdilHindistan

I think it is tooling gap. pip freeze is not that helpful when you want some packages to be pinned but not all.

pip-tools exists to help you with this exactly. TL;DR you can track your direct deps in a requirements.in file, having loose caps or none at all but then you use pip-compile which generates requirements.txt with the whole tree of transitive (indirect) dependencies and you can re-generate it over time to update the tree to newer pins for the whole thing (instead of pip freeze). Then, you can pip install -r requirements.in -c requirements.txt and this will allow you to have a separate place to track your direct deps along with a separate lockfile with all the pins that help you keep your env reproducible as those pins will be know good versions of everything.

[ignaloidas]

The security benefits are of course appreciated - but not the way that we got to know about them. Cryptography is a transitive dependency of a transitive dependency of dependency of our code. Our 6 small team that does not deal with code in any way related to cryptography should not need to carefully track the changes in packages that deep in the tree. One of our dependencies updated today - and with it updated cryptography. Our builds started failing. We got to learn about this package, and it's deceptive versioning scheme.

You say you know who your users are. You don't. You don't know any of the users that install your package through a package manager. Ubuntu, Debian, Alpine, Fedora, Gentoo, etc. all maintain packages of this library for the architectures you just inadvertently removed support from. You don't know how many of them install this library, and how important it is for them.

You might not have known that Python tools are horrible when it comes to non-python dependency management. Now that I think about it, maybe it's more worth to just use packages of distributions, that would have prevented all of this from happening. I wish I could on this project. But yet, if I did, you wouldn't see me as a user.

You say it's for the better, you say that the old platforms are not worth keeping secure and should just move on - ignoring the point that by removing support for them, you silently left them insecure, unable to get important updates and vulnerable for attacks. You say the extra dependency is alright, since it fixes a lot of potential problems that could arise if continuing to use C. And this might be right. But you know what also deals with a lot of security sensitive stuff and is written in C? Linux. There has been numerous suggestions to rewrite parts of it in Rust, to improve security. But those suggestions were rejected. Why? Because Linux developers knew, that adding Rust might block their users from using Linux. Linux developers don't know all of their users. But they trie to care about all of them, whether they know them, or not. Linux developers care about their users. You don't.

[virusdave]

I'm just here for the whining of people who's build systems are using nondeterministic versions and/or unpinned upstream dependencies.

Folks, if you do this, you're going to get what you deserve. Don't blame the maintainers of this software for your own choice to float dependencies. The level of snide and hate here for maintainers is nuts, and is a solid example in the disincentives to quality software present in the OSS community.

Do better!

[alex]

@virusdave notwithstanding the fact that I agree about the merits of pinning deps, I don't think this is a productive way to engage.

[MrMino]

I am fairly confident this change should be rolled back ASAP.

@alivcor please read the whole thread before launching demands.

@alex I think this horse has been beaten enough already. Can we please close this thread? Is there anything else to say on this topic?

[webknjaz]

@alex I think it may be a good idea to lock this topic as it just gets a lot of hateposting and all of the pinning improvement advices have been given already...

[alivcor]

@MrMino I did read the thread - the advice given here is passing the blame to someone else (in this case - users). Strongly echo what @ignaloidas has pointed out very well - _You say you know who your users are. You don't._

I appreciate the forward facing changes and improvements made, but unless the software being built (whether open-source or otherwise) fails for the very problems it aims to solve - I feel its a lose-lose.

[alex]

Nobody is blaming anyone, we're trying to figure out how to design resillient systems (https://how.complexsystems.fail/) that work for as many use cases as is practicable. Doing this requires understanding the types of workflows which were impacted.

Certainly it's my hope that organizations which are impacted will run blameless post-mortems to attempt to see how their own processes for managing third party dependencies can be improved.

That said, you've provided no details on your problem. You showed up, complained that something didn't work (without even including so much as an error message) and demanded we revert our release. I think you need to seriously rethink how you engage with open source projects, as that conduct will get you nowhere.

If you want to include error messages and instructions for reproducible, we're more than happy to figure out a) how to fix what you're running into, b) how we can improve the situation to make the next person in that set of circumstances have an easier time. But until then, you don't appear to be even attempting to contribute to this discussion constructively.

[webknjaz]

@alivcor no, the advice was that pinning your deps is a responsible thing to do. And that a blame culture is counterproductive is just common sense that people tend to forget and therefore need to be reminded about.

[alivcor]

That said, you've provided no details on your problem. You showed up, complained that something didn't work (without even including so much as an error message) and demanded we revert our release. I think you need to seriously rethink how you engage with open source projects, as that conduct will get you nowhere.

@alex Now that's simply incorrect (and you pointing out my thinking process is actually blaming users to rethink their process!). I have contributed to open source myself and I think I know what it takes. I didn't feel the need of adding an error message here because the evidence is clearly laid out by 100+ users on this thread. I didn't "demand" the release to be rolled back, I said I was _fairly confident_ that is the best resolution here - and I stand by that. Whether you do that or not - it's your choice, you guys are doing the amazing work here - not me.

I think if I would get such reaction from users of my open source software, I'd definitely lean towards THEIR perspective. As @clanzett pointed out here - open source does not mean, that you can't speak your concerns and offer suggestions - In my case and clearly with many other people here, rolling back this breaking change was the best possible resolution to prevent any more people bumping into this.

[mgorny]

I think the best thing to do here would be to provide a pure Python implementation of whatever's necessary. After all, isn't one of the points of using Python in the first place to write memory-secure code? If you're really concerned about the performance here, you can make Rust extensions optional, like many other Python projects provide optimized non-Python paths with Python fallbacks.

[alivcor]

I think I agree with @mgorny here - probably moving rust and compiled versions to extras would be a good idea.

For anyone who might be using this library and facing an issue because of transitive dependencies (airflow etc) - Use an older version eg; 3.0.0 - that will satisfy the requirements and you won't need to install any extras.

pip install cryptography==3.0

[alex]

Simply re-implementing everything comes with a number of downsides:

• There's no way to implement constant-time code in pure Python securely
• It requires us to implement all the relevant portions of the codebase twice. While I very rarely make appeals to "we are all volunteers", it's simply that case that it's not good use of our limited time to do so.
• It requires us to expand our test matrix further. We already run an incredibly large test matrix, over 50 builds on every commit. We'd need to introduce the cross-product of that matrix by which implementation we are using, or establish what subset of which we could safely ignore, (ironically, this is likely to make us even less likely to add new platforms to our test and release procedure, as this makes the overhead of new platforms higher)
• It introduces a risk of behavioral differences between implementation, as well as increased complexity.

[nikolalukovic]

For all the entitled "users" here who believe maintainers of this package should specially cater to their needs read the license:

Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

You're not paying for this product and you can't expect any special support. It seems to me that some of you people think that free and open software gives you the prerogative to expect free and open support as well. Well that's not how things work.
You've had plenty of warnings about this change in a past couple of months as well.
Using this library and being updated about its changes is your responsibility. And if you have issues down the chain with other libraries using this one then move this discussion to their forums.

[alex]

Notwithstanding the fact that I obviously believe there are some use cases that cannot be accommodated while maintaining our other desiderata (security, performance, maintainability, etc.), I don't think the warranty that all OSS licenses include is a healthy way to approach this: https://alexgaynor.net/2014/may/19/service/

[nikolalukovic]

I can accept that my previous comment about warranty and expectations of support are a bit harsh. And I also admire your altruistic take on the issue at hand.

However I'd still argue that it's unreasonable to expect, like some "users" here do, for this library to evolve and change only in a direction they want. These narrow-minded, self-centered, arrogant takes do not impress me. It's a completely ridiculous thing to even think of asking. Not to mention if their requirements clash with such meaningful and needed changes. It impedes growth and forces stagnation.

Not to mention how they immediately shift the blame to this library when they didn't to jack on their end.

Open Source projects of a certain size, scope, and importance need to take seriously the fact that we have an obligation to our users. Whether we are volunteers, or paid, we have a solemn responsibility to consider the impact of our decisions on our users. And too often in the past, we have failed, and acted negligently and recklessly with their trust.

This is a sword that cuts both ways. These "users" aren't taking their side of responsibility when working with OSS libraries.

[tidux]

Upstream developers of a security critical dependency like a crypto library (outside an ecosystem like a major Linux distro with dedicated security teams) telling people to version pin and that the only version they can use on their hardware will not be getting fixes backported is semantically equivalent to a raised middle finger. This change should not occur until all OS/distribution architecture maintainers sign off on a path forward because yes, they outrank you on this. If you orphan half the Python ecosystem on half the supported architecture/OS combinations, you will accomplish nothing besides driving people away from Python entirely.

[offbyone]

That has already been addressed:

That's because these are incredibly minority platforms, and we (speaking more broadly) can't hold up security improvements for platforms stuck in the past.

[alivcor]

I'm sorry but pointing out the license to pass the blame to users is definitely not constructive. Open source software is all about welcoming suggestions, accepting that issues happen and then working on to fix them rather than pointing out users to change their thinking process or passing blame onto them. I'm going to unsubscribe from this thread, this is definitely the opposite of how an ideal team of open source contributors must be functioning.

I am sorry to have suggested my inputs here - peace out.

[nikolalukovic]

You just have an idealistic view on what OSS should be and not what it actually is. And that is your problem.

Not to mention how only "solution" these "users" are giving is to just remove Rust dependency and that's it. That's not constructive at all.

[josephg]

I’m not sure if it’s been considered, but this might be a good candidate for a wasm build.

If the rust code is compiled to wasm, it should be able to be loaded and wrapped using any of the python wasm vms. That should allow the bundled library to run on any platform (os/arch). (Well, at least any platform that has a working wasm vm.)

Wasm is a particularly good choice for a library like this because it needs very little interaction with the host OS.

[adsharma]

I think the best thing to do here would be to provide a pure Python implementation

That would be an excellent idea. One of the issues with writing low level code with python is the level of abstraction is very high. Arbitrary precision integers is an example.

I've spent some time transpiling statically typed python code into rust. Looking to collaborate with more people interested in something like this, so one day we can use python, not C as a general purpose programming language to write safe/secure code that compiles on a large number of platforms.

[yawaramin]

@adsharma see https://github.com/pyca/cryptography/issues/5771#issuecomment-775549504

[adsharma]

There's no way to implement constant-time code in pure Python securely

Probably thinking about using python with meta-classes and all kinds of bells and whistles. Even though it's possible, my observation is that most people write fairly simple python that could be statically mapped to rust. I can think of a linter that would advise you to stick to a safe subset of python for this purpose.

Here's the code I have. It's nowhere close to being able to transpile everything needed for something as large as pyca. My hope is that some day it will.

https://github.com/adsharma/py2many/tree/main/tests/cases
https://github.com/adsharma/py2many/tree/main/tests/expected

Much of the rust work is from an upstream project called pyrs referenced in README.

[kaniini]

A few points:

• Rust is not available yet on many Alpine platforms (though you can help with this by solving the bootstrap issues, and such help would be appreciated).
• Developers who want to avoid exposing their users to the rust dependencies can use cryptography<3.4 in their requirements.txt for now.
• Unfortunately, this means that any package in Alpine which depends on py3-cryptography must now either be restricted to the archs where rust is supported or we must hold py3-cryptography back to 3.3.x releases. In any case, Alpine users could apk add py3-cryptography to speed up their builds and avoid the rust dependencies.

[kaniini]

Unfortunately, py3-cryptography in Alpine has been marked as restricted (by me) and held back to 3.3.2 until the rust bootstrap path is solved for all architectures.

We have to support py3-cryptography on architectures not presently supported by rust due to it being a dependency of other packages.

While it is true that Rust has many benefits, the ecosystem is not ready for distros which are strongly oriented toward architecture portability.

Hopefully cryptography 3.3.x will continue to receive security updates, but we cannot update it to anything that requires rust until such time that rust supports all architectures on Alpine.

[alex]

Which alpine architectures do not yet support rust?

[alexyao2015]

Hopefully, this helps some people. I recently went through the trouble of migrating my CI builds to handle this new Rust dependency and ran into several stumbling blocks with multiplatform Docker builds with QEMU.

1) Rust has to be installed manually through rustup-init.

2) Issues arise when building for arm/armv7 architechtures:

rust-lang/cargo#7451, crazy-max/ghaction-docker-buildx#172

See here for the fixes/workarounds.
https://github.com/hass-emulated-hue/base-image/commit/e530602a29ad0cbd13f39285eaee593db27802d9

[kaniini]

Which alpine architectures do not yet support rust?

• m68k (port, also not yet supported by LLVM)
• ppc64 (port)
• mips64 (release architecture)
• mips64el (port)
• mips (port)
• s390x (release architecture)
• sh4 (port, also not yet supported by LLVM)
• riscv64 (port, but likely to be included in 3.14)

[alex]

For mips64 and s390x, which are both supported by Rust upstream, what's the blocker? (Is there some existing place to read about this rather than asking a bunch of questions?)

[vielmetti]

The APKBUILD for py3-cryptography on Alpine is here for reference: https://git.alpinelinux.org/aports/tree/main/py3-cryptography/APKBUILD

[kaniini]

The blocker is that we cannot successfully cross-compile rust in the bootstrap process. I have asked in the Zulip chat about this before and nobody got back to me with advice. As we cannot cross-compile rustc, we cannot bootstrap rust on those architectures (the previous bootstrapping was done by hand using trusted packages from Void Linux).

Finally, rust lives in community at present, not main, and we are hesitant to support rust in main, because of the volatility of the rustc development process, as main has an extended lifecycle than "supported until next branch."

We do, however, support py3-cryptography in main due to it being a dependency of other software there.

[plentifullee]

This one solved it for me:
cryptography<3.4

[adsharma]

One more point about CERT's secure coding standards for C and integer overflow:

https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=87152052

These types of rules can be automated by coding at a higher level of abstraction (say subset of python) and generating rust code.

I added basic support for this (return types are auto generated). Bug reports welcome.

https://github.com/adsharma/py2many/blob/main/tests/expected/infer-ops.rs#L18

[tiran]

@ignaloidas

You say you know who your users are. You don't. You don't know any of the users that install your package through a package manager. Ubuntu, Debian, Alpine, Fedora, Gentoo, etc. all maintain packages of this library for the architectures you just inadvertently removed support from. You don't know how many of them install this library, and how important it is for them.

That's not correct.

I'm the package maintainer if PyCA cryptography on Fedora, RHEL, and indirectly also CentOS. I was involved in the process from the beginning. Over the last couple of months I worked with upstream, developers of PyO3 / setuptools_rust, and the Fedora Rust special interest group to package dependencies for Fedora. I had the first Fedora package up about 12 hours after the first release of 3.4, https://bodhi.fedoraproject.org/updates/FEDORA-2021-ae11bd2c46. It took me so long, because Fedora's python-pytest-subtests package (a test dependencies) was not up to date.

The maintainer of Debian packages was also involved in the process.

@kaniini

Which alpine architectures do not yet support rust?

* m68k (port, also not yet supported by LLVM)
* ppc64 (port)
* mips64 (release architecture)
* mips64el (port)
* mips (port)
* s390x (release architecture)
* sh4 (port, also not yet supported by LLVM)
* riscv64 (port, but likely to be included in 3.14)

FWIW, Fedora infra successfully created packages for PPC64le, s390x, and riscv64.

• https://koji.fedoraproject.org/koji/buildinfo?buildID=1704720
• http://fedora.riscv.rocks/koji/buildinfo?buildID=190686

[kaniini]

FWIW, Fedora infra successfully created packages for PPC64le, s390x, and riscv64.

The rust bootstrap issue is likely specific to Alpine, due to being a distribution that is dynamically linked and using musl. I suspect we hit an edge case while cross-compiling rust that makes the process crash. At any rate, I personally spent nearly 200 hours of time working on diagnosing the issue and discussing in the Rust Zulip server which did not go anywhere productive.

At any rate, until rust is present on all architectures Alpine supports (at least the ones officially shipping today), we cannot ship py3-cryptography which depends on it, and so we will continue to hold it back at 3.3.2.

I am willing to contribute time and infrastructure to supporting a manymusl format so that x86_64/aarch64 users may download wheels instead as an alternative, but we cannot just drop py3-cryptography from a subset of our architectures as we have packages we need to support everywhere which depends on it.

Sorry if I sound repetitive here, I am sure it is tiring. You guys have to do what you have to do (use rust, for many legitimate and beneficial reasons) and I have to do what I have to do (ensure py3-cryptography remains present on all architectures as it is part of the base repository).

[mithrandi]

The maintainer of Debian packages was also involved in the process.

FWIW, that's me. I don't have much to add to the discussion, and I won't be uploading 3.4 for bullseye given the timing (we're just going into the start of freeze) but I did provide feedback to the cryptography maintainers around this change and I see no problems supporting it in bullseye+1. Debian currently builds rust on all of the release arches as well as a number of ports: https://buildd.debian.org/status/package.php?p=rustc

[Flamefire]

Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

I'd have one:
Instead of using your own versioning scheme use what many others use and are used to: semantic versioning.
I.e. instead of 3.4.x it should have (likely) been 34.x.y, see the definition of semantic versioning.
As shown here not using it is surprising for many, especially if every project was to invent its own versioning scheme so you'd need to read each an every documentation to find out what a version means. Also note that PIPs new version resolution scheme basically assumes SemVer

[tiran]

Did you just said «f&ck the minorities, it is only majorities who deserves our labor»?

You feel entitled, don't you? Unless you have a paid service contract, you are not in a position to tell us how we spend our free time on OSS. You may ask politely and we may decide to accept or decline your petition.

(Spoiler alert: There are no paid contracts)

By the way, do you know, that actually you probably have more "alpine" users (all the CI/CD containers) than all the others?

Can you backup your claim with facts and hard numbers? Setting your personal attacks and swear words aside, we would love to see good statistics in regards of distribution and architecture.

Speaking in my position as a CPython core dev:
In case you actually want to help instead of just vent your anger, please consider to contribute to Python's ecosystem. CPython is lacking CI for Alpine and musl libc-based distributions. Other major distributions have contributed resources to CPython's buildbot farm. There are or were also crashers and performance regressions related to Alpine in CPython. The article https://pythonspeed.com/articles/alpine-docker-python/ lists several issues with Alpine. We like to see these issues fixed, but we lack resources.

[apollo13]

I'd be curious to know the reason for using alpine in CI/deployment for python as opposed to the debian based images. Having musl as opposed to libc means one has to compile dependencies from source because manylinux wheels will not work. Compilation in turn means longer build-times etc… So I really wonder why one would prefer alpine.

I for one applaud the changes done here and it is just fair if CI runs break, it is just a sign that someone wasn't pinning their dependencies so the reproducibility might be questionable anyways.

[kaniini]

I feel the need to step in and clarify that @msva is not an Alpine developer. Please do not associate his hostility with us.

Can you backup your claim with facts and hard numbers?

I don't think anyone here doubts that Alpine is widely used. I don't think that there's really any value in bragging about what distro has what user count, but there are many docker containers built on Alpine, just as there are Ubuntu, Debian, etc containers.

The pythonspeed.com article about Python and Alpine is largely unrelated to Alpine itself, but instead the lack of wheel support for musl distributions, but Alpine being the most widely used musl distribution gets blamed.

There are also workarounds for those issues, such as using the python modules we already package (which is a lot of python modules, definitely all the "big" ones, though using our packages have their disadvantages of course).

I think the primary solution is to define a generic ABI for musl distributions, so that musl distributions may have wheels too.

If contributing VMs for buildbot CI would help, feel free to reach out to me, I would be happy to set some up.

[tiran]

@Flamefire

Instead of using your own versioning scheme use what many others use and are used to: semantic versioning.

As far as we know this would not have helped much. The majority of affected users did not have any version pinning for transient dependencies.

By the way, the version schema is documented here https://cryptography.io/en/latest/api-stability.html#versioning

[Arfrever]

When you are re-writing code in Rust, you could just leave previous code written in Python / C as fallback, instead of actively deleting it, and ask community of users on those architectures etc. to provide patches when necessary.

[kaniini]

I'd be curious to know the reason for using alpine in CI/deployment for python as opposed to the debian based images.

In my case, I prefer the CI environment to match production. I use Alpine in production for everything, because I am literally one of the people who designed its featureset according to my specific requirements.

As for why, it's simple: musl is a better libc for my workloads, and many parts of it have substantially simpler and more robust implementations than glibc. For example, it is possible to make locale code in glibc crash.

Alpine also gives me a toolchain that opportunistically enables hardening features like FORTIFY without having to explicitly enable it -- instead you must explicitly opt out of hardening features. Alpine also gives me userspace tools such as our package manager which feel nice to use.

Having musl as opposed to libc means one has to compile dependencies from source because manylinux wheels will not work. Compilation in turn means longer build-times etc…

There are alternative solutions, for example, using system packages instead of building a virtualenv (which is a waste of time for a container hosting a single app anyway), which are even faster than using pip to begin with. But we should work on a manylinux equivalent for musl distributions.

I'm also told one can also cache layers in Docker to speed up builds (which is very nicely leveraged if you can order your layers in the right order), but I don't personally use Docker.

[tiran]

I feel the need to step in and clarify that @msva is not an Alpine developer. Please do not associate his hostility with us.

Don't worry, I don't associate the user with Alpine. You have been reasonable and helpful. :+1:

Can you backup your claim with facts and hard numbers?

I don't think anyone here doubts that Alpine is widely used. I don't think that there's really any value in bragging about what distro has what user count, but there are many docker containers built on Alpine, just as there are Ubuntu, Debian, etc containers.

My question totally not motivated by a bragging contest. :)

I'm honestly interested in good statistics. OSS projects have limited resources and cannot accommodate all possible user cases and users. Good statistics would help to identify where we can archive more bang for less bucks.

[apollo13]

@Arfrever In theory a nice idea but I doubt in practice this will work. What happens as soon as you change part of the interface and the community doesn't come up with those changes quickly? Also with security relevant code you really don't want multiple implementations.

[ambv]

When you are re-writing code in Rust, you could just leave previous code written in Python / C as fallback, instead of actively deleting it, and ask community of users on those architectures etc. to provide patches when necessary.

@Arfrever, this is a no-go because:

• it creates a false sense of security for users who would be unknowingly running the old code because of a wheel/build problem;
• it doubles the code that realistically needs to be maintained while requiring the Python side of things to interface with either one consistently;
• there is no magical "community of users" waiting to work on this, the cryptography maintainers are the community we've got.

[kaniini]

For what it's worth, I think that simply committing to maintaining cryptography 3.3.x for an extended period of time while the rougher edges surrounding rust portability get smoothed out (through initiatives such as the Rust GCC frontend, for example) would solve this for everyone.

I'd be willing to volunteer to help with maintenance of cryptography 3.3.x. I am sure others would volunteer to help make 3.3.x an LTS release, too.

[ambv]

⬆️ this is how things get solved in open source. People, take note.

[apollo13]

@kaniini Those are all very good arguments and certainly valid. But I do not think that most people who are using Alpine are choosing Alpine for those reasons. Do not get me wrong, I am not trying to bash Alpine here or anything. But the status quo is that wheels do not work and as such I think the original python images might be better for users that just decided on Alpine because other people told them it is faster/smaller. I guess the "Know your tools"-sentence applies to this situation as well, people have different needs and should choose accordingly. In an ideal world manymusl would exist, but in the meantime we have to deal with the status quo.

There are alternative solutions, for example, using system packages instead of building a virtualenv (which is a waste of time for a container hosting a single app anyway), which are even faster than using pip to begin with.

Sure, but that won't work if the system packages don't allow for multiple versions of a package etc. That is why we got in this mess in the first place :) As far as I know many people do not use virtualenv in containers but install directly which makes sense imo. I go even as far as having a multi-stage image where I install into /opt/something and then copy that over into the new container + PYTHONUSERBASE to point python towards it. Works wonders (and also applies to Alpine so you do not need the compiler etc in the final image)

I'm also told one can also cache layers in Docker to speed up builds (which is very nicely leveraged if you can order your layers in the right order), but I don't personally use Docker.

Yes you can but you need to do some tricks like manually copying requirements.txt first to not kill the cache with every change to your app etc. In the end it is an art :D

[kaniini]

I agree that with interpreted languages like Python, it is a harder cost-benefit analysis, and I also agree that a lot of people who say "you can speed up your builds" or "you can make your images smaller" with Alpine are not selling the whole story.

But I think people benefit from running their Python workloads on Alpine regardless, due to the hardened toolchain and high quality Python packaging you get in Alpine.

I think a manymusl wheel format would be beneficial, but one of the main reasons why I like using Alpine for Python workloads is because there aren't prebuilt wheels. This allows me to build those dependencies with my own toolchain, and thus I do not have to trust the binaries included in the wheel. I suspect this is also why people have not pursued a manymusl wheel format yet -- they likely have the same philosophy I have on the matter, that they would prefer to build the binary packages themselves at deploy time with their own trusted toolchain. (I am aware that it is possible to configure pip to ignore wheels anyway, but if there are no wheels, then you don't have to configure anything :upside_down_face:)

I agree that using system packages for Python packages is not a good scenario if you are running a setup where multiple applications might be deployed in the same container. But for a lot of scenarios they are overlooked in favor of using pip, even though they are faster than using pip at all.

And I do think that many people use Alpine for their builds because of the package manager.

apk add -t .build-dependencies python3-dev build-base ffi-dev ...; [...]; apk del .build-dependencies is something that people have explicitly told me is a feature they love, because it helps them keep their images tidy. I wish other package managers had that feature, I find myself missing it frequently when I am using Debian or Fedora.

[ambv]

I'm sorry, if previous message was considered a very hostile

Honestly, @msva, I would remove your original comment. [edit: he did.]

For the others reading, Vadim is dead wrong because:

• Alpine is "affected" for 99.99% of its users in the sense that they have to build their own wheel, not because cryptography no longer works on it
• Alpine was already in this situation for any pip install-ed Python package because it cannot use wheels on PyPI
• for everybody else this is either a non-issue, an issue requiring a pip upgrade and/or installing a Rust compiler which is already there
• Gentoo and Alpine maintainers expressed that they can only upgrade a package when it works on all their CPU architectures, including those which are obsolete CPU architectures that CPython itself is not officially supported on, you can find the full list above
• additionally, Alpine's got a hopefully temporary problem of bootstrapping Rust on musl on two architectures
• Alpine and Gentoo could have packaged the new version of cryptography for their majority CPU platforms to speed up builds for those users but their packaging philosophy adopts an "all-or-nothing" philosophy
• this is not actionable by the maintainers of cryptography because they are not interested in supporting said CPU architectures: they have limited resources and those architectures are a drop in the bucket

[Arfrever]

• Fortunately CPython is written in C and still works on alpha, hppa, ia64, m68k, s390.
• Even on x86_64, building Rust may require many hours (around 10 - 20 hours on slightly older x86_64 hardware), so packaging new version of cryptography will not speed up anything.
  (And on some fast x86_64 system when I can measure it, building Rust (with parallelization) requires about 1 hour 40 minutes.)

[kaniini]

Alpine and Gentoo could have packaged the new version of cryptography for their majority CPU platforms to speed up builds for those users but their packaging philosophy adopts an "all-or-nothing" philosophy

In the case of Alpine, this isn't because of a philosophical issue. We would have to introduce slotting to the package in order to allow different architectures to have different versions of the py3-cryptography system package, which would be a substantial amount of work (largely because we've never slotted a Python package before).

In our case it would just make more sense to help maintain 3.3.x packaging for the community until such time that rust has equivalent portability as GCC. That is something I am willing to assist with, if the community is interested in making 3.3.x an LTS branch that can get us through for a year or two.

The light at the end of the portability tunnel is ultimately the Rust GCC frontend, which seems to be going full steam ahead, and rust itself as a language appears to largely have stabilized in the past year, making such a frontend viable. That means that having 3.3.x as an LTS branch is probably feasible, because we can probably expect to see Rust support land in GCC 12, 13 or 14.

[tiran]

Fortunately CPython is written in C and still works on alpha, hppa, ia64, m68k, s390.

Speaking as a CPython core dev:
CPython does not support these platforms. You are on your own. We won't fix bugs for these platforms -- unless somebody contributes the bug fix, it's trivial, and does not cause any potential issue for other platforms. We will also break these platforms, if it makes our lifes easier.

You should not expect that you can run latest software works on decades-old hardware and have support from upstream.

[kaniini]

You should not expect that you can run latest software works on decades-old hardware and have support from upstream.

But my Amiga... :frowning_face:

In all seriousness, I think that there is value in supporting those architectures. Weird architectures can and do expose security bugs, so it's helpful for smoke testing if nothing else. And if the community is willing to help you do it, I don't see why upstream has to be hostile to it.

[Arfrever]

I have not seen majority of other CPython developers so actively trying to break these architectures as you are saying.
I see various places in CPython code with conditionals on __alpha__, __hppa__, __ia64__, __m68k__ and __s390__.

[naparuba]

And yes, claimin that "users have no right to dictate developers what to do in OSS, since they was not paid any cent" is right

Yes, that's all. Just stop here.

The only right you have it to fork it you are not agree (thanks OSS).
The licence do not open you any rights to complain, even it it break your workflow, your fault to not follow upstream updates and think all is done for your specific workflow.
That's just so simple.

Congrats to @tiran and @ambv for the good work, the whole majority of users is happy to have more secure cryptography, even it it means I'll have to update some few lines in my build scripts. But that's the price to pay, it's ok.

[kaniini]

To get back on topic -- what do people think about an LTS branch for cryptography 3.3.x, which we would support until some point shortly after the Rust GCC frontend is merged and proven? This would be a reasonable solution that provides an answer for those needing a more portable package now while the portability issues get resolved over time.

[Flamefire]

@tiran

As far as we know this would not have helped much. The majority of affected users did not have any version pinning for transient dependencies.

By the way, the version schema is documented here https://cryptography.io/en/latest/api-stability.html#versioning

I read through this thread and found multiple users expecting semantic versioning and a raise in the major version for such a change. Some have then found that linked page but my comment was that (reasonable) expectations were broken which is almost always a bad thing.
So if cryptography used SemVer and increased major versions only for real breaking changes (i.e. removed API), which are rare, then a major version increase would stick out and cause attention. IMO this might have helped.

And yes even though your versioning scheme is documented: People expect SemVer which is the quasi-standard for versioning, especially for Python packages.
Just imagine you having 20-30 dependencies in your software (stack) and each uses a different versioning scheme. How would this be able to be handled at all? Not only would you need to check compatibility for all on their various doc pages, all your dependencies must do too and agree here. Of course you can say: "just pin your version". Ok, then 2 of your dependencies do but to a different, but actually compatible version. So installation will fail for no real reason.

For cryptography: How do you distinguish between releases where you added an API, fixed a bug in an implementation detail or removed an API? These are 3 different things, but the current versioning scheme can only support 2. This leads to unnecessary strict pinning of versions (if done) and hence incompatibilities which could be avoided by just using a standard versioning scheme.

I hope this helps.

BTT: An LTS 3.3 branch sounds like a good solution.

[ljmccarthy]

Probably the most expedient method of supporting minority platforms in Rust would be to add a backend to LLVM that emits plain C code, which can then be fed in to GCC or whatever. Or maybe write an LLVM IR to C decompiler, which would not even require co-operation with upstream. Sounds like a fun project I would like to work on, if I had any spare time...

[alexandernst]

@tiran Don't let people change your mind. This is the way! 💪

[ncopa]

@alex With all that said, let's get constructive!

:+1:

@alex 5. It's clear that, despite our best efforts to communicate this, folks were taken by surprise. Where would you have liked us to post this circa 6 months ago?

This is a tricky problem because downstream consumers can not be expected to participate in all upstream discussions when they have hundreds (thousands?) of dependencies, so even with best effort done to communicate this, it is impossible to avoid breakages. Here are a few ideas for reducing the surprise:

• What you already did, as a warm up, openly communicate via traditional channels like twitter, blog posts, mailing lists long time ahead of the breaking change.
• Use semver. Even if you do have a documented versioning scheme, many (most?) people don't read the documentation and will get a surprise when a breaking change is introduced without first number in version increases. (I would definitively been bit by this)
• Use a transition period:
  
  
  
  • first make a release where rust is introduced but C is still supported and the default. A compile flag can be used to enable rust. Big red/blinking warnings are displayed in the build logs that C support will be removed in future version. I believe more people look over the build logs than reading the docs.
  
  
  • make another release where rust is the default but C still supported. A compile flag can be used to use the C variant. but with the same warning.
  
  
  • ideally support both for a while. I know this may not be possible or realistic, but it would be helpful.
  
  
  • make a release without the C support.
  
  

[mithrandi]

So if cryptography used SemVer and increased major versions only for real breaking changes (i.e. removed API), which are rare, then a major version increase would stick out and cause attention. IMO this might have helped.

I am personally a fan of SemVer, but under SemVer this would have been a minor version bump as no API changes were made!

[kaniini]

One other thing I want to point out with my Alpine s390x maintainer hat on that's been kind of grinding my gears.

@mgorny's complaint about s390 not being supported by rust is ridiculous. Linux has not supported booting on traditional 31-bit s390 since Linux 4.1 in 2015. If you can't boot s390 (not s390x) with a modern kernel, it is absurd to demand that other projects support s390.

This is why Alpine only supports s390x from day 1, and why musl does not have a 31-bit s390 port.

[ncopa]

@tiran Other major distributions have contributed resources to CPython's buildbot farm. There are or were also crashers and performance regressions related to Alpine in CPython. The article https://pythonspeed.com/articles/alpine-docker-python/ lists several issues with Alpine. We like to see these issues fixed, but we lack resources.

I feel you. I'd love to see those issues fixed as well but also struggle to keep the head above the surface. I think we (alpine) should contribute to the pythons buildbot farm. Will try find someone that can help with that.

The musl python wheels issue is somewhat more tricky. I believe https://bugs.python.org/issue43112 needs to be solved first.

@tiran I'm honestly interested in good statistics. OSS projects have limited resources and cannot accommodate all possible user cases and users. Good statistics would help to identify where we can archive more bang for less bucks.

I don't have exact numbers but:

• https://hub.docker.com/_/alpine/ reports 1B+ downloads. As comparison ubuntu has 1B+, Debian 500M+, Fedora 50M+
• Our primary CDN package repository hosted by fastly currently reports a hit rate on between 600-800 req/s. This does not include all the other mirrors we have.

This of course does not tell anything about the python usage on alpine, and is probably small numbers compared to the numbers a popular project like python cryptography operates with. I still believe that supporting less popular platforms and architectures is beneficial for python as it helps shake out corner case bugs and in often improves the overall code quality.

[tiran]

@mgorny's complaint about s390 not being supported by rust is ridiculous. Linux has not supported booting on traditional 31-bit s390 since Linux 4.1 in 2015. If you can't boot s390 (not s390x) with a modern kernel, it is absurd to demand that other projects support s390.

Thank you very much! That's a valuable piece of information. I filed https://bugs.python.org/issue43179 for __s390__.

[kyrias]

@msva:
I'm sorry, if previous message was considered a very hostile, but anyways, as it is already been notified, cryptography's re-invented versioning kinda conflicts with https://www.python.org/dev/peps/pep-0440/#version-scheme

which causes failure for "compatible release" (~=) dependency operator, which, in turn, brakes many things.

This isn't true though, as defined the ~= operator requires a constraint that is dependent on the versioning system used by the dependency in question. If you specified ~= 3.3 then you yourself is saying that 3.* releases are compatible, which is an incorrect constraint for this package. ~= used correctly for cryptography would require specifying three segments, e.g. ~= 3.3.0.

[MarkSG93]

This could be rectified by sorting out the git tags? This is a very clear breaking change and should be treated as such in the versioning of the package.

[MarkSG93]

It sounds like y'all are getting an unexpected lesson about the flip-side of open source software. I would like to push back on some of the entitlement I'm sensing reading in this thread.

Before I begin, I'd like to remind you that security is a numbers game. If the cryptography maintainers can help 90% of their users by switching to a modern memory-safe language, then they'd be irresponsible holding back just because among the remaining 10% there exist fringe platforms which can't even run a Rust compiler.

You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer is worth more than the time of cryptography maintainers. They are volunteers as well, y' know. You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.

In fact, your only real recourse long-term is exactly to band up volunteers to make Rust work on your platform. Or pay some hired guns to do the initial work for you. After all, you will be seeing more and more projects depending on this programming language as time goes on. Yelling at creators of those projects for choosing a language that is inconvenient for you is both ineffective and laughable.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

I invite you to do that. Please put your money and time where your mouth is. Report back in a year's time how it went.

Nobody is saying that it's a bad idea. Don't push a minor patch version for a very clear breaking change and none of these conversations would even be happening. Stop making assumptions about peoples systems and software and use correct versioning. If this was cryptography 4.0.0 then nobody would care because all their builds and docker images still work, but because it's 3.x a lot of us are up a creek without a paddle.

[noonien]

ITT: People that use (probably not formally verified) CRYPTOGRAHPY packages, distributed as BINARY BLOBS, that cannot be verified to be compiled from a certain commit, versions of which get AUTOMATICALLY UPDATED in their CIs, making even source builds untrusted.

You guys are surprised your builds break? And then blame the author?

I suggest you ask for a refund.

[tamasgal]

Holy crap, now I need to install Rust on like 20 base Docker images which we use for around 300 software projects in our collaboration. Seriously?

[kaniini]

Thank you very much! That's a valuable piece of information. I filed https://bugs.python.org/issue43179 for __s390__.

To be clear, hppa and m68k are still quite alive and well in Linux, please don't use that as justification to remove support for those archs. I like running Python on my Amiga.

[tiran]

Holy crap, now I need to install Rust on like 20 base Docker images which we use for around 300 software projects in our collaboration. Seriously?

No, you don't. There are multiple documented workarounds like updating pip to >19.1 or setting an env var. It's all explained in the error output.

[ignaloidas]

@tiran I've just run the queries on the BigTable of PyPi download data. It shows what pypistats.org doesn't, and that is that 87% of cryptography downloads over the last 7 days were source downloads. This goes against your claims of the majority of users using binary wheels. You don't know who your users are.

Edit:

Query for reference:

SELECT file.type, COUNT(*) AS num_downloads FROM `the-psf.pypi.file_downloads` WHERE file.project = 'cryptography' -- Only query the last 7 days of history AND DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) AND CURRENT_DATE() GROUP BY file.type;

[msva]

I've deleted all my comments in this issue to stop affecting anyone who can be affected in negative manner by them.

But I still want to describe my point of view: despite of authors are have all the rights about their code, I personally think that becoming an industry standard adds something like "moral restrictions" and responsibility, even over "all of those who uses it wrong way".

[tiran]

@tiran I've just run the queries on the BigTable of PyPi download data. It shows what pypistats.org doesn't, and that is that 87% of cryptography downloads over the last 7 days were source downloads. This goes against your claims of the majority of users using binary wheels. You don't know who your users are.

the-psf.pypi.file_downloads is a download statistics, not a users statistics.

You might be right that majority of our users don't use binary wheels. You might draw wrong conclusion from the statistic, because the data might not reflect a large chunk of our users. Amongst others packages or container image layers are cached at multiple levels. Linux distros and Python distributions like Anaconda have their own distribution channels. It's impossible to draw accurate user statistics from the download stats.

[MrMino]

@kaniini I gather that you are interested in a "manymusl" format. I created a thread for it on Python Discourse, I invite you to share your thoughts: https://discuss.python.org/t/wheels-for-musl-alpine/7084/15

The largest obstacle so far is that there'd have to be a champion for the standard. While I'd love to try to contribute in this way, I'm pretty sure it's above my head for now.

[ignaloidas]

You might be right that majority of our users don't use binary wheels. You might draw wrong conclusion from the statistic, because the data might not reflect a large chunk of our users. Amongst others packages or container image layers are cached at multiple levels. Linux distros and Python distributions like Anaconda have their own distribution channels. It's impossible to draw accurate user statistics from the download stats.

Yet you used statistics from exact same source to claim that the majority of your users use Linux. It is impossible to draw any reliable user statistics for a project with such wide usage, yet you try to justify your choices with claims about your users that you made up.

[MrMino]

You might draw wrong conclusion from the statistic

@tiran @ignaloidas

I don't think its even the right statistic. Ask JFrog how many of their clients have Artifactory, and how many instances those clients have. If you count in those (how would you even count them?), then _maybe_ you can get any sort of real world value for downloads.

Counting it in terms of users is IMO the wrong way to go about it. If I have a small business that does ~1000 CI runs a day, and I lead the business alone, do you count me in the same way as the guy next door who accidentally downloaded paramiko?

These stats say nothing about the scale of the problem, and arguing over them is a waste of time.

[naparuba]

But I still want to describe my point of view: despite of authors are have all the rights about their code, I personally think that becoming an industry standard adds something like "moral restrictions" and responsibility, even over "all of those who uses it wrong way".

That's where I am not agree and think the OSS will burnout most mainteners: it's up to tje industry to manage all this additonal load (because it is, and lot of overload for such a project), not to upstream.

If it's managed by a big corpo, it's ok: they have all the power to manage such a load (they already doing it for internal use)
But I think force all upstream projects to manage it is a very wrong thing. All the overload must be managed by where the money/power/time is: the industry itself.

[Absolucy]

Probably the most expedient method of supporting minority platforms in Rust would be to add a backend to LLVM that emits plain C code, which can then be fed in to GCC or whatever. Or maybe write an LLVM IR to C decompiler, which would not even require co-operation with upstream. Sounds like a fun project I would like to work on, if I had any spare time...

@ljmccarthy That backend did used to exist. It was removed in LLVM 3.1, which was released in 2012. I think there's some unofficial C backend for modern LLVMs, but there's a decent chance it's not quite production-ready.

To get back on topic -- what do people think about an LTS branch for cryptography 3.3.x, which we would support until some point shortly after the Rust GCC frontend is merged and proven? This would be a reasonable solution that provides an answer for those needing a more portable package now while the portability issues get resolved over time.

@kaniini Problem is, that probably won't happen for _years_, and does anyone really want to take on the burden of maintaining a
legacy branch intended for devices that probably won't even make up 0.1% of it's actual users?

However, if you're willing to do so, that's great!

[rfink]

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

Yep, forced to do the same (AWS Glue broke when importing this for requests lib)

[ddevault]

Switching to Rust and locking out dozens of platforms is worse for security, because now those users have no choice but to rely on outdated security-critical software OR to port Rust to their platform, which is a gargantuan effort (hi, someone who has tried - and failed - to port Rust to a new platform here). A lot of those people can't afford to "upgrade" to newer hardware - much of which brings its own security concerns, like UEFI.

C is not evil. C is not a bad programming language. C is a universally supported language. Cargo culting Rust is harmful to the ecosystem.

[pfalcon]

There were questions on how the announcement/notifications could be handled better. Well, it's fair to say that this change affects the Python ecosystem as a whole, and so release notification should have been posted to the https://mail.python.org/mailman3/lists/python-announce-list.python.org/ mailing list. And indeed it was! But I'd say it was pretty meek: https://mail.python.org/archives/list/[email protected]/thread/J6VZ22LY5IS3WG5SMICUSWQQOK5SAFTQ/

Given the effect it caused, it should have a subject:

PyCA cryptography 3.3.2 and 3.4 released - SYSTEM BREAKAGE ahead

and should have started with:

###################### WARNING! #########################
Installing PyCA cryptography from source (as happens with
older pip releases) now requires Rust to be installed. If
you use old pip (if you didn't check, you likely do) and
don't have Rust installed, all packages depending on
PyCA cryptography will be broken!
###################### WARNING! #########################

And I'm only half-joking here, your recent release announcement before/with such a big change really contain too few exclamation marks and all-caps.

And I second what other people said - dear cryptography developers, you are doing the great work of maintaining security package as volunteers! But not only that, you also BREAK SYSTEMS of other people! Please take not less pride in that than in your development work. And if you don't want to take pride in that (why? it's fun), then maybe indeed spend a bit less time hasting to write more Rust, and a bit more time writing "scary" (read: disclosing the truth to users) (pre)release announcements.

Thanks!

[alex]

C is in fact very bad and I have a mountain of research to prove it: https://www.usenix.org/conference/enigma2021/presentation/gaynor

[ddevault]

C is in fact very bad and I have a mountain of research to prove it: https://www.usenix.org/conference/enigma2021/presentation/gaynor

https://en.wikipedia.org/wiki/Confirmation_bias

[alex]

You read my slides very quickly! I can tell you took them seriously and didn't just drop that link immediately.

[ddevault]

You're right - I don't take your research seriously!

[alex]

I recommend not using this software package then, regardless of what languages its written in. (Of course, I suspect that's already the case -- you're not a user, you just showed up to complain because this is your hobby horse.)

[ddevault]

I am a cryptography user, as a matter of fact, and a packager for affected systems. But thanks for the ad-hominem speculation, appreciate it.

[alex]

My apologies. I regret engaging with you at all.

First: this is going to be the second to last last comment on this thread. I think it's run its course. Folks should please feel empowered to file new issues, but I'd request that you only file issues for either a) new issues, b) concrete improvements not already discussed here.

Second: Thank you to all the folks who have engaged constructively here. Our release is better for your contributions.

Third: The _last_ comment on this thread will be a summary of everything discussed here, I'll leave that after I finish writing it.

[alex]

I now want to summarize the state of various discussions here:

Outdated Rust installations

We now take advantage of an option in setuptools-rust to provide a clear error message when a user's Rust installation is too old.

In the future, we'll be somewhat conservative in increasing our MSRV.

Update: We have lowered the MSRV to 1.41 in https://github.com/pyca/cryptography/pull/5823 and this change was released in 3.4.5.

No Rust installation at all

We've now documented very aggressively how to obtain a Rust installation, and to update pip (which will solve the problem for folks on manylinux-compatible distributions).

No future steps are expected.

Alpine

Wheels

It looks like there's folks actively interested in specifying many-musl. Excellent! Once that's defined and supported in pip, we'll ship wheels for that platform.

Bootstrapping Rust on supported platforms

We'll reach out off-list to see how we can support this effort.

LTS

We'll engage on the thread that @tiran filed.

SemVer

We'll consider it, however, I am skeptical for the simple reason that this change would not have required a major version bump (no public APIs were changed). Therefore it's clear that what folks wanted was a major version bump _regardless_ of semver.

Comms

In the future, when we know about a change like this well in advance, we'll include it in our CHANGELOG as soon as we know about it.

If a platform for proactively reaching out to PyPI users exists, we'll use it.

Alpha, S390, HPPA, HPUX, AIX, etc.

Best of luck.

I'd encourage you to contribute platform support to LLVM or contribute to gccrs.

My work to promote language-level memory safety will continue unabated.

Miscellany

I'm sure I missed something (though I did just re-read this thread in its entirety).