Nixpkgs: Convert remaining python2 applications over to python3

[jonringer]

In https://github.com/NixOS/nixpkgs/pull/101929 we have many important programs (e.g. cachix) still using python2 in their builds. This list doesn't constitute python2 applications, but rather packages which have python2Packages.cryptography somewhere in their dependency graph. So these will be affected when python2Packages.cryptography does get marked as vulnerable.

This issue to track the conversion process over to python3, packages still needing to be converted are listed below. This list isn't exhaustive, just those that use the soon-to-be-marked-vulnerable pythonPackages.cryptography:

Finding the dependency

For most dependencies, it should be pretty obvious where python2 comes from, for more "difficult" packages. You may need to do some digging.

nix-tree + nix-instantiate

you can run nix-shell -p nix-tree --run "nix-tree $(nix-instantiate default.nix -A <package>) to get the entire build dependency tree, then search for the cryptography package, and then you should be able to trace which dependencies are introducing it.

nix why-depends

alternatively, you can use nix why-depends
nix why-depends --all -f default.nix <package> python2Packages.cryptography can also be used, however, this will require you to re-build the package, which may take more time than parsing the dependency tree above

• [x] alibuild #102111
• [x] amazon-glacier-cmd-interface
• [x] aria2 #100191
• [ ] asciidoc-full-with-plugins
• [ ] babashka (through graal)
• [x] broadlink-cli #102162
• [x] buttersink (packaging doesn't work with python3)
• [x] cachix
• [x] carddav (currently, upstream only offers python2)
• [x] CastXML #101996
• [x] check-esxi-hardware
• [x] clj-kondo (through graal)
• [x] cloudmonkey (currently, upstream only offers python2)
• [x] csv_fast_export https://github.com/NixOS/nixpkgs/pull/102375
• [ ] datadog-agent (non-trivial to update https://github.com/DataDog/datadog-agent#getting-started)
• [ ] dd-agent (duplicate package? non-trivial to update https://github.com/DataDog/datadog-agent#getting-started)
• [x] euca2ools (currently, upstream only offers python2)
• [x] gdown
• [x] git-review
• [x] glslviewer https://github.com/NixOS/nixpkgs/pull/102365
• [x] gnss-sdr - taken cared of in #99685
• [x] gnuradio - taken cared of in #99685
• [x] gnuradio-with-packages - taken cared of in #99685
• [x] google-app-engine-go-sdk
• [x] google-cloud-sdk
• [x] google-compute-engine
• [x] gqrx - taken cared of in #99685
• [x] gr-ais - taken cared of in #99685
• [x] gr-gsm - taken cared of in #99685
• [x] gr-limesdr - taken cared of in #99685
• [x] gr-nacl - taken cared of in #99685
• [x] gr-osmosdr - taken cared of in #99685
• [x] gr-rds - taken cared of in #99685
• [ ] graal
• [x] haxor-news #102391
• [x] hercules-ci-agent
• [x] inspectrum - taken cared of in #99685
• [ ] jvmci
• [ ] kodi-plugin-yatp
• [x] mercurial
• [ ] mx (not sure which package this is, likely related to graal though)
• [ ] mysql-workbench (this will be difficult to update, links to python, and fails with python3.8)
• [x] ndn-cxx #102248
• [ ] nixops (currently, upstream only offers python2) https://github.com/NixOS/nixops/issues/1242
• [x] ocropus (currently, upstream only offers python2)
• [x] opae
• [x] ovito #102250
• [x] pantsbuild.pants (already broken)
• [x] persepolis
• [ ] pipreqs #102389
• [ ] pulseaudio-dlna-unstable (in progress, https://github.com/masmu/pulseaudio-dlna/tree/python3 , but not in releasable state)
• [x] pyside-apiextractor
• [x] pyside-generatorrunner
• [x] python3.7-aria2p #100191
• [x] python3.7-pygccxml #101996
• [x] python3.7-pyside
• [x] python3.7-pyside-shiboken
• [x] python3.7-pyside-tools
• [x] python3.8-aria2p #100191
• [x] python3.8-pygccxml #101996
• [x] python3.8-pyside
• [x] python3.8-pyside-shiboken
• [x] python3.8-pyside-tools
• [x] qradiolink - taken cared of in #99685
• [ ] rabbitvcs (in progress) #102378
• [x] rmlint
• [ ] sage (in progress) #101447
• [x] smugline removed: #102610
• [x] trac (currently, upstream only offers python2)
• [ ] tsung https://github.com/processone/tsung/pull/352
• [x] uget
• [x] uget-integrator
• [x] uutils-coreutils https://github.com/NixOS/nixpkgs/pull/102247
• [x] wal-e #102264
• [x] yle-dl
• [x] yoda
• [x] zabbix-cli (currently, upstream only offers python2)

[SuperSandro2000]

 python3.7-aria2p
 python3.7-pygccxml
 python3.7-pyside
 python3.7-pyside-shiboken
 python3.7-pyside-tools
 python3.8-aria2p
 python3.8-pygccxml
 python3.8-pyside
 python3.8-pyside-shiboken
 python3.8-pyside-tools

Why are those in this list? Do they have a python2 variant?

[jonringer]

not entirely sure. It could be that some library they use, uses python2 to do something like generate docs. So they aren't directly using python2 packages.

[primeos]

Why are those in this list? Do they have a python2 variant?

I assume those are mostly cases of Python 3 packages that depend on some non-Python package that in turn depends on some Python 2 package (e.g. python3Packages.aria2p -> aria2 -> ... -> python27Packages.cryptography).

The following can help to navigate the dependency trees more efficiently (/ to search for the Python 2 cryptography and then h to navigate up in the dependency tree):

$ nix-tree $(nix-instantiate -A python3Packages.aria2p)

---

Should I try to write a script (or does someone already know/have one) to ping the maintainers of the affected packages?

[SuperSandro2000]

then h to navigate up in the dependency tree):

The root is on the left and then use vim style key bindings or arrow keys.

[domenkozar]

cachix fixed in 59c53bc62e1e25015bd8deedd7252ee5037314b3

[jonringer]

I added some directions on how to locate cryptography in a dependency tree

[jonringer]

oh, nix-tree is way easier to navigate

[jonringer]

I tried packaging asciidoc-py3, but everything they do has xml imports to web urls.... :(

[mkenigs]

Looks like amazon-glacier-cmd-interface has python2 code: https://github.com/uskudnik/amazon-glacier-cmd-interface/blob/9f28132f9872e1aad9e956e5613b976504e930c8/glacier/glacier.py#L40

[SuperSandro2000]

The last real commit is over 6 years ago https://github.com/uskudnik/amazon-glacier-cmd-interface/commits/master. Maybe think about removing it.

[mkenigs]

@SuperSandro2000 removed it

[jtojnar]

@jonringer our XML packages include findXMLCatalogs hook which should make them available when you add them as dependencies.

[prusnak]

• uutils-coreutils in https://github.com/NixOS/nixpkgs/pull/102247
• ndn-cxx in https://github.com/NixOS/nixpkgs/pull/102248
• ovito in https://github.com/NixOS/nixpkgs/pull/102250

[freezeboy]

Some packages are still only available for python2 :facepalm::

• trac
• zabbix-cli
• cloudmonkey (old print in code)
• ocropus

[freezeboy]

inspectrum is simply depending on gnuradio also in the list, so it can be removed from here. gqrx too

gdown is packaged a python*Packages independant, so there is no problem.

[freezeboy]

babashka doesn't seem to depend on python but it uses graalvm which expression is quite big (and not up to date, but I prefer to ping @volth or @hlolli to patch this one)

[hlolli]

@freezeboy when https://github.com/NixOS/nixpkgs/pull/99631 gets merged, python2x wont be a dependency of graalvm anymore.

[volth]

I do not understand why python.org's decision to stop support caused such a massive cleanup in nixpkgs and why to invest time into that (and to long lasting bug hunting after the transition)

There are still maintained python2 forks (including commercially supported, for example https://github.com/ActiveState/cpython), it is possible to cherry-pick CVE fixes from there (if anyone concerted in security of tools which run only in buildtime sandbox)

[freezeboy]

opae must be updated to version 2.0.0-1

new version depends on pybind11, but there is a hacky cmake module trying to download it, I don't want to patch the cmake modules.

[freezeboy]

@volth maybe for the python interpreters but libraries might not get maintainance from their developers for python2 branch, so it is safer I guess to transition the maximum number of programs when possible

[SuperSandro2000]

There are still maintained python2 forks (including commercially supported, for example ActiveState/cpython), it is possible to cherry-pick CVE fixes from there (if anyone concerted in security of tools which run only in buildtime sandbox)

I wouldn't do that. At some point Python 2 should just be thrown out and if software is not updated until that date it should be marked as insecure, broken or removed.

[jonringer]

@freezeboy I updated the packages which upstream only provides a python2 variant

[freezeboy]

Thank you @jonringer, I just spotted tsung also which has a pending PR to support python3 (https://github.com/processone/tsung/pull/352), but not yet packaged unfortunately

[freezeboy]

carddav-util also is written in python2 and the project has no activity since 2018

[freezeboy]

euca2ools is also written in python2 even with latest release from 2017

[freezeboy]

glslViewer too, even if it has more recent updates is using python2 syntax

[jonringer]

I'm doing glslviewer now, the python2 scripts are very simple, and able to just use 2to3 with no problem.

[jonringer]

@thoughtpolice @domenkozar @rvl do you mind taking a look at bumping datadog, I'm not super experience with go, and looks like they made the build process painful (api key, and a bunch of wrapped invoke tasks) https://github.com/DataDog/datadog-agent

[doronbehar]

@jonringer I marked some gnuradio related applications with an [x] since they are taken care of in #99685 .

[jonringer]

@anna328p do you mind taking a look at hercules-ci-agent

[jonringer]

list should be up-to-date now

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-should-stable-nixos-prioritize/9646/55

[rvl]

We stopped using datadog-agent and started hosting our own prometheus monitoring.
The datadog-agent build process is quite painful, so it might be a bit of work to update to the latest version.

[freezeboy]

In addition there are 2 versions of datadog-agents in nixpkgs

[jonringer]

@rvl I noticed that, looks like they have a multi-language build now with cmake, go, and python https://github.com/DataDog/datadog-agent#getting-started

[omasanori]

Could I ask you to pin this issue, if possible? It is worth gaining publicity, as no one probably wants to maintain Python 2 until Autumn 2021.

[glittershark]

@jonringer https://github.com/NixOS/nixpkgs/pull/102693 gets us graal and all the related stuff

[glittershark]

not mx, though.

[jonringer]

besides graal packages, I think we are close to closing this.

[dasJ]

@jonringer asciidoc-full-with-plugins should be fixed (on master) with #102398

[SuperSandro2000]

This PR #102919 removes Python2 from Mono.

[domenkozar]

I'm going to unpin this issue as I'd like to get attention on Apple+Arm issue.

[jonringer]

@domenkozar that's fine, this is almost complete. Will probably wrap it up over the weekend