Packages: Python 2: Transition programs to Python 3

[jefferyto]

This is a meta-issue to track progress for transitioning programs that use Python 2 to Python 3.

These should be packages that use Python in some way, as part of their core functionality. Packages that provide a Python 2-only library or plugin will be tracked in a separate issue (TBD).

| Section | Package | Status | Maintainers | Related issues/PRs |
| --- | --- | --- | --- | --- |
| devel | asu | :star: Done | @aparcar | #10976 |
| lang | node | :star: Done | @nxhack | #10179 |
| lang | node-mozilla-iot-gateway | :white_check_mark: Resolved | @ratkaj | comment, #9670 |
| mail | mailman | | @Shulyaka | |
| net | aircrack-ng | :star: Done | @jbenden | comment, #11145 |
| net | danish | :no_entry_sign: Won't Fix | @smutt | comment |
| net | iotivity | :white_check_mark: Resolved | @hauke | #11587 |
| net | nmap | :star: Done | @nunojpg | #10162, #11605 |
| net | obfsproxy | :no_entry_sign: Won't Fix | @jefferyto | comment |
| net | openvswitch | :star: Done | @yousong | #8797 |
| net | radicale | :star: Done | | comment, #9082 |
| net | samba4 | :star: Done | @Andy2244 | #10707, #10781 |
| net | scapy | :star: Done | @MikePetullo | #8824 |
| net | seafile-ccnet | :star: Done | @commodo | #11565 |
| net | seafile-seahub | :star: Done | @commodo | #11565 |
| net | seafile-server | :star: Done | @commodo | #11565 |
| net | sysrepo | :star: Done | @mislavn | #9491, #10554 |
| net | uwsgi-cgi | :star: Done | @Ansuel | #9843 |
| sound | upmpdcli | :star: Done | @ignisf | comment, #9687 |
| utils | bluez | :star: Done | | #9080, #9094 |
| utils | gammu | :star: Done | @aTanW | #10186, #10171 |
| utils | hplip | :star: Done | @luizluca | #9323 |
| utils | yunbridge | | @blogic | comment |

Maintainers: If your program provides a Python 2 library package, please be reminded to keep it around (if possible) until the mass removal in 2020.

Please comment on / reference this issue as things develop. I will keep this post updated "periodically" :joy:

[neheb]

Nice! Want this pinned?

[neheb]

Maintainer of aircrack-ng is @jbenden

[jefferyto]

@neheb

Nice! Want this pinned?

Sure :laughing:

Maintainer of aircrack-ng is @jbenden

Updated - thanks!

[jefferyto]

Added devel/asu (ping @aparcar) because while the server itself uses Python 3, it also wants to setup an imagebuilder environment on target and so is pulling in Python 2. (This is mostly speculation though, I haven't looked too deeply into the details.)

If so, sounds like asu won't be resolved until OpenWrt core has moved to Python 3.

[smutt]

I'm currently rewriting Danish in Rust so there will be no Python 3 version.
https://github.com/smutt/danish-rust

I hope to be done with it by December and I'll call it Danish 2.0. The Python version will be abandoned and only the Rust version will be supported going forward.

The current Python 2 version worked fine for its purposes, and I've been running it for over a year on my home OpenWRT box. But it was an experiment in doing middle-box DANE for HTTPS that I now want to get more serious about and also bring to a more general Linux audience, not just OpenWRT. And for that I needed to re-implement in something like Rust or C/C++.

[jefferyto]

@smutt No problem - thanks for the update :smile:

[Andy2244]

Samba4.10.x will only fully support python3 from here on out, so if i get over this issue #8944 we will be fine.

[jefferyto]

@Andy2244 Thanks for the update!

[aparcar]

asu is python3 only, @dangowrt did the porting for various dependencies, however only Python3 if I recall correctly...

[jefferyto]

@aparcar So you're saying the +python-light in these lines (labelled IMAGEBUILDER_DEPENDS) should be Python 3 and not Python 2?

[aparcar]

Good point, I'm not sure if the ImageBuilder needs python2 or can handle also 3, the docs require python...

@dangowrt do you know if the ImageBuilder itself is already Python3 compatible? If not, that's kind of a new issue...

[danielfdickinson]

radicale 1.x is never going to Python3 -- radicale2 replaces radicale package and requires Python 3 -- the reason for renaming and having both in tree was to give current radicale users a chance to transition to radicale2 (requires a database conversion, which I think has to be done from radicale 1.x later version).

[jefferyto]

@cshoredaniel Thanks for the update - I guess I have two questions:

1. Is there any reason why a radicale 1.x user would want to stick with 1.x and not upgrade to 2.x?

2. Is there a way we can prompt 1.x users to upgrade to 2.x, before the 1.x package is gone (and they potentially lose the ability to convert their database)? Maybe log a warning to syslog on every startup? :laughing:

[jefferyto]

@cshoredaniel Wait, sorry I just checked the radicale package and there are Py2 and Py3 versions of the package?

If this is true, we can just remove the Py2 version and keep the Py3 version... But I guess the question still is, should we keep 1.x around?

[ignisf]

upmpdcli's songcast web interface should be compatible with python3. Will take a look at it and update.

[danielfdickinson]

@jefferyto Oh, you're right, I forgot there is a Py3 radicale, just not a Py2 radicale2...I think we should keep radicale 1.x for 19.0x but have a deprecation alert (actually there might already be such -- it's been a while since I last looked and I might be confusing with another distro).

[jefferyto]

@ignisf Thanks for the update!

[luizluca]

For hplip, it should be compatible with python3 since 3.15.2. I'll test, commit and report.

[luizluca]

@jefferyto , hplip is clean. It didn't actually need python for OpenWrt usage.

[jefferyto]

@luizluca thanks for the update!

[jefferyto]

I asked the folks at Tor about obfsproxy. It's not actively maintained anymore and obfs4proxy is the official successor, so I think this will be Won't Fix for obfsproxy.

[Andy2244]

Done for samba 4.10+ via #9440.

[jefferyto]

@Andy2244 thanks for the update!

[Andy2244]

Had to revert from 4.10 to samba 4.9, will try python3 with 4.11 again, so still open.

[MartB]

9579 also affects libtalloc which is a compile breaker

[neheb]

upmpdcli is almost done: https://github.com/openwrt/packages/pull/9687

[neheb]

node-mozilla-iot-gateway was removed from the tree.

[neheb]

@nxhack deals with node nowadays.

[jefferyto]

@neheb Thanks for the updates :+1:

[nxhack]

@neheb
The latest version of node.js(v12.x) is supporting python3.

But the v12.x series is still not stable, big endian support has been dropped, and cross-build support has been delayed, making it difficult to port to embedded systems.

It will take a little more time.

[jefferyto]

@nxhack Thanks for the update - v12 is scheduled to enter LTS status in October, do you think these issues can/will be resolved by then?

[nxhack]

@jefferyto
I'm working hard ;) on porting. However, MIPS version (including little endian) may disappear.

[neheb]

Do MIPS devices even have enough RAM to run node?

[nxhack]

it works but slow with extroot + swap.
without hardware fpu, super slow.

but it works excellent on armv6 armv7 with fpu.

[danielfdickinson]

See https://github.com/openwrt/packages/pull/9843 for a resolution to an issue with uwsgi-cgi -- it uses python for it's build system, and was relying on system python2 (if present) instead of (required) python3 (and it needs to the openwrt hostpkg python (for distutils), not system python3 or staging_dir_host/bin/python3 (because of missing distutils)).

[jefferyto]

@cshoredaniel Thanks for the update - I wasn't even aware that uwsgi-cgi relied on Python :joy:

[nunojpg]

Hi, I currently don't have a device with enough flash to run Python. Could someone test the Ndiff updated to Python3 from #10162?

[aTanW]

gammu package does not depend on python.

[neheb]

It has a conditional dependency on it.

From my reading of the git log, it seems it supports Python 3: https://github.com/gammu/gammu/search?o=desc&q=python&s=committer-date&type=Commits

[neheb]

Based on my reading of the aircrack-ng code, it doesn't care which version of python is used?

https://github.com/aircrack-ng/aircrack-ng/blob/32a9915b6873a3bcf2bdbb7116fd9fe01ab44154/build/m4/aircrack_ng_ext_scripts.m4#L44

[neheb]

Very unlikely that YunBridge will be ported: https://github.com/arduino/YunBridge/issues/31

[aTanW]

@neheb It has no python code in source tree. Docs may lag. Python bindings in separate package python-gammu in PyPI.

[jbenden]

Based on my reading of the aircrack-ng code, it doesn't care which version of python is used?

https://github.com/aircrack-ng/aircrack-ng/blob/32a9915b6873a3bcf2bdbb7116fd9fe01ab44154/build/m4/aircrack_ng_ext_scripts.m4#L44

The build scripts will accept any python version, and the setup.py might handle any version too. But a quick check shows that the actual tooling does not pass a py3 syntax check. I do think a PR was sent in that fixes the py3 syntax errors, though...

-Joe

[neheb]

@jbenden on a sidenote, that procps-ng dependency is partially bogus. I posted a patch that replaces ps + grep with only pgrep (BusyBox ps is missing functionality. pgrep works fine) but no progress has been made on that end: https://github.com/aircrack-ng/aircrack-ng/pull/2066

[commodo]

test

I forgot to mention this earlier.
I also don't have many routers at-hand to test stuff.
I usually spin-up an x86 VM and test some packages; admittedly some packages [like nmap] do require more setup than others, but some basic testing is doable via VM.

[Ansuel]

@jefferyto uwsgi port is done

[jefferyto]

@Ansuel thanks for the heads-up :+1:

[jefferyto]

@aparcar @nxhack @Shulyaka @jbenden @nunojpg @Andy2244 @commodo @mislavn @blogic:

Just a gentle reminder to update your package(s) from Python 2 to 3 (or to indicate your package(s)'s "Won't Fix" status). (If you are already working on this, then by all means please continue.) As you may know, Python 2 will reach end-of-life in 2 months.

(After these program packages are updated/resolved, we will also need to sort through all Python 2 library packages.)

[neheb]

I'll handle sysrepo here: https://github.com/openwrt/packages/pull/9491

Upstream has support for both versions.

[jbenden]

aircrack-ng's master repo now supports Py3. There is no planned release date, yet.

[commodo]

@jefferyto
Apologies for the silence on this.

Regarding seafile, that is a bit on-hold.
The suite is a bit of a monster [in terms of Python code-size], and they haven't done any release with Python3; there is some code converting it to Python3, but it's still in progress. I tried to update it to Python3 myself, but that was more work than I like [I wasted 3-4 days trying to convert it to Python3].

seafile is bit of a pain for me, also because of Django 1; while Django LTS is currently at version 2, they still use Django 1. At least I found the elegant solution to split django into 2 packages [ django1 & django] and limit django1 to Python2, and make django Python3 only.

So, for now, what's done, is that all seafile deps both support Python 2 & 3.
I'll see about maybe trying to rework the entire Seafile package to be easier to maintain; this is mostly with the intent of converting it to Python3.
Hopefully they'll take my PRs; I opened one a few months ago and it's still open with no comments.

[jefferyto]

@neheb Thanks for the update!

@jbenden I was going to ask if it's possible to backport the changes, but I see their last release was from last December and master is more than 500 commits ahead... Would packaging from master be an option?

@commodo I see they have merged PRs for Python 3 (haiwen/seafile-server#238, haiwen/seahub#4076, haiwen/ccnet-server#35) but they are all for master, while their current releases (I think) are from the 7.0 branches. Their roadmap has the Python 3 migration for 7.1 in December, so perhaps we will have to wait and see.

For haiwen/seahub#3607, they may not be concerned with cross-compiling, since they distribute their own binaries. It may be a patch we carry ourselves.

[commodo]

@commodo I see they have merged PRs for Python 3 (haiwen/seafile-server#238, haiwen/seahub#4076, haiwen/ccnet-server#35) but they are all for master, while their current releases (I think) are from the 7.0 branches. Their roadmap has the Python 3 migration for 7.1 in December, so perhaps we will have to wait and see.

yep; i was also thinking about waiting for a new release [which includes Python3 support]; i wanted to do an upgrade to 7.0 [a few weeks ago], but did not see the benefit it if I can't switch to Python3;

well, the thing is i tried to see about maybe backporting Python3 to 7.0 and got annoyed, so, decided not to do it just yet;

hopefully i'll have some time to clean the packaging a bit;
the way it's packaged in OpenWrt now, is a bit better than before, but I feel I would like to re-visit it; it feels like the whole thing [packaging seafile] was rushed when it was added;

[neheb]

sysrepo should be good to go.

[jbenden]

@jefferyto The changes should not be back-ported, yes. There has been talk of pushing out a new release for months, but hasn't gone anywhere - yet. If possible, I think it should be packaged from a known working commit, directly from GitHub. Thoughts?

[jefferyto]

@jbenden Sounds good - I guess the question is how to find a commit that is known good.

[tiagogaspar8]

OK, just an update, I saw from the makefile that asu is already suing python3.
Also, shouldn't mailman be updated to version 3?
My question is: when january comes will the python package removed? If so what happens t the other packages?

[jefferyto]

@tiagogaspar8

OK, just an update, I saw from the makefile that asu is already suing python3.

asu still depends on python-light.

Also, shouldn't mailman be updated to version 3?

That depends on the maintainer (@Shulyaka) or someone else contributing the update.

My question is: when january comes will the python package removed? If so what happens t the other packages?

Please see https://github.com/openwrt/packages/tree/master/lang/python#python-2-end-of-life.

[neheb]

According to: https://gitlab.com/mailman/mailman/blob/master/src/mailman/docs/install.rst , Python 3 is supported and Python 2 is not.

edit: Ah I see. Mailman needs to be transitioned to version 3.

[tiagogaspar8]

asu still depends on python-light.

You're right! Sorry.

That depends on the maintainer (@Shulyaka) or someone else contributing the update.

I'd do it but I have no idea how.

edit: Ah I see. Mailman needs to be transitioned to version 3.

Yeah ahahaha I hope someone can do it until january.

[Andy2244]

Quick update regarding samba4, there is now a official patch that solves the main issue that prevented me to switch to the 4.11 branch aka python3. I think i can work around, provide my own patches for the remaining issues.

[Andy2244]

Ok samba4 is done via #10781

PS: updated PR

[commodo]

I'll reference some parts from this comment https://github.com/openwrt/packages/pull/10952#issuecomment-571480826

I've re-adapted some of the notes from that comment here.

Regarding Seafile:

• it has released version 7.1.1 which seems to support Py3
• 
  
  • it looks like the 2to3 tool was run on some libs, and that broke some Py2 compatibility;
  
  
• so, it looks like Seafile 7.1.1 needs to be switched to Py3
  
  
  
  • which also needs python-django1 to become python3-django1 [no idea if it it's worth adding a temp python3-django1 alongside python-django1
  
  
  • this also means that all Django packages [compressor, appconf, etc] ,need to drop Py2 variants; well, we could keep them, but if only Seafile needs them I am not sure about keeping them
  
  

In any case, I have this branch WIP https://github.com/commodo/packages/commits/seafile-update3
I'll see about maybe splitting some commits or just having them as they are there, no idea.....

Upgrading some packages now (particularly gunicorn, openpyxl & pillow is stuck because they start to use Py3 lang constructs.
Example:

guessing install --prefix="/usr" --root="/home/sandu/work/openwrt/django-update/build_dir/target-i386_pentium4_musl/pypi/pillow-python/Pillow-7.0.0/ipkg-install" --single-version-externally-managed ;
  File "./setup.py", line 304
    yield from self.features

SyntaxError: invalid syntax

Something similar is for gunicorn, and I forgot what it was for openpyxl.
Typically these show up in bytecode compilation.

For gunicorn 19.0.0, I think I patched something to make it skip the file. For 20.0.0 [to be upgraded] I think I would need to patch/skipped another file. And for Pillow I would also need to patch/skip.

So, now, I feel like I have one of 2 options:

1. Defer a lot of Python package upgrades until later [end-of-March or whenever]
2. Continue upgrading and drop some Py2 variants where possible/needed/the-case

I prefer 2, but that also brings a whole chain of deps, so it's not always easy what to skip/upgrade or not.

[jefferyto]

@Andy2244 Forgot to say thanks for your efforts :+1: :joy:

@commodo Thanks for the update - I've posted some comments to #10952.

@jbenden There should be a release of aircrack-ng "very soon" (aircrack-ng/aircrack-ng#2097).

@Shulyaka @blogic Any updates? Your packages will be removed after March if they are not updated to work with Python 3.

[jefferyto]

Added iotivity to the list - I missed #9727 being merged :man_facepalming:

@hauke Do you intend on removing the package?

[hnyman]

@jefferyto @commodo
python2 mass removal nearing us?

From lang/python README:

A mass removal event ("The Snap") will occur on 31 March 2020, or 2 weeks before the freeze for a 20.x release, whichever is sooner. The exact date will be confirmed when the 20.x release schedule is known, or by 15 March 2020.

All Python 2 packages (the Python 2 interpreter, programs that depend on Python 2, and Python 2-only libraries) will be removed during this event.

[commodo]

I'll let @jefferyto take the lead on this one.
I'm slightly more present these days.
So, I could probably help out a bit more.
Given the recent pandemic stuff, I'm doing work-from-home, so I get fewer interruptions.

Also, the details of the mass removal are not that important to me [i.e. when we do it, or how we do it].

[jefferyto]

Yes, still happening :joy: I'm not sure if it will be 31 Mar or 1 Apr, but basically the plan is:

• Open several giant PRs (one at a time) to remove packages, each one removing packages with no Python 2 dependants (including the programs listed in this issue that are marked Won't Fix or appear to be abandoned by their maintainers, i.e. mailman, iovitity, yunbridge)
• At the same time, opening PRs in openwrt/packages-abandoned to add those packages (but under a python2 namespace/prefix, to avoid conflicting with python which we may decide to use again here)

AFAIK nmap is the only outstanding package that needs work (#10162), I'm going to try to get to it soon.

[jefferyto]

A brief update: I'm still working on the PR (I think one giant PR will do), but it may take a day or two more than expected (due to a combination of large number of packages that need editing and compile testing, and laziness :joy: )

[hnyman]

Might be better to do it in 2-3 phases, so that there would be less testing for circleci at the same time.

If you have packages ready for the action, just submit the first batch.

Giant PRs sound always dangerous ;-)

[jefferyto]

@hnyman You're right - funny how something can seem like a good idea until you say it out loud :joy:

[jefferyto]

The transition for programs is complete so I'll close this issue. Thanks to everyone for your kind assistance :pray: :joy:

[jefferyto]

Hmm I guess closed issues can stay pinned. @neheb @hnyman can you help unpin this issue?

[hnyman]

We could leave it pinned for a while.
I am quite sure that there will be some questions, as not all casual users have been aware of the removal plan.
Keeping it pinned for a few weeks might reduce unnecessary issues being opened.

[jefferyto]

Sure - sounds good :joy:

[hnyman]

Apparently no major breakage from the python2 removal yesterday.

I looked at the faillogs of the first package architectures that buildbot has built after the removal (e.g. mips_24kc): https://downloads.openwrt.org/snapshots/faillogs/mips_24kc/

There seems to be only some breakage in the telephony feed:

• kamailio-5.x, maintained by @jslachta and @micmac1 , apparently still tries to use python2
• freeswitch-stable, maintained by @micmac1 , also tries to use python2

Both fail with almost identical error:
Makefile:235: /builder/shared-workdir/build/sdk/feeds/packages/lang/python/python-host.mk: No such file or directory

Otherwise it seems to be the usual suspects, not related to python2 removal.

[jslachta]

@hnyman, I will have an update to kamailio-5.x within a few days with a complete removal the python2 usage. Thank you for notifying me.

[commodo]

onet thing i noticed real quick
we'd probably need a new big-sweep to remove PACKAGE_python3-xxxx cond-deps;

i didn't like them much and if we can remove it, i would

[hnyman]

@jslachta

I will have an update to kamailio-5.x within a few days with a complete removal the python2 usage. Thank you for notifying me.

Great, as there is already a new bug filed about that, as users run into that warning regularly.
https://bugs.openwrt.org/index.php?do=details&task_id=2982

[jslachta]

@hnyman I have updated the kamailio-5.x package in master branch. Since the python support in kamailio was not really used, I completely removed the python support. I am going to reintroduce it later.