Homebrew-core: Formulas still using Python 2

[fxcoudert]

Python 2.7 is going away soon… and we still have 61 formulas using it (out of 389 Python formulas). We need to start thinking about migrating them, or dropping them.

Below the formulas are sorted by install count for 30 days, so we can prioritise our work. Some formulas were audited in the past (I added a does not support Python 3 comment), but not all. Some may have newer versions compatible with Python 3.

• [ ] python@2 (75,761): See WIP #47750

Top formulas

• [x] node@10 (17644) #47836
• [x] py2cairo (13674): #47831
• [x] node@8 (8389): ☠️ EOL at end of year #47750
• [x] pygobject (4841): ☠️#47750 dead with pygtk
• [x] [email protected] (1750): #47840 needed for gnuradio and opencv@2
• [x] opencv@2 (1421): #47832
• [x] vte (1327): ☠️ dead with pygtk #47750
• [x] terminator (1327) ☠️ dead with pygtk #47750
• [ ] offlineimap (720): #47834
• [x] gnuradio (716): we tried to update it twice, but failed: #44870 #43753 -> Will use system Python for the time being: #47871

High usage

• [x] mysql-utilities (454) ☠️ EOL: https://downloads.mysql.com/archives/utilities/ (users are encouraged to migrate to MySQL Shell.) #47750
• [x] volatility (404): #47870
• [x] ledger (304): can be built without Python https://github.com/Homebrew/homebrew-core/pull/47053
• [x] pwntools (289): #47873
• [x] [email protected] (264)
• [x] uwsgi (262)
• [x] metricbeat (190) #47985
• [x] molecule (178) #48409, #47933
• [x] [email protected] (149)
• [x] gwyddion (133)
• [x] ipython@5 (114) ☠️ [upstream support ending at the end of 2019]: (https://github.com/Homebrew/homebrew-core/pull/47511#issuecomment-562813166) Will be removed with python@2 #47750
• [x] [email protected] (111) ☠️ #48082
• [x] sslyze (105) update currently failing https://github.com/Homebrew/homebrew-core/pull/45750. Will use system python for the time being: #47904
• [x] mesos (105) #47986
• [x] audacious (98) #47988
• [x] bup (88) #47592
• [x] [email protected] (82) #47953
• [x] ooniprobe (62) #47989
• [x] subliminal (61) #48025
• [x] pdf-redact-tools (56) #48026 No commits since Mar. 2018. Project seems dead. https://github.com/firstlookmedia/pdf-redact-tools/pull/34.

Lower usage

• [x] auditbeat (49) #48040
• [x] [email protected] (48) #47952
• [x] charm-tools (46) #47275
• [x] dnsviz (42)
• [x] docker-cloud (30) ☠️ #47752
• [x] dxpy (29)
• [x] git-remote-hg (28) #48348
• [x] mkvtomp4 (26) #48328
• [x] aap (25) ☠️ #47745
• [x] cvs2svn (23) ☠️ #47744
• [x] closure-linter (23) ☠️ #47741
• [x] llnode (22) #47740

Stragglers

• [x] git-plus (20) #47830
• [x] archivemail (19) ☠️ #47700
• [x] appscale-tools (19) #47951
• [x] osc (18) #42510
• [x] headphones (18) ☠️ #48087
• [x] apm-server (18) ☠️ #48085
• [x] bubbros (17) ☠️ #47701
• [x] ddar (16) ☠️#47702
• [x] curlish (16) ☠️#47703
• [x] bzr-fastimport (16)☠️ #47704
• [x] blastem (15) #47270
• [x] tnote (14) ☠️ #47705
• [x] redo (14) ☠️ #47750
• [x] nicotine-plus (13) ☠️ #47748
• [x] hg-flow (12) ☠️ #47750
• [x] ino (11) ☠️ #47699
• [x] euca2ools (11) ☠️ #47698
• [x] logentries (10) ☠️ #48084
• [x] muttils (9) ☠️ #47697

[Bo98]

Pinging @chrmoritz since he knows a bit about the node@10 situation.

[Bo98]

On the topic of node@8, I can already say to remove that with Python 2. They share an EOL date.

[fxcoudert]

Pinging @tschoonj for his knowledge on the GTK world, and apps like terminator and vte, which are among the most-used in this list

[tschoonj]

I don't think terminator has properly worked since we migrated to the Gtk quartz backend many years ago, because vte doesn't seem to support it properly. So probably these two are ripe for deletion.

Deleting pygtk would make sense, seeing as it has been unmaintained for many years, and people are supposed to use gobject-introspection instead. I guess this would give rise to complaints though 😄

[Bo98]

py2cairo

We have py3cairo, so that will be an eventual delete.

py3cairo could also be renamed to pycairo.

[bayandin]

osc and dnsviz switched to Python 3 in https://github.com/Homebrew/homebrew-core/pull/47052

[bayandin]

Uhh, I wish this done after Python 3.8

[bayandin]

ipython@5 — the last version with python2 support, so it's ok to delete them together.

[fxcoudert]

@bayandin there is no hurry, we can put it on hold

[chrmoritz]

Regarding the nodesituation:

• node and node@12 are compatible with Python 3 without patching (pending v13.2.0 in #47042)
• node@8 isn't worth fixing because it will be EOL together with Python 2 at the end of the year
• node@10 is more difficult, because it has high install count and will be supported until April 2021
  
  
  
  • I don't think we will get upstream Python 3 supported because the bundled V8 version it uses is too old for Python 3 support => we would need to patch it ourself
  
  
  • I was able to make configure Python 3 compatible my applying a couple of upstream patches (most of them apply cleanly, some required some trivial backporting): https://github.com/chrmoritz/homebrew-core/commit/0e5c670beafe65f1284b6d998ef00c01554a3901 (mostly the same patches that landed in v12.13.1 and some older ones)
  
  
  • Then I tried to make building with make Python 3 compatible, but unfortunately the outdated V8 version in node@10 hast a lot of scripts used during the build which are not Python 3 compatible (and we can't simply backport the whole V8 upgrade here obviously)
  
  
  • I tried to copy the newer Python 3 compatible scrips over, but unfortunately there were some interface changes making them not compatible with the rest of the older codebase / the gyp files, progress that far can be found
  
  
  • So we would need to just backport the Python 3 compatible fixes from V8 here, which is still quite a bit of work to do
  
  

[Bo98]

node and node@10 are compatible with Python 3 without patching

* node and node@12?

[xu-cheng]

FYI, the list is not completed. You are missing the following formulae:

$ rg 'uses_from_macos "python@2"'
Formula/ginac.rb
17:  uses_from_macos "python@2"

Formula/blockhash.rb
18:  uses_from_macos "python@2"

Formula/aws-elasticbeanstalk.rb
16:  uses_from_macos "python@2"

Formula/bazaar.rb
16:  uses_from_macos "python@2"

Formula/fail2ban.rb
15:  uses_from_macos "python@2"

Formula/autojump.rb
16:  uses_from_macos "python@2"

Though they have no effect on macOS, you would still need to address them somehow.

[EricFromCanada]

auditbeat/metricbeat: migration in progress at https://github.com/elastic/beats/pull/14798
charm-tools: migrated in #47275

[issyl0]

The bazaar VCS (with its uses_from_macos python@2 in our formula) doesn't support Python 3. It hasn't had any updates from Canonical since version 2.7.0 (the current in Homebrew) which was released in 2016. There's a fork called breezy that works with Python 3.

[dawidd6]

There is also asciidoc which uses system python2.

[alebcay]

After discussion in https://github.com/Homebrew/homebrew-core/pull/47511, looks like ipython@5 will be up for deletion at the end of the calendar year as upstream will be dropping support.

[iMichka]

Please update the list from @fxcoudert whenever possible.
All the formulae marked with a ☠️ will be removed on 01.2020 in the same PR as python@2.

[iMichka]

Regarding PyGTK:
https://github.com/NixOS/nixpkgs/issues/75478

PyGTK is by all means a dead project.

I read some stack overflow questions, and it looks like the deprecation for pygtk has started somewhere in 2011 already ...

This would mean we could get rid of the following formulae:
pygtk diffuse gtk-mac-integration nicotine-plus pygtkglext pygtksourceview terminator vte zim

This would simplify the Python 2 EOL for some of the top formulae in our list.
The issue is the high number of downloads for PyGTK, I fear that we may break a lot of workflows ...
install: 4,802 (30 days), 15,253 (90 days), 59,294 (365 days)

@Homebrew/maintainers any opinion on this?

[EricFromCanada]

terminator has a Python 3 milestone, but I'd be surprised if they got a release out before the end of the year.

[iMichka]

I think that we have two options:
1) Let the full pygtk diffuse gtk-mac-integration nicotine-plus pygtkglext pygtksourceview terminator vte zim stack depend on system Python 2. This leaves the door open for terminator to migrate (though it looks like this may take some time)
2) Add this list of formulae to #47750 and delete everything with Python@2. pygtk is dead and unmaintained, so we will remove it one day or another anyway ...

[SMillerDev]

I'd say it's easy enough to add a formula once it meets the standards again. 🚮 for me

[tschoonj]

pygtk is only an optional dependency of gtk-mac-integration, so I would recommend to just drop pygtk from the list of dependencies.

I would vote to delete the pygtk formula. People should really not be using this anymore.

[MikeMcQuaid]

2. Add this list of formulae to #47750 and delete everything with Python@2. pygtk is dead and unmaintained, so we will remove it one day or another anyway ...

This seems reasonable to me. It's going to be less painful for people to have these dropped in one than a few different times.

pygtk is only an optional dependency of gtk-mac-integration, so I would recommend to just drop pygtk from the list of dependencies.

👍

[iMichka]

I was not able to get gtk-mac-integration going with Python 3 and pygobject3. It always tries to fall back to system python 2. Does somebody want to have a look? I'll proceed with the other packages. Else, gtk-mac-integration may be deleted too during the Python@2 removal.

[tschoonj]

@iMichka See #47799

[iMichka]

Here is the srategy I have used for all the changes done until now:

• Stuff that is completely unmaintained (no new releases in the last 2 years) and has a low donwload count: deleted right away.
• Stuff that is maintained and has a low download count: I opened an issue upstream and will give them a few more days to update: if this does not work, we will migrate these to system Python 2 (uses_from_macos "python@2")
• Stuff from the pygtk stack: pygtk diffuse pygtkglext pygtksourceview terminator vte zim and EOL stuff like node@8 are added to the python@2 deletion PR: #47750, and will be deleted all at once.
• Remaining stuff with high download count is migrated to uses_from_macos "python@2", which gives these package some more months to live on mac (node@10 py2cairo [email protected] opencv@2 offlineimap ...), but eventually these will be deleted too at one point

I must say that some packages are in a sort of grey area. I hope I did not do too many mistakes ... but all of these had 11 years to prepare. We need to draw a line somewhere.

[MikeMcQuaid]

• Stuff that is completely unmaintained (no new releases in the last 2 years) and has a low donwload count: deleted right away.

I would personally say this should only happen if it has a zero download count or is broken.

• Remaining stuff with high download count is migrated to uses_from_macos "python@2", which gives these package some more months to live on mac (node@10 py2cairo [email protected] opencv@2 offlineimap ...), but eventually these will be deleted too at one point

I would say these get deleted when macOS drops Python 2.

[iMichka]

Stuff that is completely unmaintained (no new releases in the last 2 years) and has a low donwload count: deleted right away.

I would personally say this should only happen if it has a zero download count or is broken.

We do delete formulae on a regular basis which do not have a zero download count, and which are sometimes not broken. EOL formulae for example. And I often see things being deleted from homebrew-core, even if in principle we could put more effort to maintain these. It is hard to have a clear guideline here.

I do not believe I took too many risks deleting these formulae. They all had 11 years to be ready, and looked really dead to me. If anybody opens an issue because their favourite tool is gone, I'll add it back.

We do not have a proper deletion process, but this is maybe something we should discuss somewhere else, as this is not solely related to Python 2, and is more a general topic.

Remaining stuff with high download count is migrated to uses_from_macos "python@2", which gives these package some more months to live on mac (node@10 py2cairo [email protected] opencv@2 offlineimap ...), but eventually these will be deleted too at one point

I would say these get deleted when macOS drops Python 2.

Agreed. Not sure when this will be, but I hope it will not take 10 more years. node@10 will be EOL mid-2020 anyway. The others may remain though because I do not know if they have a formal EOL deadline.

I don't know why you closed this issue? I would really like to finish off the list.

There should be only 2 things remaining here:

• either the formula will use uses_from_macos "python@2
• or it will be deleted on 01.01.2020 with this PR: #47750

[chrmoritz]

node@10 will be EOL in April 2021 (and not 2020), so we might get a problem here too, if Apple decides to drop Python 2 in macOS 10.16. However I've not completely given up patching in Python 3 compatibility yet.

Also whats the plan with the python symlink after python@2 gets removed? Should the python symlink (pointing to Python 3) stay hidden in libexec or should it be normally linked to HOMEBREW_PREFIX together with python3?

[jonchang]

Also whats the plan with the python symlink after python@2 gets removed? Should the python symlink (pointing to Python 3) stay hidden in libexec or should it be normally linked to HOMEBREW_PREFIX together with python3?

Unfortunately PEP 394 does not seem provide much guidance on this, only that it should point to the same thing as either python2 or python3.

I think pointing python to python3 is only sensible once macOS no longer ships system Python 2, to avoid shadowing the system version.

[iMichka]

Yes, we will strictly follow PEP 394. Last time we tried to do something else, there was a juge backslash from the community. So python will "never" point to python3. Unless upstream decides to review the PEP.

So if Python 2 is removed from macOS, you will have no python executable anymore. (unless Apple decides something else, who knows ...).

I personally disagree with the PEP, and think they should have given more details on what happens after 2020. But we will follow it as we have no other choice.

[fxcoudert]

Closing the tracking issue. Thanks everyone!

[iMichka]

offlineimap is still open #47834

[fxcoudert]

Yeah, you can reopen if you feel like it, but it seems a bit overkill to have a tracking issue for one open PR :)

[iMichka]

Forgot to say that I was OK to close this issue :)