Homebrew-core: Python 3.9 Migration

[Rylan12]

Python 3.9 has been released and added to homebrew-core!

Currently, there are 42 formulae in homebrew-core that depend on [email protected]. The next step in the migration process is to switch as many formulae as possible to depend on the new [email protected] formula.

See https://github.com/Homebrew/homebrew-core/issues/62201#issuecomment-715182529 for a list of formulae to be upgraded.

No need to ask before working on this. If there's a formula that you can update, go ahead and open a PR and we'll get to it as soon as we can!

---

This is a great first issue to work on if you're new to Homebrew and if you're participating in Hacktoberfest, this is a great way to knock out some genuine PRs that will be greatly appreciated!

Here are the basic steps to migrate a formula to Python 3.9:

1. Change depends_on "[email protected]" to depends_on "[email protected]"
2. Change any other occurrences of [email protected] to [email protected]
3. Add a line with revision 1 (if the formula already has a revision, increment the number by one)
4. Test the change locally by running brew install -s <formula>, brew test <formula> and brew audit --strict <formula>
   
   
   
   • Try to fix any failures locally first. If you need help, feel free to open a PR, but please be prepared to continue to work on it.
   
   
5. Open a PR with the name: <formula>: migrate to [email protected]
6. Done!

If you're a new contributor and want to help out, make sure to read our countributing guidelines and check out How To Open a Homebrew Pull Request in our documentation

[mayank]

@Rylan12 can I pick this one? Please assign to me!

[Rylan12]

@mayank thanks for your interest! Please feel free to work on this!

This is too big of a project for one person to tackle on their own, but feel free to open a PR to update one or a few formulae.

[LSmith-Zenoscave]

@mayank and @Rylan12 I would like to help if at all possible since this could be quite an undertaking.
Feel free to assign me if that's ok!

[mayank]

@Rylan12 I need to install all the formulas on mac? Is there anything like docker for this?
I can write some testing script which will run all at once.

[fxcoudert]

There are 234 formulas that are independent: they depend on [email protected] only directly, they have not reverse dependencies. I will deal with those first.

[Rylan12]

@LSmith-Zenoscave and @mayank, this is not an issue that will be closed with one PR that updates all 400 formulae. The best way to do it is just to make a few PRs, each with one formula. Take a look at some of the PRs that have been made so far if you need a model for what we're looking for.

If you're not already a Homebrew user this probably isn't a great issue for you. If you want to get more involved, though, feel free to check out some of the links I've added above for information about how to get set up.

[Rylan12]

@fxcoudert: can you post the list of independent formulae (maybe link a gist?) so others can help out with those?

[Rylan12]

FYI to anyone looking to help out here: The current virtualenv version we're using doesn't seem to support Python 3.9 so we should hold off on updating lots of formulae until that issue's been resolved.

Edit: I've started a fix in https://github.com/Homebrew/brew/pull/8873

[fxcoudert]

The list of independent formulas is below. If you don't mind, I'm happy to open PRs for all of them once virtualenv is fixed: it will literally take 2 minutes, the script is in place, and is already tracking locally which ones I have submitted or not.

There are plenty of other small "clusters" of formulas that can be migrated and where crowd-sourcing will be necessary.

ansible-lint
asciinema
atdtool
athenacli
auditbeat
austin
autopep8
aws-elasticbeanstalk
aws-google-auth
aws-shell
awscli@1
awscurl
awslogs
awsume
azure-cli
b2-tools
bagit
bandcamp-dl
beancount
bedtools
bento4
binwalk
black
borgmatic
buku
bumpversion
c7n
ccm
certbot
cfn-lint
charm-tools
choose
cloudformation-cli
coconut
codemod
codespell
commitizen
conan
cookiecutter
cppcheck
cppman
credstash
csvkit
csvtomd
ctemplate
datasette
ddgr
deheader
diceware
distcc
dnstwist
docker-compose
docker-squash
doitlive
dxpy
eg-examples
esptool
euler-py
eye-d3
fabric
fades
fava
fbi-servefiles
flake8
flawfinder
flintrock
flit
fonttools
fpp
gandi.cli
gcovr
gimme-aws-creds
ginac
git-filter-repo
git-plus
git-remote-codecommit
git-review
git-revise
gitup
glances
global
googler
goolabs
grc
grip
hashpump
heartbeat
homeassistant-cli
honcho
howdoi
hsd
httpie
hy
i386-elf-gdb
instalooter
internetarchive
iproute2mac
jailkit
jc
jenkins-job-builder
jinja2-cli
jrnl
keepassc
keepkey-agent
khal
khard
legit
liblouis
liblouis
libtcod
link-grammar
liquidctl
litecli
livestreamer
lizard-analyzer
localstack
mackup
macvim
magic-wormhole
mdv
metricbeat
micropython
mitie
mkdocs
mongo-orchestration
mps-youtube
mycli
nbdime
nicovideo-dl
node@12
notifiers
nyx
oci-cli
openstackclient
osc
packetbeat
pass-git-helper
passpie
pastebinit
percol
peru
pgcli
pipgrip
pipx
pius
platformio
poetry
ponysay
pre-commit
proselint
pssh
pwncat
pwntools
pygitup
pygments
pyinstaller
pyinvoke
pylint
python-markdown
python-yq
pyvim
q
rabbitmq
rbtools
recon-ng
redo
remarshal
reorder-python-imports
restview
robot-framework
rom-tools
rst-lint
rtv
s3ql
sceptre
scour
shallow-backup
shyaml
singular
snakemake
spoof-mac
sqlite-utils
sqlparse
ssh-audit
sshuttle
sslmate
statik
stone-soup
stormssh
streamlink
subliminal
supervisor
svtplay-dl
sync_gateway
tarsnapper
termius
termtosvg
thefuck
theharvester
todoman
toot
tox
trailscraper
translate-toolkit
trash-cli
trezor-agent
tvnamer
twarc
twine-pypi
twoping
twtxt
txt2tags
unoconv
vdirsyncer
vim
vpn-slice
vsts-cli
wakatime-cli
watchman
watson
weboob
whatmp3
x86_64-elf-gdb
xonsh
xxh
yapf
ydcv
ydiff
yle-dl
yosys
you-get
zabbix-cli
znc

[Rylan12]

Thanks, @fxcoudert. I've removed the community help tags for now. I jumped the gun a little bit—let's deal with the first round internally and then I'll reopen to the community.

I see that PRs are being generated, what's the plan for those? Mass approval of the ones that pass CI?

What's the plan for the virtualenv change now that the fix has been merged? Are we going to wait until Monday for a new tag or move ahead?

[fxcoudert]

The PRs should not be merged before the next Brew tag ships, which is expected on Monday (2020-10-12). That will give them time to go through CI, and we can see how many break, and fix them. After Monday, I welcome review and approval of the PRs.

[Rylan12]

The PRs should not be merged before the next Brew tag ships, which is expected on Monday (2020-10-12). That will give them time to go through CI, and we can see how many break, and fix them. After Monday, I welcome review and approval of the PRs.

Got it, thanks.

[fxcoudert]

Brew 2.5.4 is tagged. Feel free to review / accept the pull requests.

[LSmith-Zenoscave]

There are 234 formulas that are independent: they depend on [email protected] only directly, they have not reverse dependencies. I will deal with those first.

Can you share how you came up with those numbers? I found a similar list but with 144.
I just looked for any files that depends_on "[email protected]" with no other deps; call it deps_on_py
Then found any files that were in a depends_on; call it any_deps_on
If there are 144 files that have a depends on python3.8 and only python3.8 are not depended on from my findings.

Perhaps I had some missed?
I can add my script to generate these if needed

---

edit made to adjust correction in my findings (forgot quote in a grep)

[fxcoudert]

• Start with the list of formulas that directly depend on [email protected]
• Remove those that are a dependency in another formula
• For each remaining formula, use brew deps --tree --include-build --include-test to remove those that if they also depend indirectly on [email protected]

[NicoHaase]

I'm not really sure if this is connected, but could you have a look at https://github.com/Homebrew/discussions/discussions/37 and check whether upgrading to Python 3.9 broke docker-compose?

[fxcoudert]

There's a label, and help is welcome on any relevant PR. I don't think there is much use for this meta-issue, maybe we should close it?

[cclauss]

I missed the label... Thanks!

[Rylan12]

There's a label, and help is welcome on any relevant PR. I don't think there is much use for this meta-issue, maybe we should close it?

Are we definitely going to do the mass migration in https://github.com/Homebrew/homebrew-core/pull/62560? If so, yeah I think this can be closed. If that's not going to work, there are still almost 200 formulae to be migrated so it might be worth keeping open in that case.

[alexchandel]

Sounds like some of glib gobject-introspection pyqt protobuf bazel libtorch [email protected] gdal vtk ansible emscripten gnuradio kibana libtensorflow libtensorflow@1 mame nest nwchem opencv opencv@3 semgrep wxpython will be done separately.

This may be controversial, but I hope this is complete before 3.10 is released.

[xxyzz]

I wanna ask a dumb question: is it OK to upgrade python, python3 and python@3 aliases to [email protected]? I guess it won't affect other formulas.

[iMichka]

This question is best asked on https://github.com/Homebrew/discussions/discussions, but yes you can update your own alias if you want to.

[fxcoudert]

This is the list of formulas that remain to be migrated to Python 3.9. Almost all of them are independent, and should already have a PR open to which volunteers can contribute:

• [x] afflib (#63357)
• [x] airshare (https://github.com/Homebrew/homebrew-core/pull/62208)
• [ ] auditbeat (#63381)
• [ ] azure-cli
• [ ] binwalk (https://github.com/Homebrew/homebrew-core/pull/62260)
• [ ] charm-tools (https://github.com/Homebrew/homebrew-core/pull/62269)
• [ ] distcc
• [ ] fail2ban
• [x] fdroidserver (https://github.com/Homebrew/homebrew-core/pull/63174)
• [ ] filebeat (#63381)
• [ ] gdb (https://github.com/Homebrew/homebrew-core/pull/62792)
• [ ] gdbgui (https://github.com/Homebrew/homebrew-core/pull/62792)
• [ ] gjs (#63464)
• [ ] global (https://github.com/Homebrew/homebrew-core/pull/62319)
• [ ] heartbeat (#63381)
• [ ] i386-elf-gdb (https://github.com/Homebrew/homebrew-core/pull/62332)
• [x] jrnl (#63448)
• [ ] keepassc (#62341)
• [ ] lc0
• [x] ldns (#63086)
• [ ] mdv (#62357)
• [ ] mypy
• [x] nicotine-plus (#63397)
• [ ] onnxruntime
• [ ] packetbeat (#63381)
• [ ] passpie (#62375)
• [x] s3cmd (#63174)
• [ ] s3ql
• [ ] salt
• [ ] sceptre (#62412)
• [x] supervisor (#62429)
• [x] thefuck (#62436)
• [ ] theharvester (#62437)
• [ ] twtxt (#62449)
• [ ] x86_64-elf-gdb (#63422)

[fxcoudert]

Also part of the migration are test and audit failures that should be investigated and fixed:

10.15

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit csound --online --git --skip-style
brew audit cxxtest --online --git --skip-style
brew audit dnsviz --online --git --skip-style
brew test --verbose gnuradio
brew audit mitmproxy --online --git --skip-style
brew audit mkvtomp4 --online --git --skip-style
brew install --verbose --build-bottle redex

10.14

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit anime-downloader --online --git --skip-style
brew audit cmark-gfm --online --git --skip-style
brew audit cmark --online --git --skip-style
brew audit code-server --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit csound --online --git --skip-style
brew audit gnuradio --online --git --skip-style
brew test --verbose gnuradio
brew audit gprof2dot --online --git --skip-style
brew audit rdiff-backup --online --git --skip-style
brew install --verbose --build-bottle redex
brew audit redex --online --git --skip-style

10.13

brew audit vapoursynth --online --git --skip-style
brew audit gtk-doc --online --git --skip-style
brew audit uhd --online --git --skip-style
brew audit cryptominisat --online --git --skip-style
brew audit genometools --online --git --skip-style
brew audit git-cola --online --git --skip-style
brew audit gitfs --online --git --skip-style
brew audit gitless --online --git --skip-style
brew test --verbose gnuradio
brew audit libproxy --online --git --skip-style
brew fetch --retry pytouhou --build-bottle --force
brew install --verbose --build-bottle pytouhou
brew install --verbose --build-bottle redex

[tieugene]

I don't know what is the source of problem (migrating to py39, brew bug/feature or something else), but as for newly (from scratch) installed homebrew and [email protected] on it:

1. In RTFM:

Unversioned symlinks for python, python-config, pip etc. are installed here:
$(brew --prefix)/opt/python/libexec/bin

There is no $(brew --prefix)/opt/python/ directory

1. brew info [email protected]:
   ...
   Python has been installed as
   /usr/local/bin/python3

No, there is no /usr/local/bin/python3

1. brew switch is useless

I don't know how to work with this "python3".
To call it as "/usr/local/Cellar/[email protected]/3.9.0/Frameworks/Python.framework/Versions/3.9/bin/python3.9" is too complex.

[fxcoudert]

@tieugene the [email protected] binary currently resides in /usr/local/opt/[email protected]/bin. We'll be migrating to that as our primary formula soon, and then it will be in /usr/local/bin/python3 (which means our documentation is currently wrong, indeed).

[shinysuittheory]

I've been trying to reinstall mpv from source so as to get the app bundle for macOS, similarly to the closed and now locked "Missing mpv.app bundle #46532", but keep getting an error which I believe has something to do with python but I can't figure out what. Can anyone shed some light? Wasn't sure if to create a separate issue

link to error text from terminal - https://pastebin.pl/view/a5a1e0d0

[SMillerDev]

You should ask the mpv developers, homebrew doesn't provide support for modified builds.

[fxcoudert]

@hjmallon please open a new issue and provide all information requested, so we can help you

[fxcoudert]

@Raptoaaah please open a new issue and provide all information requested, so we can help you

[fxcoudert]

At this point we have migrated all the main Python formula to python 3.9.
425 of our Python-related formulas have been migrated to Python 3.9, and 22 are still using Python 3.8, due to incompatibilities with Python 3.9

I am closing this tracking issue, as it mostly attracts bug reports (for which we prefer people to file new issues). Any help with the remaining PRs is welcome, at least in reporting bugs to upstream so they get fixed at some point.