Pipx: Once system python and pipx upgraded, cannot install new app if some previous old shared_venv existed

[hcoohb]

Describe the bug

On Manjaro/Arch, I was with python 3.7.4.
To ensure stability of some apps (and as some apps are not fully working on 3.8 that I new was coming) I also had python 3.7.5 installed through pyenv.
I use this python to install my apps with

pipx install --python ~/.pyenv/versions/3.7.5/bin/python flexget
pipx inject flexget transmissionrpc

Then my system got upgraded from 3.7.4 to 3.8 (Note that my 3.7.5 pyenv stayed)
I reinstalled pipx on 3.8 on system.
Then I had error thrown when installing new pipx no matter what versions I tried:

$ pipx install pycowsay --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /usr/bin/python -m venv --without-pip /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay
/home/hcooh/.local/pipx/venvs/pycowsay/bin/python: No module named pip

pipx > (rmdir:16): removing directory /home/hcooh/.local/pipx/venvs/pycowsay
'/home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay' failed

or even trying to use the pyenv:

pipx install --python ~/.pyenv/versions/3.7.5/bin/python pycowsay --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.pyenv/versions/3.7.5/bin/python -m venv --without-pip /home/hcooh/.local/pipx/venvs/pycowsay
pipx > (run:97): running /home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay
/home/hcooh/.local/pipx/venvs/pycowsay/bin/python: No module named pip

pipx > (rmdir:16): removing directory /home/hcooh/.local/pipx/venvs/pycowsay
'/home/hcooh/.local/pipx/venvs/pycowsay/bin/python -m pip install pycowsay' failed

Running the already installed app (using the 3.7.5) would work fine and even pipx runpip xxx list
I realised then that .local/pipx/shared/pyvenv.cfg mentioned python version 3.7.4.
So I renamed the .local/pipx/shared folder differently. It recreated the shared lib for python3.8.
I had to then symlink the pyenv folder of python3.7 with:

ln -s ../../../../.pyenv/versions/3.7.5/lib/python3.7/ ~/.local/pipx/shared/lib/python3.7

And then install run fine and the already installed app run fine too

How to reproduce

See above

Expected behavior

pipx Install install should work after a pyhton upgrade even if hte previous shared folder still exists.
i think

[itsayellow]

Hm. I'm trying to figure out how I might reproduce this on my own system. There's a lot going on in your issue description.

Do you have a simplified way to reproduce this that I could try, possibly using pyenv?

For the devs, it appears that Venv.create_venv() is being called, which would execute shared_libs.create(). This happens without error, so either its creating a shared library dir or finding one it thinks is valid.

Since it can't find pip, it seems as though either pipx_shared.pth is absent, in the wrong place, or contains a bad path to the shared libraries. Or something out of my conception...?

Unless I can somehow reproduce this on my system I'm pretty lost as to what is going on.

[darkrain42]

@itsayellow, I'm not sure this is reproducible with pyenv, but I can detail how I encountered it / can reproduce it on Debian (where sid recently went from /usr/bin/python3 being 3.7 to 3.8) without anything other than the system Python, and maybe that will help.

In detail:

• docker run --rm -it debian:bullseye # this will stop working when python3.8 migrates to Debian testing, but it's faster than upgrading from buster.

(All steps now inside the container)

• apt -y update && apt -y install --no-install-recommends python3-pip python3-setuptools python3-venv python3-wheel
• python3 -m pip install pipx
• pipx install yq

yq is now in ~root/.local/pipx/venv

Update python3 to python3.8:

• sed -i -e 's/bullseye/sid/g' /etc/apt/sources.list
• apt -y update && apt -y upgrade

root@e861fc867d19:/# python3 --version
Python 3.8.2
root@e861fc867d19:/# pipx
Traceback (most recent call last):
  File "/usr/local/bin/pipx", line 6, in <module>
    from pipx.main import cli
ModuleNotFoundError: No module named 'pipx'
root@e861fc867d19:/# 

Reinstall pipx:

• python3 -m pip install pipx

At this point, trying to upgrade an existing app or install a new one fail with a "no module named pip" error

root@e861fc867d19:/# pipx upgrade yq
/root/.local/pipx/shared/bin/python: No module named pip
Failed to upgrade shared libraries
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pipx/shared_libs.py", line 70, in upgrade
    run(
  File "/usr/local/lib/python3.8/dist-packages/pipx/util.py", line 134, in run
    raise PipxError(f"{cmd_str!r} failed")
pipx.util.PipxError: '/root/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install -q --upgrade pip setuptools wheel' failed
/root/.local/pipx/venvs/yq/bin/python: No module named pip
'/root/.local/pipx/venvs/yq/bin/python -m pip install --upgrade yq -q' failed
root@e861fc867d19:/# 

# pipx install --verbose jq
pipx > (_package_name_from_spec:93): Determined package name: jq
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /root/.local/pipx/venvs/jq
pipx > (run_subprocess:112): running /root/.local/pipx/venvs/jq/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /root/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /root/.local/pipx/venvs/jq/bin/python --version
pipx > (run_subprocess:112): running /root/.local/pipx/venvs/jq/bin/python -m pip install jq
/root/.local/pipx/venvs/jq/bin/python: No module named pip
pipx > (install_package:189): '/root/.local/pipx/venvs/jq/bin/python -m pip install jq' failed

pipx > (rmdir:18): removing directory /root/.local/pipx/venvs/jq
Error installing jq.
root@e861fc867d19:/# 

[itsayellow]

Have you tried pipx reinstall-all? This command is supposed to help for situations when the system python changes, and has been known to work for that situation on the Mac.

If that doesn't work, what does pipx list return? It should show a lit of packages with warnings that they are missing a valid python interpreter.

Did you uninstall the old jq before trying to install the new jq? That may be important.

The other thing I can think of that might possibly be affecting you is that you are installing pipx to the system as opposed to using python3 -m pip install --user pipx as is our recommendation.

[darkrain42]

It's subtle (I should probably have picked better example packages for the demo), but note that the first package was yq (YAML) and the second was jq (JSON). Sorry :)

• pipx reinstall-all fails during the 'install' portion of reinstalling the first package. (Leaving the package uninstalled, and losing pipx_metadata.json!)
• pipx list works fine
• If I run the pip / pipx portions of the above as a non-root user, it still reproduces.

Before upgrading the system Python:

root@cc012cba1af1:/# su - paul
paul@cc012cba1af1:~$ python3 -m pip install pipx
...
paul@cc012cba1af1:~$ export PATH="$PATH:$HOME/.local/bin"
paul@cc012cba1af1:~$ which pipx
/home/paul/.local/bin/pipx
paul@cc012cba1af1:~$ pipx install yq
  installed package yq 2.10.0, Python 3.7.6
  These apps are now globally available
    - xq
    - yq
done! ✨ 🌟 ✨
paul@cc012cba1af1:~$

After upgrading the system Python:

root@cc012cba1af1:/# su - paul
paul@cc012cba1af1:~$ export PATH="$PATH:$HOME/.local/bin"
paul@cc012cba1af1:~$ pipx
Traceback (most recent call last):
  File "/home/paul/.local/bin/pipx", line 6, in <module>
    from pipx.main import cli
ModuleNotFoundError: No module named 'pipx'
paul@cc012cba1af1:~$ python3 -m pip install pipx
Collecting pipx
...
Successfully installed argcomplete-1.11.1 click-7.1.1 distro-1.4.0 pipx-0.15.1.3 userpath-1.3.0
paul@cc012cba1af1:~$ pipx list
venvs are in /home/paul/.local/pipx/venvs
apps are exposed on your $PATH at /home/paul/.local/bin
   package yq 2.10.0, Python 3.7.6
    - xq
    - yq
paul@cc012cba1af1:~$ pipx install jq
/home/paul/.local/pipx/venvs/jq/bin/python: No module named pip

Error installing jq.
paul@cc012cba1af1:~$ pipx reinstall-all --verbose
pipx > (needs_upgrade:51): Time since last upgrade of shared libs, in seconds: 144.02421402931213. Upgrade will be run by pipx if greater than 2592000.0.
pipx > (needs_upgrade:51): Time since last upgrade of shared libs, in seconds: 144.0251932144165. Upgrade will be run by pipx if greater than 2592000.0.
pipx > (uninstall:292): removing symlink /home/paul/.local/bin/yq
pipx > (uninstall:292): removing symlink /home/paul/.local/bin/xq
pipx > (rmdir:18): removing directory /home/paul/.local/pipx/venvs/yq
uninstalled yq! ✨ 🌟 ✨
pipx > (run_subprocess:112): running /usr/bin/python3 -m venv --without-pip /home/paul/.local/pipx/venvs/yq
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/yq/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/yq/bin/python --version
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/yq/bin/python -m pip install yq
/home/paul/.local/pipx/venvs/yq/bin/python: No module named pip
pipx > (install_package:189): '/home/paul/.local/pipx/venvs/yq/bin/python -m pip install yq' failed

pipx > (rmdir:18): removing directory /home/paul/.local/pipx/venvs/yq
Error installing yq.
paul@cc012cba1af1:~$ pipx list
nothing has been installed with pipx 😴
paul@cc012cba1af1:~$

Some further bits of information:

This _seems_ to be specific to using a python3 -m pip (or pip3) to install pipx, which causes pipx's shebang to be #!/usr/bin/python3.

When I use python3.7 -m pip install pipx, then pipx's shebang is versioned (#!/usr/bin/python3.7), and as long as Debian's python3.7 package is still installed, everything keeps working.

Further, if I then uninstall python3.7 (apt -y autoremove --purge) and reinstall pipx (python3.8 -m pip install pipx), pipx is able to detect it needs to reinitialize the shared packages (note the python3.8 -m venv --clear /home/paul/.local/pipx/shared below)

root@624077171e9c:/# su - paul
paul@624077171e9c:~$ pipx --verbose
-bash: /home/paul/.local/bin/pipx: /usr/bin/python3.7: bad interpreter: No such file or directory
paul@624077171e9c:~$ python3.8 -m pip install pipx
Collecting pipx
...
Successfully installed argcomplete-1.11.1 click-7.1.1 distro-1.4.0 pipx-0.15.1.3 userpath-1.3.0
paul@624077171e9c:~$ pipx install --verbose jq
pipx > (_package_name_from_spec:93): Determined package name: jq
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running /usr/bin/python3.8 -m venv --without-pip /home/paul/.local/pipx/venvs/jq
pipx > (run_subprocess:112): running /usr/bin/python3.8 -m venv --clear /home/paul/.local/pipx/shared
pipx > (upgrade:62): Upgrading shared libraries in /home/paul/.local/pipx/shared
pipx > (run_subprocess:112): running /home/paul/.local/pipx/shared/bin/python -m pip --disable-pip-version-check install --upgrade pip setuptools wheel
Collecting pip
...
Successfully installed pip-20.0.2 setuptools-46.0.0 wheel-0.34.2
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/jq/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/jq/bin/python --version
pipx > (run_subprocess:112): running /home/paul/.local/pipx/venvs/jq/bin/python -m pip install jq
Collecting jq
  Downloading jq-0.1.6.tar.gz (35 kB)
...

[itsayellow]

OK, I finally had a chance to run your very complete docker instructions. Thank you for that!

I can reproduce what you're talking about. And it doesn't seem to help if you install pipx using pip install --user.

First problem, yq stops working after system python upgrade

What seems to be happening is that in ~/.local/pipx/venv/yq/bin all the pythons are being linked back only to /usr/bin/python3. This is not what happens for me on my Mac: there each venv's python is linked to a very particular version of python on my system (e.g. /usr/local/Cellar/python/3.7.7/python3.7.). So it doesn't come up empty looking for a non-existent python (which we handle), but the python it finds has a different minor version than before. This causes site packages to be in the wrong place (they are in ~/.local/pipx/venvs/yq/lib/python3.7/ but the new python3.8 is looking for ~/.local/pipx/venvs/yq/lib/python3.8/ which doesn't exist)

I see the same behavior on my Ubuntu system. The python interpreter link in all the venvs points to /usr/bin/python3, not a specific version like /usr/bin/python3.7.

The workaround when you first install yq would be this pipx command:

pipx install --python /usr/bin/python3.7 yq

Then after upgrading the system python yq will still work. (I tried it and it works for me.)

My suggestion to the pipx developers is that when we determine the system python, maybe we should resolve the link back to the original binary filepath which would get a specific python sub-version. (e.g. in utils.py we should use DEFAULT_PYTHON = Path(sys.executable).resolve()). I'm not sure if that's the best solution (it would cause sub-minor python upgrades to break pipx venvs) but it's a solution.

Second problem, reinstall of pipx will not install after system upgrade

I think this is similar to the venv problem described above. The shared libs have their own idea of a python binary, and on linux this seems to be linked to a generic binary without a sub-version (i.e. /usr/bin/python3) which then screws up the shared lib venv when the python minor version number changes. It appears to pipx that at least the python binary is valid, but the shared_lib venv has no site-packages under the the new python3.8 only under the old python3.7.

shared_libs.py, _SharedLibs.is_valid() only checks if we have files of the python binary and the pip binary, but it doesn't check that the library pip is still accessible for the current python. When the python the shared_libs is linked to gets changed from 3.7 to 3.8, then we no longer have the package pip under the right directory in the shared_libs venv.

[itsayellow]

btw the workaround after an upgrade of the system python to get pipx working (after reinstalling pipx):

rm ~/.local/pipx/shared/bin/python3
ln -s /usr/bin/python3.7 ~/.local/pipx/shared/bin/python3

Or at least I assume that should be the workaround, but when I tried it, it found pip but then failed spectacularly trying to build a wheel for jq.

[itsayellow]

It seems that we need a better _SharedLibs.is_valid() that is more in-depth and can trigger a reinstall of the shared libs if there is no pip library found.

[darkrain42]

Or at least I assume that should be the workaround, but when I tried it, it found pip but then failed spectacularly trying to build a wheel for jq.

Sorry, that's another example of "I should have chosen better packages as examples." It fails because of no C compiler.

Using "yq" and then "httpie" seem like better example packages -- neither require compilation.

[metinsenturk]

I think I encountered a similar problem after upgrading my python install from the brew. The solution that worked for me is pipx reinstall-all.

To illustrate the problem and how it gets resolved, the following are the commands I did.

 me@MacBook-Pro  ~  python --version
python 3.8.3

 me@MacBook-Pro  ~ brew upgrade
...
all done!
 me@MacBook-Pro  ~ python --version 
python 3.8.5
 me@MacBook-Pro  ~  pipx list
venvs are in /Users/me/.local/pipx/venvs
apps are exposed on your $PATH at /Users/me/.local/bin
   package autopep8 has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package black has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package flake8 has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package jupyterlab has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package pycowsay has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package pylint has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8
   package virtualenv has invalid interpreter /usr/local/Cellar/[email protected]/3.8.3/bin/python3.8

 me@MacBook-Pro  ~  pipx reinstall-all
...
 me@MacBook-Pro  ~  pipx list
venvs are in /Users/me/.local/pipx/venvs
apps are exposed on your $PATH at /Users/me/.local/bin
   package autopep8 1.5.4, Python 3.8.5
    - autopep8
   package black 20.8b1, Python 3.8.5
    - black
    - black-primer
    - blackd
   package flake8 3.8.4, Python 3.8.5
    - flake8
   package jupyterlab 2.2.8, Python 3.8.5
    - ...
   package pycowsay 0.0.0.1, Python 3.8.5
    - pycowsay
   package pylint 2.6.0, Python 3.8.5
    - ...
   package virtualenv 20.0.33, Python 3.8.5
    - virtualenv

[itsayellow]

This last example is related to #493