Pipx: Provide better error message when running pipx upgrade on a pre-0.15 venv

[mgedmin]

Describe the bug

Today I've tried to pipx upgrade virtualenv and it failed:

Traceback (most recent call last):
  File "/home/mg/.local/bin/pipx", line 8, in <module>
    sys.exit(cli())
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 192, in run_pipx_command
    venv_dir, package, pip_args, verbose, upgrading_all=False, force=args.force
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/commands/upgrade.py", line 34, in upgrade
    package_metadata = venv.package_metadata[package]
KeyError: 'virtualenv'

I've then manually upgraded pip, setuptools and pipx via

~/.local/pipx/venvs/pipx/bin/pip install -U setuptools pip
~/.local/pipx/venvs/pipx/bin/pip install -U pipx

and tried to see if the latest pipx also fails, by running pipx upgrade pipx, and I got

Traceback (most recent call last):
  File "/home/mg/.local/bin/pipx", line 8, in <module>
    sys.exit(cli())
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 192, in run_pipx_command
    venv_dir, package, pip_args, verbose, upgrading_all=False, force=args.force
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/commands/upgrade.py", line 34, in upgrade
    package_metadata = venv.package_metadata[package]
KeyError: 'pipx'

How to reproduce

pipx upgrade pipx

mg@blynas: ~ $ pipx upgrade --verbose pipx
pipx > (run_pipx_command:141): Virtual Environment location is /home/mg/.local/pipx/venvs/pipx
Traceback (most recent call last):
  File "/home/mg/.local/bin/pipx", line 8, in <module>
    sys.exit(cli())
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 192, in run_pipx_command
    venv_dir, package, pip_args, verbose, upgrading_all=False, force=args.force
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/commands/upgrade.py", line 34, in upgrade
    package_metadata = venv.package_metadata[package]
KeyError: 'pipx'

Expected behavior

No crash, working upgrade.

[mgedmin]

I see pipx list fails with the same error:

venvs are in /home/mg/.local/pipx/venvs
apps are exposed on your $PATH at /home/mg/.local/bin
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/commands/common.py", line 95, in get_package_summary
    package_metadata = venv.package_metadata[package]
KeyError: 'ansible-lint'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mg/.local/bin/pipx", line 8, in <module>
    sys.exit(cli())
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/main.py", line 195, in run_pipx_command
    return commands.list_packages(venv_container)
  File "/home/mg/.local/pipx/venvs/pipx/lib/python3.7/site-packages/pipx/commands/list_packages.py", line 33, in list_packages
    for package_summary in p.map(get_package_summary, dirs):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
KeyError: 'ansible-lint'

[mgedmin]

This appears to affect only my old pipx venvs. I can pipx install cowsay and then pipx upgrade cowsay says it's already at the latest version.

[mgedmin]

I see that my old venvs do not have pipx_metadata.json files, but new ones do. Some kind of a backwards-incompatible change in a recent pipx release? I must go read the changelog.

[uranusjr]

Yes there was a release recently that introduced the metadata storage. That sounds correct. cc @itsayellow

[mgedmin]

Ah, I see this is explicitly mentioned in https://pipxproject.github.io/pipx/changelog/ for version 0.15. I need to pipx reinstall-all.

I wish pipx would tell me that "required pipx metadata is missing, please reinstall ansible-lint with pipx reinstall ansible-lint" this instead of failing with an obscure KeyError.

I wish pipx reinstall existed for reinstalling one venv, because when pipx reinstall-all fails halfway through (because I pipx installed ~/src/some-internal-tool) there's no way to resume without reinstalling all the packages already reinstalled successfully.

I wish pipx reinstall-all wouldn't uninstall pipx itself, causing the reinstall-all process to fail halfway through with a FileNotFoundError: [Errno 2] No such file or directory: '/home/mg/.local/pipx/venvs/pipx/bin/python' when it tries to re-create the pipx venv.

I wish pipx list would show me the names of the packages I have installed even if they lack metadata, so I would know what names to pass to the hypothetical pipx reinstall.

Would you be willing to accept pull requests to improve the UX of this situation?

[mgedmin]

So far the easiest workflow of reinstalling all the things I have (while skipping pipx itself) is to do pipx list, note the package name in the KeyError, then repeat

pipx uninstall that-package; pipx install $_; pipx list

This still gives me the chicken-and-egg problem of pipx itself. It's mostly a self-inflicted problem: I wanted to be able to pipx upgrade pipx, so instead of using pip install --user pipx, I manually created a venv in ~/.local/pipx/venvs/pipx and pip installed pipx in there.

pipx run pipx install pipx --force seems to have fixed my pipx installation. (I probably didn't need the pipx run prefix, but it's what I tried and it worked).

Hmm, if pipx install --force $pkgname works, do I need to pipx uninstall anything?

[uranusjr]

How did you install pipx in the beginning? I believe ppipx is no longer supporting this kind of installation precisely for this kind of chicken-and-egg situation.

[mgedmin]

How did you install pipx in the beginning?

As I said before, I manually create a venv in ~/.local/pipx/venvs/pipx and pip install pipx in there.

If you want the exact commands, the script is here: https://github.com/mgedmin/scripts/blob/1d9ef11b5a044aa1394765dbc05fabc41c16b804/pipx. I've recently added the pipx install --force pipx step to it to create the missing metadata file.

I should probably stop doing this?

[uranusjr]

I’d say it’s fine since you know what you’re doing, just remember pipx is not really designed to deal with some edge cases :)

[itsayellow]

Unfortunately this is a one-time growing pain from pre-0.15.0.0 to post-0.15.0.0 where we actually started tracking metadata. Since you seem to have an untypical pipx install, you are ending up having to do more stuff manually. Sorry!

[itsayellow]

Fixed via #414