Pipx: Update injected packages.

[schinckel]

As mentioned in #72, it's currently not possible to update dependencies/injected packages in pipx.

Should this be supported in some form? What form should this take?

pipx update <foo>

Should this update all injected packages and dependencies, or not?

[pmav99]

Blindly updating dependencies can be problematic. You need to know that the main package is compatible with the newer version. If strict semver could be assumed, then maybe updating of minor versions and patches could be justified, but it cannot.

In the past pip was using --upgrade-strategy=eager which was especially problematic. Since version 10 (released last Summer?), I think that the default changed to --upgrade-strategy=only-if-needed which is a much safer default.

  -U, --upgrade               Upgrade all specified packages to the newest available version. The handling of dependencies depends on the upgrade-strategy used.
  --upgrade-strategy <upgrade_strategy>
                              Determines how dependency upgrading should be handled [default: only-if-needed]. "eager" - dependencies are upgraded regardless of whether the currently installed version
                              satisfies the requirements of the upgraded package(s). "only-if-needed" -  are upgraded only when they do not satisfy the requirements of the upgraded package(s).

That being said, i have not really tested it. Hopefully, in a not so far away future, every python package will be using pipenv/poetry to define their dependencies and pipx will be able to install the specific set of dependencies that the main package's developers chose/approved.

Until then, if --upgrade-strategy=only-if-needed is working OKish, then I guess it is not a problem to implement this, but if there are problems, I think it is just simpler for pipx to do nothing and let users manually solve any problems when they appear (it is just a virtualenv after all) and/or pipx uninstall foo followed by pipx install foo.

[taketwo]

Hopefully, in a not so far away future, every python package will be using pipenv/poetry to define their dependencies and pipx will be able to install the specific set of dependencies that the main package's developers chose/approved.

I'd like to point out that the injected packages are not necessarily dependencies of the "main" package. (Consider the example from README, we inject requests into ptpython REPL.) Thus in general case we don't (and never will) have a set of "officially approved" versions to match.

In my opinion, if we installed the main package and then injected more packages into it _without_ explicitly specifying versions, then we are in the "use latest and greatest" mode. It's therefore logical to eagerly upgrade everything. Things might break, yes, but then again they might have been broken from the start if unversioned packages are injected.

[cs01]

Agreed with @taketwo that the whole point of injecting dependencies is to add a new package that is NOT depended on by the package that was installed by pipx (the requests/ptpython example is apt here).

This introduces a challenge to pipx to basically inspect pip freeze and determine which packages were dependencies from the main package and which were injected. I don't know of a canonical way to do this. It's possible to hack something together to do this (for example could create a different temp virtual env, install there, inspect output of pip freeze to get set of dependent packages, but this is fraught with corner cases).

I think a good middleground might be to do something like this.

pipx install ptpython
pipx inject ptpython requests  # installs latest requests at time of injection

1 month later user wants to upgrade an injected package and requests has a newer version published

pipx upgrade ptpython
pipx inject --pip-args="--upgrade" ptpython requests  # user manually upgrades injections with pip-args

This is fairly manual, especially if many packages were injected. But like @pmav99 said it's just a virtual env, so if the user wants to do something more elaborate they can source activate the virtual env and do whatever they want:

source ~/.local/pipx/venvs/ptpython/bin/activate
pip install ...

---

Below is something else I would be open to implementing if there is interest to make running pip for a specific package a little easier. It feels a little verbose but at least doesn't force you to remember where venvs are stored.

pipx run-pip PACKAGE PIPCMD [PIPARGS]

so you could do things like

pipx run-pip ptpython install requests --upgrade
pipx run-pip ptpython freeze > requirements.txt
pipx run-pip ptpython install -r requirements.txt

inject would be an alias for

pipx run-pip PACKAGE install INJECTED_PACKAGE

---

To summarize

1. Automatically upgrading all injected packages would be difficult to get right for pipx and doesn't seem worth it to me
2. Users have some workarounds available
3. I'm open to discussing/adding commands to pipx such as pipx run-pip PACKAGE PIPCMD [PIPARGS]

[fbergroth]

Below is something else I would be open to implementing if there is interest to make running pip for a specific package a little easier. It feels a little verbose but at least doesn't force you to remember where venvs are stored.

pipx run-pip PACKAGE PIPCMD [PIPARGS]

How about pipx shell PACKAGE to spawn a shell within the venv, similar to poetry/pipenv.

[schinckel]

Since all injected packages are installed through pipx anyway, perhaps it could store the list of injected packages explicitly, and use that to determine what is not an automatically installed dependency.

[taketwo]

@cs01 I think that the inject command should be more than just a shortcut for "activate corresponding virtualenv and install this package". I'd like to see it as a "service" where pipx also promises to take care of the injected package by upgrading it as needed. If we had to advertise this, the slogan could be "inject and forget". The user installs packages, the user injects packages, and the only command needed to bring the system up to date is pipx upgrade-all.

Contrast with the manual workflow you just proposed. After the upgrade of the "main" packages the user needs to invoke a (fairly convoluted) upgrade command for each injected package separately. It's already rather awkward, leave alone actually remembering what is injected where (pipx list does not tell!)

I think an idea of keeping around a list of injected packages is worth exploring. Having such a list would allow to improve UX for multiple pipx commands:

• upgrade/all: able to run upgrades on injected packages
• list: additionally list injected packages and their versions
• reinstall/all: able to re-inject extra packages after re-installing the main one

[pmav99]

I think that this is ultimately a question of what type of app pipx wants to be.

If pipx is about installing python packages on a separate virtualenv environment while also adding the binaries to $PATH, then, it already does that. Part of what is being discussed here is more like an application that manages virtualenvs and could be seen as more or less out of scope. And to be completely clear, I am not saying that pipx could not be that too; I am just saying that this is something different.

My main concern is that upgrading packages is not straight forward. If you want to keep everything working, you will ultimately have to resolve dependencies which is a seriously non trivial problem and is certainly out of scope.

In the end, I think that having a shell command would be much easier from an implementation point of view + much more flexible in what would let people do.

That being said:

• the inject mechanism is really useful, because some times there are "undeclared" dependencies, or optional dependencies or whatever and you need someway to handle these.
• I also agree that keeping a list of injected packages would be nice. So a couple of flags could be added to pipx list. E.g. --injected could list the injected packages too and --all which would do a pip freeze on each environment.

[schinckel]

@pmav99 IMHO it's not about managing virtualenvs, the fact that we use virtualenvs to isolate the requirements or plugins for a given binary is incidental.

But (I think) you need to have some mechanism for being able to update injected packages. I can think of at least one tool I use daily (mercurial) that sometimes has a new version that breaks the functionality of plugins I have installed - being able to pipx update-injected mercurial mercurial_keyring or whatever would be much nicer than having to manually go in and update them.

Having said that, then yes, a shell command would mean that you would then have the scope of the target binary to update packages, however I agree with you that we are then part of the way towards pipenv and friends... ;)

Likewise, blindly updating injected packages is also a road to ruin: less so in the case of plugins, but more in the case of "optional" dependencies.

Perhaps some mechanism for listing injected packages, and the installed+new version numbers? Some way of then marking which ones should be updated (and perhaps a way of rolling back to the previously installed version number)?

(I'm just spitballing ideas now though).

[feigaoxyz]

Just sharing my 2 cents.

• Along the line of pipx shell and pipx run-pip, how about a generic pipx run --env=ptpython pip install requests? This can unify the inject variants and since we already have a run command. Also, it is kinda future-proof.
• About being like pipenv and poetry, I guess as long as pipx separates packages by creating venvs, it is inevitable. Maybe pipx can even use pipenv or poetry behind the curtain to deal with dependencies and upgrades, as this has been proven to be a difficult problem and lots of effort has been put into pipenv and poetry already.
• One vote for tracking injected packages. Either pipx's own packages.json, requirement.txt, or even Pipfile or pyproject.toml etc.

[hcoohb]

I would personally love
pipx run-pip PACKAGE PIPCMD [PIPARGS]
As I need to track when a package has a new version to check changelog if it is a major version bump before upgrading (I do not want to blindly update some package)
Thanks

[cs01]

The runpip command has been added in 0.13.

[itsayellow]

Now that we have metadata (#222), pipx knows the list of injected packages, and these could be upgraded during pipx upgrade.

[cs01]

If we go that route, I would suggest we upgrade injected packages by adding a new --injected flag to pipx upgrade to avoid upgrading things that might cause conflicts.