Pipx: Feature proposal: Speed up pipx by using a shared pip for venvs

[pfmoore]

On Windows, under Python 3.7.3 and later, python -m venv --without-pip is essentially instant (it takes 0.5 sec, where python -m venv takes around 13 sec, on my PC). Saving the cost of installing pip in every venv would therefore be significant (especially for pipx run).

I believe this should be possible, by installing a single copy of pip in ~/.local/pipx, and then putting a .pth file in the newly created environment(s). This may not be 100% compatible with having pip installed in the venv (particularly for packages with dependencies on pip) so it should probably be opt-in, based on a flag somehow. Upgrading the shared pip would need some thought as well, as pip install --target doesn't support upgrades.

Would this be an acceptable feature to add? I'd be willing to do the implementation work, but I would need to know if it's something that would be accepted (and I'd likely need guidance on how to design the UI for the feature so that it fits with the rest of pipx).

I've no idea if the feature would be useful on non-Windows systems - I don't have a feel for whether venv creation without pip is fast on other systems (they don't use the same fast mechanism as Windows does).

[cs01]

under Python 3.7.3 and later, python -m venv --without-pip is essentially instant (it takes 0.5 sec, where python -m venv takes around 13 sec, on my PC). Saving the cost of installing pip in every venv would therefore be significant (especially for pipx run).

Agreed! I have noticed and thought a little about how this as well. It is slow on non-Windows platforms as well. My solution only applies to pipx run, which caches the virtual environments for 2 weeks. I am all for finding a way to do this more broadly though :smile: .

I guess what is up for discussion is the best way to implement this, and .pth files seem like a decent way to do this. I have never heard of the .pth file before. For anyone reading and curious, here is a brief summary from the docs:

A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path.

I believe this should be possible, by installing a single copy of pip in ~/.local/pipx, and then putting a .pth file in the newly created environment(s). This may not be 100% compatible with having pip installed in the venv (particularly for packages with dependencies on pip) so it should probably be opt-in, based on a flag somehow.

This sounds pretty promising, although a little magical. But that is okay with me so far.

One of pipx's features is that it allows you to associate a venv with various versions of Python. So a user might install black with Python 3.7, and flake8 with Python 3.3, for example. In each case when something like pipx upgrade black or pipx upgrade flake8 is run, the pip from the associated venv is used, which in turn uses the Python from the associated venv. Can a single pip be reliably used across venvs with different Python versions?

Some other questions that come to mind:

• pipx upgrades pip every chance it gets. Is is possible for pipx to continue doing this with the new .pth scheme?
• if a user opts-in, does it apply to all pipx-managed packages? or can some packages be installed with pip included, with and some without? Would batch commands like upgrade-all still work?
• Is there a way to add this meta-info to pipx list? (I would assume checking for the existence of the .pth file would suffice, but want to confirm)

Would this be an acceptable feature to add?

I am definitely interested in this. The most important part IMO would be to perform all possible validations as early as possible, in this case at the time the user attempts to install a package with this feature turned on. What I really want to avoid is having something fail at run time in a non-obvious way.

[pfmoore]

This sounds pretty promising, although a little magical

It is a bit magical, but .pth files are a fairly standard feature (setuptools uses them for installing projects in "develop" mode, for instance). Basically, all it does is put the directory containing pip onto sys.path, so it shouldn't cause any fundamental issues.

Can a single pip be reliably used across venvs with different Python versions?

Yes, definitely. Pip has no version-specific code (that's visibly the case, because it's distributed as a .py2.py3 wheel) and sharing the code among different versions of Python is OK. The .pyc files will be version-specific, but that should be fine (on Python 3, each version has its own .pyc file in __pycache__, and assuming we're only supporting Python 2.7, there's only one version of a Python 2 .pyc file, so no clash there).

pipx upgrades pip every chance it gets. Is is possible for pipx to continue doing this with the new .pth scheme?

It is, and that's probably easier than (say) making the user manage it themselves. There's some fiddliness involved, but I'll sort out the details as I develop the code, and discuss any decisions. It's definitely the sort of "I'd need advice on design" thing I'd want to discuss before making anything final.

if a user opts-in, does it apply to all pipx-managed packages? or can some packages be installed with pip included, with and some without?

I can see ways of doing either. My inclination would be to assume that the best approach would be to have it apply to every package, but allow the user to say "package X needs its own copy of pip". That may be over-engineered, though. For 99.9% of packages, they don't actually want or need pip to be installed - it's only there so that pipx can manage the code (something like black doesn't depend on pip at all). Those packages can just use a shared pip with no issues or downsides at all, I suspect. But the occasional package that has pip as an actual dependency (I honestly can't even think of one right now) might need special handling. It's something I can think about further, though.

Would batch commands like upgrade-all still work?

I see no reason why they wouldn't.

I am definitely interested in this.

Excellent! For now, that's all I really wanted to confirm. I'll put together a basic proof of concept implementation, and we can discuss the details further once I have some code to point to. It might take me a little while to get something together (my free time is pretty variable right now) but I will post something as soon as I can.

What I really want to avoid is having something fail at run time in a non-obvious way.

Agree 100%. And this is precisely the sort of design principle that I'm interested in, so that I can make sure I stay true to how you want pipx to behave.

[cs01]

Thanks so much! This all sounds very good. Looking forward to it. No rush though, I understand personal time is valuable.

[pfmoore]

OK, so it turns out that the basic idea is easy - it doesn't even need the .pth file. You can run pip from any virtual environment simply by doing python dir/that/pip/is/installed/in (note, no -m), so all we need is a copy of pip held "somewhere".

The easiest way to have a local copy of pip that we can upgrade as needed is to put it in a venv of its own. And indeed this seems to work pretty well.

I've created #168 as a demonstration of what I mean. It's working, in the sense that I've confirmed that I can install and run black using it locally. The CI shows a lot of failures, so there's still plenty to do, though.

Also, this current version doesn't do anything about keeping the shared venv up to date. The main complexity there is mostly around deciding exactly when to do the update - I'm thinking of "no more than one check per pipx run, when a venv is created or something is installed or updated". But there are some edge cases like pipx runpip that I need to think about some more.

[pfmoore]

@cs01 #168 is looking reasonably OK now. I still need to add docs and tests, and add an "opt in" flag to allow people to choose shared or not. Also, I need to switch one of my PCs over to using the modified pipx, so that I can give it some "real world" usage. But I'd appreciate a review from you if possible.

(You can ignore the test suite changes, IIRC all of those were just to get the tests sort-of working on Windows - I might look at a further PR at some point to add Appveyor and get the tests working properly on Windows).

[cs01]

This has been added in 0.14.0.0.