Tox: Allow specifying pip version

[OrangeDog]

Currently it appears tox always installs the latest pip when (re)creating virtualenvs.
However sometimes (like today) the latest version doesn't work, or you haven't yet accounted for breaking changes.

Tox should have a way to control the version of pip.
Perhaps a pip_version config property, or allowing install_command to be multiple lines.

[asottile]

The _latest_ behaviour here is due to virtualenv's downloading behaviour. If you want a more "consistent" version, I might suggest using tox-virtualenv-no-download#wait-why.

Related: #448

[gpotter2]

The latest behaviour here is due to virtualenv's downloading behaviour.

Is that still accurate ? Since today (new release?!), some of our AppVeyor builds (Python 2.7 builds) fail with an outdated pip version.

The issue is explicit:

[...]
  File "C:\projects\scapy\.tox\py27-windows\lib\site-packages\pip\_vendor\_markerlib\markers.py", line 113, in marker_fn
    return eval(compiled_marker, environment)
  File "<environment marker>", line 1, in <module>
  File "<environment marker>", line 1, in <module>
NameError: name 'platform_system' is not defined
You are using pip version 8.1.1, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

=================================== log end ===================================
ERROR: could not install deps [mock, setuptools>=18.5, cryptography, coverage, python-can]; v = InvocationError(u"'C:\\projects\\scapy\\.tox\\py27-windows\\Scripts\\python.EXE' -m pip install mock 'setuptools>=18.5' cryptography coverage python-can", 2)

I bumped into https://github.com/tox-dev/tox/issues/765 then here :/

[gaborbernat]

soon you can override the default version into pip, in the meantime please upgrade virtualenv alongside tox to get later defaults.👌

[gaborbernat]

And yes with the latest version we no longer download latest pip.

[gaborbernat]

When installing tox use the force reinstall flag to get latest virtualenv 😀

[gpotter2]

Thanks for this light-speed answer :smile:

Does tox uses the pip version of the virtualenv ?
I'd rather only update pip

[gaborbernat]

tox creates virtual environments that have their seed packages (pip, setuptools, wheel) with the version bundled within the virtualenv installed alongside it. you can of course use the deps then to upgrade to install another version. Or just upgrade virtualenv to have newer bundles.

[jayvdb]

There is https://github.com/pglass/tox-pip-version which does what this issue requests, I believe

GitHub

pglass/tox-pip-version

A tox plugin to select your pip version. Contribute to pglass/tox-pip-version development by creating an account on GitHub.

[pglass]

you can of course use the deps then to upgrade to install another version

@gaborbernat This doesn't actually work in my experience. It will install a version of pip, yes, but tox will not actually use that new version of pip to install the rest of the dependencies.

The reason is because tox runs _one_ pip command to install everything, like python -m pip install pip==19.1 requests (see example below). It won't start using pip version 19.1 until the _next_ pip command is run.

[testenv]
deps =
    pip==19.1
    requests

Some other things I've tried:

• Trying to change the install command, like install_command = pip install -U 'pip==18.1' {opts} {packages} has the same issue.
• Using tox-run-before plugin, but this runs commands prior to the creation of the tox virtualenv (so it affects the currently active non-tox venv or else your system-wide installation, which is yucky), which brings me to the next point...

It is true that if you install a specific version of virtualenv prior to running tox, you can change the effective version of pip that tox uses to install packages. But, this still falls short in some ways:

• This is not a solution for users of tox-venv, which uses python -m venv instead of the virtualenv package
• It's not obvious which version of the virtualenv package comes with a particular version of pip (poor user experience)
• Releases of the virtualenv package lag behind releases of pip (from a day to up to a week), so this isn't a viable solution in the "critical" period when a new release of pip breaks you

Hence, I did make a plugin for this (see previous comment), but it would be much nicer for this to be a native feature.

[gaborbernat]

This will be fixed on the virtualenv side, I'm actively working on https://github.com/pypa/virtualenv/issues/1366, I hope to have something by the end of the month.

[ssbarnea]

Any update on that? I am looking for a workaround that can be used today, without waiting for a newer version of virtualenv.

[asottile]

@ssbarnea do you want "latest" -- you can re-enable virtualenv's downloading by setting the testenv download option:

[testenv]
download = true

[gaborbernat]

No direct progress on this. Made some progress on the tox rewrite that will allow me to test the virtualenv rewrite... Coming along but this is a major effort (fixing a lot of hard issues while at it - e.g. lazy configuration load), so probably will last into September.

[The-Compiler]

FWIW I'm using tox-venv because I've had some trouble with virtualenv in the past. This means I get whatever pip which ships with Python's ensurepip (as of Python 3.8.1, that's pip 19.2.3). However, I needed to install manylinux2014 wheels, which requires pip 19.3...

Installing tox-pip-version and setting pip_version = pip in [testenv] to get the latest pip worked beautifully - thanks @pglass! :tada:

[gaborbernat]

tox-venv will become obsolete once https://github.com/pypa/virtualenv/pull/1481 lands, and that will also solve this issue on virtualenv side 👍

[The-Compiler]

I've been hearing about virtualenv rewrites for the past five years or so, so I wonder how this one turns out - good luck! :crossed_fingers:

[gaborbernat]

This one is very close to ship-able, 😊 so will actually happen, the latest end of February! I'm fully committed now to deliver it.

[Kentzo]

Updating virtualenv that is installed alongside tox may not be an option since it may not support target python.

E.g. my host Python is 3.7 and my target Python is 2.6 / 3.4. Therefore I can install the most recent tox, but I cannot install the most recent virtualenv with most recent virtualenv dependencies.

[gaborbernat]

Hello @Kentzo sadly we no longer support Python2.6, even at the target level. Python3.4 should still work.

[Kentzo]

I just hope that this issue will not resurface in the future. tox's dependency on virtualenv_support is a major PITA.

[gaborbernat]

Not sure what alternatives you're proposing here 🤷‍♂

[Kentzo]

Ability to update pip / setuptools / etc inside bootstrapped virtualenv before installing any of the deps. Perhaps an ability to execute arbitrary shell code.

[gaborbernat]

Definitely no to arbitrary shellcode; ability to do those via virtualenv will be provided; to be fair it already is via the --download flag of virtualenv.

[gaborbernat]

This should be now possible by setting VIRTUALENV_PIP=19.0.0 with virtualenv 20.

[rouge8]

This should be now possible by setting VIRTUALENV_PIP=19.0.0 with virtualenv 20.

@gaborbernat is there a way to tell tox what version of virtualenv to use? Or do we have to make sure people are running virtualenv 20+ another way?

[gaborbernat]

You can use the requires key in the configuration.https://tox.readthedocs.io/en/latest/config.html#conf-requires