Tox: [question] Any way to avoid "current" python as basepython?

[sanjioh]

Hi,

first of all, thanks for your great work on tox :)

I have a question regarding the base Python executables used by tox to create virtualenvs.

I have a dedicated virtual environment into which tox is installed. The tox script is symlinked to ~/.local/bin to have it in PATH.

I've noticed that, whenever tox needs to create a new virtualenv, if the required Python version matches the one tox is running with, the new virtualenv is created with tox's "own" Python as base Python (as opposed to looking for the executable in PATH).

This leads to the creation of a virtualenv using another virtualenv's Python executable as base Python, which I'd like to avoid - instead, I'd like to have all virtualenvs created from Python distributions installed at the system level (in my case, using the official python.org installers for MacOS).

I've tracked this behaviour to this: https://github.com/tox-dev/tox/blob/master/src/tox/interpreters/unix.py#L14-L15

Is it possible to force the use of base interpreters other than the one tox is running with?
Is this a valid use case for a tox plugin? If so, any hints about how to get started?

Thanks!

[BenjamenMeyer]

From the tox documentation seems like you want to set basepython:
https://tox.readthedocs.io/en/latest/config.html#conf-basepython

While normally one would just set py37 to get Python3.7, from the documentation at least it seems you could use a path the interpreter too - so /usr/bin/python3.7 should get you the system python 3.7 version....

I'm not sure there's a way to do it with any system independence though; so you'd likely be locking yourself into your specific build environment/OS set - meaning building on Linux may work but then trying to build on Mac would likely break, and Windows would definitely break.

💵 FWIW

tox configuration specification — tox 3.14.1.dev5 documentation

[sanjioh]

Thanks for your reply.

I’m aware of basepython, but setting it to an absolute path poses (at least) two problems:

1) as you already said, the configuration becomes strictly system dependent
2) in case of auto-generation of environments (with factors) the configuration becomes impractical (basically a [testenv:<envname>] section for every factor combination, just to set basepython)

[BenjamenMeyer]

Agreed on those limitations... only other way I can easily think to control it would be to give yourself a wrapper around tox so it only has PATH set for only the system pythons to be available.

#!/bin/bash
PATH=<standard system paths>
tox ${@}

and

@ECHO OFF
PATH=<standard system paths>
tox %*

Limitation becomes having to have those wrappers on the various platforms, and it'd be best to call them something other than tox otherwise the tox line has to refer to the real tox... though I guess you could just drop it into ~/.local/bin too as tox and it should auto-sort itself out.

[sanjioh]

Thanks, that’s something worth trying. I’m not sure if tox’s own virtualenv would become “unreachable”, though.

The other path I came up to is a tox plugin to add custom behaviour to the tox_get_python_executable hook. But before going down that path I wanted to be sure it cannot be done otherwise :)

[BenjamenMeyer]

@sanjioh I say "unreachable" because of PATH ordering - the first path in which an executable is found (e.g tox or python) is the one used, which is the issue you're running into - it's finding it under ~/.local/bin instead of /usr/bin:/usr/local/bin:...etc; so you just need some way to remove ~/.local/bin from the PATH search so the system one becomes available.

Likewise, in the scripts I referred, if PATH wasn't modified and the script's name was also tox then you'd end up with the script infinitely calling itself instead of calling out to the real tox unless you hard-coded tox's path into the script. It's probably just easier (especially on Mac with Brew installs of python and tox; or Windows for similar reasons) to just manage the PATH var in the script. If you know enough script-foo you can come up with some interesting magic to remove a PATH from PATH and re-export it - but that's a bit more complex then I'd like to get in here (and I have no way to prove the Windows variant of it either - so you'd end up having to debug it if you needed Windows).

Any how... good luck!

[gaborbernat]

Is it possible to force the use of base interpreters other than the one tox is running with?
Is this a valid use case for a tox plugin? If so, any hints about how to get started?

Yes, make a plugin for it. There's no way to force things, your interpreter declares no basepython; which is equal to saying any python will do; as such tox-s own python is the one we can acquire with the least work, hence we use that.

[sanjioh]

As a follow up, I made this https://github.com/sanjioh/tox-interpreters
Any feedback welcome :)

GitHub

sanjioh/tox-interpreters

Customize Python interpreters used to create tox testenvs. - sanjioh/tox-interpreters