Tox: Support for selecting via Python build CPU architecture

[tr00st]

Would it be feasible to allow switching of Python environments based on CPU architecture?

I've been recently attempting to arrange builds of binary wheel files using Tox to automate the environment switching. This is particularly frustrating on Windows, as getting a working VC++ toolchain is a non-trivial exercise. To deal with this, I'm deploying scripts to our CI server that builds wheels of any binary packages we use, and uploads them to an isolated pypiserver instance. Tox currently works well for iterating over the versions available, but I've been unable to find a way to

Given the "py" launcher tracks CPU architecture, I've been able to come up with a way of patching this support in for Windows (see https://github.com/tr00st/tox/commit/b7dc030a4ed03971de0ec460b1ecdc87b642edbb) - the patch I've made means a user could set up a set of environments, eg at worst:

[testenv:py35-x86]
basepython = python3.5-32
[testenv:py36-x86]
basepython = python3.6-32
[testenv:py37-x86]
basepython = python3.7-32
[testenv:py35-x64]
basepython = python3.5-64
[testenv:py36-x64]
basepython = python3.6-64
[testenv:py37-x64]
basepython = python3.7-64

This is more than enough for my scenario, but I'm aware this wouldn't solve the issue for Linux users - hence I'm holding off a PR for now. Any idea if this would be desirable, and if so how one might go about this on linux? (things that come to mind are running the python interpreter with a quick one-liner or using "file" package to inspect any potential python executable).

Happy to put a PR through if this is desirable as-is, but thought it would be worth discussing any other possible solutions.

[gaborbernat]

I think the main problem is that there's no standard now for how to discover these additional targets. Any suggestions? It's fine if it's a Windows-only feature only though in which case yeah let's do this. I would have a basepython_architecture in this case though. On Linux/Mac x64 should be skipped.

[mrh1997]

Proposal:

• mac: all pythons are 64bit as far as I now => ignore all environments with basepython_architecture = 32
• linux: all pythons are of the same architecture as the distributions architecture => ignore all environments which explicitly enforce the other architecture.

Admittedly this is not a 100% solution, as it is possible to manually create 32bit binaries on 64bit linuxes (as on windows), but reality is different. On windows both architectures are explicitly distributed for 64bit windows systems, while on linux you can only install binaries for the same architecture as your system uses.

[gaborbernat]

This is now supported by virtualenv; we'll probably switch to that part of tox 4 rewrite.

[gaborbernat]

My available efforts at the moment are aimed at fixing this as part of https://github.com/tox-dev/tox/issues/1394, but that probably will take a while (ETA September).

[jayvdb]

While waiting for tox 4, this can be done by using https://github.com/jayvdb/tox-add-factor

[schlamar]

@jayvdb I don't think your proposal is correct. Please note that this feature request wants to allow different Python architectures on one host (for example the environments Python 3.7 32bit and Python 3.7 64bit on the same Windows x64 host). Your proposed plugin handles different architectures on different hosts AFAIS.

[schlamar]

For now, I handle this with a small plugin (so I don't have to patch tox).

import subprocess

import pluggy

hookimpl = pluggy.HookimplMarker('tox')


@hookimpl
def tox_get_python_executable(envconfig):
    basepython = envconfig.basepython
    if not basepython.startswith('py '):
        return None

    cmd = basepython.split() + ['-c', 'import sys; print(sys.executable)']
    try:
        path = subprocess.check_output(cmd)
    except (OSError, subprocess.CalledProcessError):
        return None
    return path.decode('ascii').strip()

And configure my environments:

[testenv:py37]
basepython = py -3.7-32

[testenv:py37-64]
basepython = py -3.7-64

Feel free to upload this on PyPI if someone is interested.

[ionelmc]

Note that you can use this trick:

[testenv]
basepython =
    py37: {env:TOXPYTHON:python3.7}
    py38: {env:TOXPYTHON:python3.8}
    ...

And then set TOXPYTHON in the environment to the preferred arch python.exe.

[gaborbernat]

This is now possible by virtualenv understanding these, and users can set basepython accordingly (which is passed through virtualenv) https://virtualenv.pypa.io/en/latest/user_guide.html#python-discovery