Tox: impro basepython version selection option and document it

[ssbarnea]

At this moment basepython is documented as a solution for pointing to a specific python version on a certain environment.

Historically this was succesfully used to force use of python2 or python3 for certain environments which were known to produce unpredictable results on one of them.

Still, there is one issue which I was not able to fix not to find any documentation on how to do approach it.

You have one project that you want to test with py27, py35, and py36 and run the linting and documentation using python3. This would work on many systems unless you happen to be on once where python3 executable points to python3.4 which happens to be version provided by your distribution (like CentOS 7, quite a common development environment).

You would want to run with a py3 blend that your support but you don't really know if this is going to be py35 or py36 as some platforms have py35 installed, others may have only py36, and some may have both.

So how do I configure tox.ini in order to be able to build all environments without using non blessed interpreters?

[tox]
envlist = py27,py35,py36,docs
skip-missing-interpreters=True

[testenv]
commands = echo

[testenv:docs]
basepython = ????
# we already know that putting `python3` would use py34 on some machines
# also we know that using `python3.5` or `python3.6` here would also fail on some machine
# how to make it use py35 or py36 here?
commands = python --version

[gaborbernat]

@ssbarnea this is by design, and there is no real solution for this. We're hesitant to provide a way for this because it would require us probing for interpreters and then picking maybe the latest? At this point, your environment will a best effort instead of deterministic. We avoided opening this can of worms. You have two paths ahead:

• create a symlink on path to point to a Python 3 (e.g. local bin path, can point to python3.6 on centos) to hide away the system Python,
• create a tox plugin that auto-discovers existing interpreter and picks the latest (I would encourage this later approach) after configuration (e.g. could update python3 to point to python36 if existing, python35 otherwise, etc).

Can you please detail why: This would work on many systems unless you happen to be on once where python3 executable points to python3.4 which happens to be version provided by your distribution (like CentOS 7, quite a common development environment). this is not good enough. The environment has python3 as a requirement, and python3.4 satisfies this.

[ssbarnea]

@gaborbernat Since when the python3 or python3 are deterministic? These are also listed in official documentation as examples.

Taining with operating system is not an option for multiple reasons, one of them being that it even be impossible to do (not to mention the security aspects).

The way I described the ticket this is a bug because 3 supported python versions are listed and another one (py34) is used instead.

In my development environment I have about 7-10 python versions installed in order to accomodate the needs of various python based packages where I contribute changes. None of them is using all versions of pythons from my machine but I expect them to be able to declare which ones are to be used for testing (or not to to be used).

I hope we will be able to find a workaround that would not involve a change outside tox.ini, as the contained value of tox is something we all appreciated over time.

[gaborbernat]

Deterministic in the sense that the caller knows it will link to python3 executable or py -3. tox environments doesn't have relation to each other. Just because of you listed py37 py3 will not point to anything else than python3. Also setting to py35 or py36 is not an option, because what if either of them do not exist? If you want py35 put python3.5. If you want py36 put python3.6.

[gaborbernat]

Closing as sounds to me we did not find any actionable items for this. The issue brought up is by design and works as expected.

[ssbarnea]

3 months later I am finding myself writing the same bug report again but luckly github suggestions payed off and indicated me the old one, so I will paste it as a comment.

The problem becomes more serious as more projects started to drop support for a various range of previous py3 versions (some dropping py34, some also py35) and which works very bad across various production distributions which do have older py3 versions.

As a tox user, I want to run a specific environment with any python>=3.6, which means that I do not want tox to use python3.4 or python3.5 by accident, something that can easily happen if these are installed.

The current basepython option only allows us to specify the executable and while it did a good job for forcing environments to be py2 or py3 only, it fails to be more specific than this.

We are not able to pin it to a specific version because on some platforms we may have only py36 and on some only py37 installed, but we are still required to run linting on all.

To be clear: the projects may still support py27,py34,py35,py36,py37 but the linting requires one of the last two versions (as latest linters dropped support for some older python versions).

Is there a workaround we can use to make tox work with that?

If tox would respect the mentioned order of python version from envlist and prefer them in that order we could use that as a deterministic way to ensure that py36/py37 are preffered in favour of older versions and fail only if these are not installed at all.

Background story: About one week ago ansible-lint made a new release 4.0.0 where they dropped support for python3.4 and python3.5 and the direct side-effect of this was that it did break LOTs of openstack components, mainly all of them that used basepython=python3 in the linters environment, something that is recommended by the our development guidelines (related to testing with python3 first and also because many linters like flake8 do give better results with latest python versions).

If you are courious about the extent of the _damage_, just have a breaf look at http://codesearch.openstack.org/?q=ansible-lint&i=nope&files=tox.ini&repos= to get an idea. As seen some projects already switched to use python2 instead to workaround that specific issue.

I do still find tox not being deterministic here and not the requested behaviour as described in the bug.

[gaborbernat]

Workaround? How about you symlink python 3 to version you need. Another option is to do not specify a python version, in this case the tox python version will be used, so just invoke tox target from the required python. This usually means latest available on the machine.

A solution to this would require support of a python_requires. it's non trivial though, but feel free to open a pr and we'll review it.

[ssbarnea]

@gaborbernat I think that you do not see the whole picture here, this is not about my local development environment, this is about CI testing where, for good reasons, we are not allowed to alter the operating system in order to make a test passing.

Still the problem applies to developers too, as even if you have all possible python versions, you may end-up failing to run "tox -e lint" due to wrong pick of version.

In this case I will rename the bug to implement support for python_requires as this would be the right way to do it, also matching the logic already included in a PEP.

@gaborbernat Are you ok to re-phrase the bug and reopen it? Or we should make another one?

[gaborbernat]

Well only if you can come up with a solution willing to implement and we agree on. To be honest I sense a feeling of entitlement and aggressivity in your voice that I'm not comfortable with it, so I would ask you politely to improve your tone.

[MarkusTeufelberger]

To be honest I sense a feeling of entitlement and aggressivity in your voice that I'm not comfortable with it, so I would ask you politely to improve your tone.

Can you cite a few concrete examples please?

[gaborbernat]

3 months later I am finding myself writing the same bug report again...

[MarkusTeufelberger]

Please follow the process outlined at https://github.com/tox-dev/tox/blob/master/CODE_OF_CONDUCT.md#enforcement then. It would also be great to give some examples of politely improved language, since English is often not the native language of people on the internet and it is a nice opportunity to improve. How would you express yourself in this situation?

GitHub

tox-dev/tox

Command line driven CI frontend and development task automation tool - tox-dev/tox

[gaborbernat]

A more polite way of saying this would have been:

Today I've run again in this issue in a different context. Let me explain more in detail what the use case is: (...). I would suggest the following solution: (...). Let me know if I missed something. I'll be happy to implement a solution and open a PR after we agreed on the approach. Thanks for working on this project!

In essence instead of bashing the current limitations let's suggest solutions, and offer to help out, rather than demand that a solution to be provided.

[obestwalter]

Written communication is hard. Especially if it is not the first language for the people involved (which I think is true for everyone in this issue).

We are all here to improve the tool that we are all relying on. In my opinion, pointing out a problem and explaining its implications in detail is just as important as coming up with a concrete solution.

If somebody is in need of more clarification or mediation, they are invited to contact me directly and I will try to help from the position of a relative outsider who just rejoined the activities after a few months of inactivity.

For now let's just enjoy the last days of this year and relax :relaxed:

[ssbarnea]

@gaborbernat I see no reasons for you to worry, I know very well how valuable is voluntary work, I do maintain few packages myself and is not easy work.

My question was purely practical, I wanted to know if a contribution would be desired on this. I assumed some someone has to do it and that it has to pass the reviews. I just wanted to avoid a case where me or someone would spend a good number of hours making a PR that ends up in limbo because the maintainer(s) no longer wanted the feature included (regardless the implementation), happened few times (not here). As implementing this feature is not a coffe-break task, I will first wait to see if others are asking for it before attempting to implement it.

BTW, Happy new year!!! and thanks for the efforth, is really appreciated.