Tox: option to skip a factor level

[gaborbernat]

For example if {py27,py36}-{nocov,cov,diffcov}. @warsaw

In this case the user might want to not run all non diffcov envs.

[warsaw]

A little more detail. In an example such as above, we generally run all the factors, however if we create an sdist, then unpack it and run tox in the unpacked directory, all -diffcov environments will fail. This is because diffcov only runs in a git repo (because it does diff comparisons against master). So when running in a non-git repository (e.g. an unpacked sdist), we want to skip all the diffcov environments.

[gaborbernat]

@obestwalter @asottile @warsaw

Here's what I propose, what if the users could do fnmatch/regex filtering in env specification?

e.g. -e '.*-(nocov|diffcov)'

[rpkilby]

btw, this would conflict with passing a list of test env names (eg, tox -e isort,lint)

[gaborbernat]

maybe have a separate env var for it then?

[rpkilby]

Maybe -re?

[gaborbernat]

I would be even fine with having just an environment variable (TOX_ENV_FILTER). This is an advanced feature for advanced users I would believe.

[warsaw]

@gaborbernat I think that would be good enough for me.