Tox: virtualenv raises deprecation warnings on python 3.4+

[rpkilby]

The virtualenv package has an embedded version of site.py and distutils/__init__.py that are copied over to a virtualenv's lib/pythonX.X directory during creation. These files are not compatible with python 3.4+ and will raise deprecation warnings on invocation (See: #555 and #955). While this is not an issue for most tox usage, it does prevent users from raising the PYTHONWARNINGS level to error in tox, since the embedded files will raise the deprecations before the test suite can run.

I usually include a warnings build in CI, so we can be aware of upcoming deprecations/API changes in our dependencies. For example,

[testenv:warnings]
ignore_outcome = True
commands = python -Werror runtests.py {posargs}
deps =
    https://github.com/django/django/archive/master.tar.gz

You can cause virtualenv to fail by either including either PYTHONWARNINGS=error in your tox config or by running python with the -Werror flag.

• PYTHONWARNINGS causes virtualenv creation to fail.
• The -Werror flag doesn't affect virtualenv creation, but will cause python to fail to load modules before the command can run.

PYTHONWARNINGS

[tox]
envlist = py36

[testenv]
commands = python runtests.py
setenv =
    PYTHONDONTWRITEBYTECODE=1
    PYTHONWARNINGS=error

Using base prefix '/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6'
Traceback (most recent call last):
  File ".../lib/python3.6/site-packages/virtualenv.py", line 2328, in <module>
    main()
  File ".../lib/python3.6/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File ".../lib/python3.6/site-packages/virtualenv.py", line 925, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File ".../lib/python3.6/site-packages/virtualenv.py", line 1132, in install_python
    copy_required_modules(home_dir, symlink)
  File ".../lib/python3.6/site-packages/virtualenv.py", line 1039, in copy_required_modules
    import imp
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 33, in <module>
    DeprecationWarning, stacklevel=2)
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
Running virtualenv with interpreter /usr/local/bin/python3.6

-Werror

[tox]
envlist = py36

[testenv]
commands = python -Werror runtests.py

Failed to import the site module
Traceback (most recent call last):
  File ".../.tox/py36-djangomaster/bin/../lib/python3.6/site.py", line 703, in <module>
    main()
  File ".../.tox/py36-djangomaster/bin/../lib/python3.6/site.py", line 683, in main
    paths_in_sys = addsitepackages(paths_in_sys)
  File ".../.tox/py36-djangomaster/bin/../lib/python3.6/site.py", line 282, in addsitepackages
    addsitedir(sitedir, known_paths)
  File ".../.tox/py36-djangomaster/bin/../lib/python3.6/site.py", line 204, in addsitedir
    addpackage(sitedir, name, known_paths)
  File ".../.tox/py36-djangomaster/bin/../lib/python3.6/site.py", line 165, in addpackage
    f = open(fullname, "rU")
DeprecationWarning: 'U' mode is deprecated

System details:

• os: OS X 10.9
• python: 3.6.0
• pip list:
  pip: 9.0.1
  pluggy: 0.4.0
  py: 1.4.32
  setuptools: 28.8.0
  tox: 2.5.0
  virtualenv: 15.1.0

[RonnyPfannschmidt]

this is an issue with virtualenv, update it and/or check with upstream

[rpkilby]

See the referenced issues - there doesn't seem to be much activity on the virtualenv front. I forgot to mention this:

Any reason python 3's venv module is not used instead of virtualenv?

[RonnyPfannschmidt]

@rpkilby mainly because it is hard to support this and python versions that don't have venv, it isn't easy to add a correct abstraction layer (its of course very easy to add some kind of abstraction layer, but the cost down the line can be immense because apis are "forever")

[rpkilby]

Did a little work in #438, and the results were fairly straightforward. It might be worth reevaluating venv when python 3.3 is deprecated in September (python 3.3's venv doesn't have the --copies option).

Overall, from a maintenance/complexity perspective this didn't seem too burdensome? Some thoughts:

• venv has feature parity with virtualenv. Swapping implementations was pretty simple.
• The use of virtualenv/venv would basically be defined by the python2/3 split, which is easy to understand conceptually. "Tox uses the venv module internally to manage venvs, uses virtualenv for backwards compatibility in python 2 distributions."
• No extra API or abstraction was necessary. I did add a private VirtualEnv._module() method, but this was a small convenience and not necessary.

---

As a side note, -Werror isn't raising the deprecation warnings directly. This is a weird interaction with pytest/pytest-cov that somehow invokes and angers the site module. This was confusing at first, because the exception doesn't provide any context of the calling method, so it appears as if the virtualenv is borked, causing the python command to fail.

[RonnyPfannschmidt]

it may come from cov-core, which ensures coverage tracking can span into subprocesses

[rpkilby]

I thought it was the subprocess handling as well, but the module code doesn't even execute. It turns out that it's the use of a .pth file, which hooks into the site module machinery.

[Zac-HD]

For the benefit of future readers, @rpkilby has since published the tox-venv package (thanks!), which fixes this issue at the cost of unsupported pythons. See #630 for discussion.

[gaborbernat]

closing in favour of https://github.com/tox-dev/tox/issues/448

[rpkilby]

Hi @gaborbernat. This ticket is about how virtualenv itself is not forward compatible with future versions of python, and may need to be replaced by venv. I don't see how this is related to #448, which is about how virtualenv upgrades pip/setuptools/wheel after a virtualenv is created.

[gaborbernat]

the general problem is being able to control the versions of packages after creating the venv

[rpkilby]

That's a separate issue. This issue is the basis for #630 and tox-venv.

[gaborbernat]

However, it will be solved as part of #448; which has the solution of merging tox-venv into core.