Tox: No module named `pip`

[Seluj78]

tox was installed inside a virtual environement created from python 3.8.0, compile from source.

I am having the error on py36 and py37 only, so here is the tox -rvv from py36 inside my 3.8 virtual env

(potodo) ➜  potodo git:(json) tox -rvv -e py36
using tox.ini: /Users/seluj78/Projects/potodo/tox.ini (pid 80569)
  removing /Users/seluj78/Projects/potodo/.tox/log
using tox-3.14.2 from /Users/seluj78/Projects/potodo/venv/lib/python3.8/site-packages/tox/__init__.py (pid 80569)
GLOB start: packaging 
GLOB sdist-make: /Users/seluj78/Projects/potodo/setup.py
  removing /Users/seluj78/Projects/potodo/.tox/dist
[80576] /Users/seluj78/Projects/potodo$ /Users/seluj78/Projects/potodo/venv/bin/python3.8 setup.py sdist --formats=zip --dist-dir .tox/dist >.tox/log/GLOB-0.log
running sdist
running egg_info
writing potodo.egg-info/PKG-INFO
writing dependency_links to potodo.egg-info/dependency_links.txt
writing entry points to potodo.egg-info/entry_points.txt
writing requirements to potodo.egg-info/requires.txt
writing top-level names to potodo.egg-info/top_level.txt
reading manifest file 'potodo.egg-info/SOURCES.txt'
writing manifest file 'potodo.egg-info/SOURCES.txt'
running check
creating potodo-0.4.0
creating potodo-0.4.0/potodo
creating potodo-0.4.0/potodo.egg-info
copying files to potodo-0.4.0...
copying README.md -> potodo-0.4.0
copying setup.py -> potodo-0.4.0
copying potodo/__init__.py -> potodo-0.4.0/potodo
copying potodo/__main__.py -> potodo-0.4.0/potodo
copying potodo/_github.py -> potodo-0.4.0/potodo
copying potodo/_po_file.py -> potodo-0.4.0/potodo
copying potodo/potodo.py -> potodo-0.4.0/potodo
copying potodo.egg-info/PKG-INFO -> potodo-0.4.0/potodo.egg-info
copying potodo.egg-info/SOURCES.txt -> potodo-0.4.0/potodo.egg-info
copying potodo.egg-info/dependency_links.txt -> potodo-0.4.0/potodo.egg-info
copying potodo.egg-info/entry_points.txt -> potodo-0.4.0/potodo.egg-info
copying potodo.egg-info/not-zip-safe -> potodo-0.4.0/potodo.egg-info
copying potodo.egg-info/requires.txt -> potodo-0.4.0/potodo.egg-info
copying potodo.egg-info/top_level.txt -> potodo-0.4.0/potodo.egg-info
Writing potodo-0.4.0/setup.cfg
creating '.tox/dist/potodo-0.4.0.zip' and adding 'potodo-0.4.0' to it
adding 'potodo-0.4.0'
adding 'potodo-0.4.0/potodo.egg-info'
adding 'potodo-0.4.0/potodo'
adding 'potodo-0.4.0/PKG-INFO'
adding 'potodo-0.4.0/README.md'
adding 'potodo-0.4.0/setup.py'
adding 'potodo-0.4.0/setup.cfg'
adding 'potodo-0.4.0/potodo.egg-info/PKG-INFO'
adding 'potodo-0.4.0/potodo.egg-info/not-zip-safe'
adding 'potodo-0.4.0/potodo.egg-info/SOURCES.txt'
adding 'potodo-0.4.0/potodo.egg-info/entry_points.txt'
adding 'potodo-0.4.0/potodo.egg-info/requires.txt'
adding 'potodo-0.4.0/potodo.egg-info/top_level.txt'
adding 'potodo-0.4.0/potodo.egg-info/dependency_links.txt'
adding 'potodo-0.4.0/potodo/potodo.py'
adding 'potodo-0.4.0/potodo/_github.py'
adding 'potodo-0.4.0/potodo/__init__.py'
adding 'potodo-0.4.0/potodo/_po_file.py'
adding 'potodo-0.4.0/potodo/__main__.py'
removing 'potodo-0.4.0' (and everything under it)

GLOB finish: packaging  after 0.36 seconds
copying new sdistfile to '/Users/seluj78/.tox/distshare/potodo-0.4.0.zip'
package .tmp/package/1/potodo-0.4.0.zip links to dist/potodo-0.4.0.zip (/Users/seluj78/Projects/potodo/.tox)
py36 start: getenv /Users/seluj78/Projects/potodo/.tox/py36
py36 cannot reuse: -r flag
py36 recreate: /Users/seluj78/Projects/potodo/.tox/py36
python3.6 (/usr/local/bin/python3.6) is {'executable': '/usr/local/bin/python3.6', 'name': 'python', 'version_info': [3, 6, 8, 'final', 0], 'version': '3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31) \n[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]', 'is_64': True, 'sysplatform': 'darwin'}
py36 uses /usr/local/bin/python3.6
  removing /Users/seluj78/Projects/potodo/.tox/py36
setting PATH=/Users/seluj78/Projects/potodo/.tox/py36/bin:/Users/seluj78/Projects/potodo/venv/bin:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/usr/local/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Library/Apple/bin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin
[80579] /Users/seluj78/Projects/potodo/.tox$ /Users/seluj78/Projects/potodo/venv/bin/python3.8 -m virtualenv --no-download --python /usr/local/bin/python3.6 py36
Running virtualenv with interpreter /usr/local/bin/python3.6
Already using interpreter /usr/local/bin/python3.6
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/seluj78/Projects/potodo/.tox/py36/bin/python3.6
Also creating executable in /Users/seluj78/Projects/potodo/.tox/py36/bin/python
Installing setuptools, pip, wheel...
done.
py36 installdeps: pytest
setting PATH=/Users/seluj78/Projects/potodo/.tox/py36/bin:/Users/seluj78/Projects/potodo/venv/bin:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/usr/local/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Library/Apple/bin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin
[80586] /Users/seluj78/Projects/potodo$ /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip install pytest
Looking in indexes: https://pypi.python.org/simple, https://pypi.apple.com/simple
Collecting pytest
  Using cached https://files.pythonhosted.org/packages/19/cf/711f1d887cb92c5155c9a1eb338f1b5d2411b50e4492a3b20e4a188a22b2/pytest-5.3.2-py3-none-any.whl
Collecting packaging
  Using cached https://files.pythonhosted.org/packages/cf/94/9672c2d4b126e74c4496c6b3c58a8b51d6419267be9e70660ba23374c875/packaging-19.2-py2.py3-none-any.whl
Collecting more-itertools>=4.0.0
  Using cached https://files.pythonhosted.org/packages/68/03/0604cec1ea13c9f063dd50f900d1a36160334dd3cfb01fd0e638f61b46ba/more_itertools-8.0.2-py3-none-any.whl
Collecting attrs>=17.4.0
  Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting pluggy<1.0,>=0.12
  Using cached https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl
Collecting py>=1.5.0
  Using cached https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl
Collecting wcwidth
  Using cached https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Collecting importlib-metadata>=0.12; python_version < "3.8"
  Using cached https://files.pythonhosted.org/packages/e9/71/1a1e0ed0981bb6a67bce55a210f168126b7ebd2065958673797ea66489ca/importlib_metadata-1.3.0-py2.py3-none-any.whl
Collecting six
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting pyparsing>=2.0.2
  Using cached https://files.pythonhosted.org/packages/c0/0c/fc2e007d9a992d997f04a80125b0f183da7fb554f1de701bbb70a8e7d479/pyparsing-2.4.5-py2.py3-none-any.whl
Collecting zipp>=0.5
  Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
Installing collected packages: six, pyparsing, packaging, more-itertools, attrs, zipp, importlib-metadata, pluggy, py, wcwidth, pytest
Successfully installed attrs-19.3.0 importlib-metadata-1.3.0 more-itertools-8.0.2 packaging-19.2 pluggy-0.13.1 py-1.8.0 pyparsing-2.4.5 pytest-5.3.2 six-1.13.0 wcwidth-0.1.7 zipp-0.6.0
py36 finish: getenv /Users/seluj78/Projects/potodo/.tox/py36 after 6.32 seconds
py36 start: installpkg /Users/seluj78/Projects/potodo/.tox/.tmp/package/1/potodo-0.4.0.zip
py36 inst: /Users/seluj78/Projects/potodo/.tox/.tmp/package/1/potodo-0.4.0.zip
write config to /Users/seluj78/Projects/potodo/.tox/py36/.tox-config1 as '26aeefd96897788c77f488126e45e6a6063c079af252e857b7045abd67bd3f75 /usr/local/bin/python3.6\n3.14.2 0 0 0\n00000000000000000000000000000000 pytest'
setting PATH=/Users/seluj78/Projects/potodo/.tox/py36/bin:/Users/seluj78/Projects/potodo/venv/bin:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/usr/local/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Library/Apple/bin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin
[80590] /Users/seluj78/Projects/potodo$ /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip install --exists-action w .tox/.tmp/package/1/potodo-0.4.0.zip
Looking in indexes: https://pypi.python.org/simple, https://pypi.apple.com/simple
Processing ./.tox/.tmp/package/1/potodo-0.4.0.zip
Collecting polib
  Using cached https://files.pythonhosted.org/packages/30/a2/e407c3b00cace3d7fc8df14d364deeecfeb96044e1a317de583bc26eae58/polib-1.1.0-py2.py3-none-any.whl
Collecting requests
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17
  Using cached https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl
Building wheels for collected packages: potodo
  Building wheel for potodo (setup.py) ... done
  Created wheel for potodo: filename=potodo-0.4.0-cp36-none-any.whl size=7609 sha256=0a1aac14e1a4bd4c40c13728a1f030e307b22728438fec7e4c756f6424f19b88
  Stored in directory: /Users/seluj78/Library/Caches/pip/wheels/c0/cd/ee/1dad18569e006002404c6c1a3939b65ec979a7c4b99c80e46c
Successfully built potodo
Installing collected packages: polib, urllib3, idna, chardet, certifi, requests, potodo
  Found existing installation: potodo 0.4.0
    Not uninstalling potodo at /Users/seluj78/Projects/potodo, outside environment /Users/seluj78/Projects/potodo/.tox/py36/bin/..
    Can't uninstall 'potodo'. No files were found to uninstall.
Successfully installed certifi-2019.11.28 chardet-3.0.4 idna-2.8 polib-1.1.0 potodo-0.4.0 requests-2.22.0 urllib3-1.25.7
py36 finish: installpkg /Users/seluj78/Projects/potodo/.tox/.tmp/package/1/potodo-0.4.0.zip after 2.63 seconds
py36 start: envreport 
setting PATH=/Users/seluj78/Projects/potodo/.tox/py36/bin:/Users/seluj78/Projects/potodo/venv/bin:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/usr/local/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Library/Apple/bin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/usr/local/opt/[email protected]/bin:/Users/seluj78/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin:/usr/local/opt/gettext/bin:/Users/seluj78/.local/bin
[80599] /Users/seluj78/Projects/potodo$ /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip freeze >.tox/py36/log/py36-0.log
ERROR: invocation failed (exit code 1), logfile: /Users/seluj78/Projects/potodo/.tox/py36/log/py36-0.log
============================================================================================================== log start ==============================================================================================================
/Users/seluj78/Projects/potodo/venv/bin/python3.8: No module named pip

=============================================================================================================== log end ===============================================================================================================
_______________________________________________________________________________________________________________ summary _______________________________________________________________________________________________________________
ERROR:   py36: InvocationError for command /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip freeze (exited with code 1)
cleanup /Users/seluj78/Projects/potodo/.tox/.tmp/package/1/potodo-0.4.0.zip

I could not find any logical explanation. All my python installations have pip installed, as well as tox's...

[Seluj78]

BUT

(potodo) ➜  potodo git:(json) /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip freeze 
attrs==19.3.0
certifi==2019.11.28
chardet==3.0.4
idna==2.8
importlib-metadata==1.3.0
more-itertools==8.0.2
packaging==19.2
pluggy==0.13.1
polib==1.1.0
potodo==0.4.0
py==1.8.0
pyparsing==2.4.5
pytest==5.3.2
requests==2.22.0
six==1.13.0
urllib3==1.25.7
wcwidth==0.1.7
zipp==0.6.0

and

(potodo) ➜  potodo git:(json) /Users/seluj78/Projects/potodo/venv/bin/python3.8 -m pip

Usage:   
  /Users/seluj78/Projects/potodo/venv/bin/python3.8 -m pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  config                      Manage local and global configuration.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  debug                       Show information useful for debugging.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 0 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.
  --no-color                  Suppress colored output

So like... what the hell ? 🤔

[gaborbernat]

freeze by default omits pip and setuptools; pass --all to get those too

[Seluj78]

(potodo) ➜  potodo git:(json) /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip freeze --all
attrs==19.3.0
certifi==2019.11.28
chardet==3.0.4
idna==2.8
importlib-metadata==1.3.0
more-itertools==8.0.2
packaging==19.2
pip==19.3.1
pluggy==0.13.1
polib==1.1.0
potodo==0.4.0
py==1.8.0
pyparsing==2.4.5
pytest==5.3.2
requests==2.22.0
setuptools==42.0.2
six==1.13.0
urllib3==1.25.7
wcwidth==0.1.7
wheel==0.33.6
zipp==0.6.0

[gaborbernat]

/Users/seluj78/Projects/potodo/venv/bin/python3.8: No module named pip

that's outside of the tox folder, so not sure how this ticket relates to tox

[Seluj78]

Well as I've just showed you, pip works file outside tox, not inside

[gaborbernat]

do you have a tox.ini that replicates this?

[Seluj78]

[flake8]
max-line-length = 88

[tox]
envlist = py36, py37, py38, flake8, mypy, black
skip_missing_interpreters = True

[testenv]
deps = pytest
commands = pytest

[testenv:flake8]
deps = flake8
commands = flake8 tests/ potodo/

[testenv:black]
deps = black
commands = black --check --diff tests/ potodo/

[testenv:mypy]
deps = mypy
commands = mypy --ignore-missing-imports --strict potodo/

on MacOS 10.14.6, Virtual env created from python 3.8.0rc1

[Seluj78]

https://asciinema.org/a/1RRm2LjWrJq8Z0rNnezDYHy6y

[gaborbernat]

That's strange, as if the call from under /Users/seluj78/Projects/potodo/.tox/py36/bin/python fallbacks to /Users/seluj78/Projects/potodo/venv/bin/python3.8 that doesn't see the pip:

setting PATH=/Users/seluj78/Projects/potodo/.tox/py36/bin:/Users/seluj78/Projects/potodo/venv/bin
[80599] /Users/seluj78/Projects/potodo$ /Users/seluj78/Projects/potodo/.tox/py36/bin/python -m pip freeze >.tox/py36/log/py36-0.log
ERROR: invocation failed (exit code 1), logfile: /Users/seluj78/Projects/potodo/.tox/py36/log/py36-0.log
============================================================================================================== log start ==============================================================================================================
/Users/seluj78/Projects/potodo/venv/bin/python3.8: No module named pip

=============================================================================================================== log end ===============================================================================================================

...

[Seluj78]

Yeah i really don't understand either

[gaborbernat]

Can you replicate this on another machine too? Otherwise debugging this would likely involve modifying various parts of tox... and seeing what's having this effect. If you could replicate this e.g. within a docker image, then I could assist.

[Seluj78]

I'll try

[asottile]

I've seen similar symptoms due to __PYVENV_LAUNCHER__ on macos as well as shebang limits -- doesn't look like the shebang limit is in play here but perhaps something else changed with respect to framework builds? pre-commit does this for example to "fix" this: https://github.com/pre-commit/pre-commit/blob/6850c27dd68dbd118c037e5cdb70957508827f67/pre_commit/main.py#L32-L36

GitHub

pre-commit/pre-commit

A framework for managing and maintaining multi-language pre-commit hooks. - pre-commit/pre-commit

[JordanReiter]

I'm encountering a similar error on Mac OS X. Attempting to run the tox command on this project: https://github.com/AngellusMortis/django_microsoft_auth

[asottile]

@JordanReiter can you show your output as well as how you installed tox?

[Seluj78]

I've ended up not using tox sadly, caused more problems than it was supposed to solve. I compiled all the Python versions I wanted myself and it works fine

Thanks for the great work you guys do, tox rocks !

I'll leave the issue open for now, feel free to close it once other people's problems are resolved !

[howryu]

I encountered same issue. Not sure how to debug this. I ended up installing tox system wide instead of inside virtualenv and trigger system wide tox.

[alfredodeza]

I am having the same problem on OSX. Like in comment https://github.com/tox-dev/tox/issues/1485#issuecomment-566294299 I am able to run the failed command without problems:

ERROR:   py36: InvocationError for command /Users/alfredo/python/anchore-cli/.tox/py36/bin/python -m pip freeze (exited with code 1)
(anchore-cli) $  /Users/alfredo/python/anchore-cli/.tox/py36/bin/python -m pip freeze
anchorecli==0.6.1.dev0
attrs==19.3.0
certifi==2019.11.28
chardet==3.0.4
Click==7.0
idna==2.8
importlib-metadata==1.4.0
more-itertools==8.1.0
packaging==20.1
pluggy==0.13.1
prettytable==0.7.2
py==1.8.1
pyparsing==2.4.6
pytest==5.3.4
python-dateutil==2.8.1
PyYAML==5.3
requests==2.22.0
six==1.14.0
urllib3==1.25.7
wcwidth==0.1.8
zipp==2.0.1

[asottile]

@alfredodeza do you have a way to reproduce this

[alfredodeza]

@asottile yes. On OSX, using latest version (Catalina):

• Install Python 3.6, 3.7, and 3.8 from python.org
• Create a Python3 virtualenv. For example: python3.6 -m venv venv
• Activate and run python setup.py develop
• Install tox, and run.

What I've found is that if I create the virtualenv with 3.6, then tox works for 3.6 but doesn't for any other Python version (e.g. 3.8 fails with the pip module message).

If I create another venv with 3.8, then tox fails on 3.6:

(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ tox -e py36
GLOB sdist-make: /Users/alfredo/python/anchore-cli/setup.py
py36 recreate: /Users/alfredo/python/anchore-cli/.tox/py36
py36 installdeps: pytest
py36 inst: /Users/alfredo/python/anchore-cli/.tox/.tmp/package/1/anchorecli-0.6.1.dev0.zip
py36 installed: You are using pip version 9.0.3, however version 20.0.2 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,-e [email protected]:anchore/anchore-cli@a4f22e6c8f9f08bd9c157efb9b34869134251e15#egg=anchorecli,certifi==2019.11.28,chardet==3.0.4,Click==7.0,filelock==3.0.12,idna==2.8,importlib-metadata==1.4.0,packaging==20.1,pluggy==0.13.1,prettytable==0.7.2,py==1.8.1,pyparsing==2.4.6,python-dateutil==2.8.1,PyYAML==5.3,requests==2.22.0,six==1.14.0,toml==0.10.0,tox==3.14.3,urllib3==1.25.7,virtualenv==16.7.9,zipp==2.1.0
py36 run-test-pre: PYTHONHASHSEED='2438988094'
py36 run-test: commands[0] | pytest -q tests
...............................................................................................................................                                                                                                                                          [100%]
127 passed in 0.35s
___________________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________________
  py36: commands succeeded
  congratulations :)
(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ tox -e py37
GLOB sdist-make: /Users/alfredo/python/anchore-cli/setup.py
py37 create: /Users/alfredo/python/anchore-cli/.tox/py37
py37 installdeps: pytest
py37 inst: /Users/alfredo/python/anchore-cli/.tox/.tmp/package/1/anchorecli-0.6.1.dev0.zip
ERROR: invocation failed (exit code 1), logfile: /Users/alfredo/python/anchore-cli/.tox/py37/log/py37-3.log
================================================================================================================================== log start ===================================================================================================================================
/Users/alfredo/python/anchore-cli/36/bin/python3.6: No module named pip

=================================================================================================================================== log end ====================================================================================================================================
___________________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________________
ERROR:   py37: InvocationError for command /Users/alfredo/python/anchore-cli/.tox/py37/bin/python -m pip freeze (exited with code 1)
(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ clear
(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ
(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ tox -e py36
GLOB sdist-make: /Users/alfredo/python/anchore-cli/setup.py
py36 inst-nodeps: /Users/alfredo/python/anchore-cli/.tox/.tmp/package/1/anchorecli-0.6.1.dev0.zip
py36 installed: You are using pip version 9.0.3, however version 20.0.2 is available.,You should consider upgrading via the 'pip install --upgrade pip' command.,-e [email protected]:anchore/anchore-cli@a4f22e6c8f9f08bd9c157efb9b34869134251e15#egg=anchorecli,certifi==2019.11.28,chardet==3.0.4,Click==7.0,filelock==3.0.12,idna==2.8,importlib-metadata==1.4.0,packaging==20.1,pluggy==0.13.1,prettytable==0.7.2,py==1.8.1,pyparsing==2.4.6,python-dateutil==2.8.1,PyYAML==5.3,requests==2.22.0,six==1.14.0,toml==0.10.0,tox==3.14.3,urllib3==1.25.7,virtualenv==16.7.9,zipp==2.1.0
py36 run-test-pre: PYTHONHASHSEED='2093547480'
py36 run-test: commands[0] | pytest -q tests
........................................................................ [ 56%]
.......................................................                  [100%]
127 passed in 0.38s
___________________________________ summary ____________________________________
  py36: commands succeeded
  congratulations :)
(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ tox -e py37
GLOB sdist-make: /Users/alfredo/python/anchore-cli/setup.py
py37 inst-nodeps: /Users/alfredo/python/anchore-cli/.tox/.tmp/package/1/anchorecli-0.6.1.dev0.zip
ERROR: invocation failed (exit code 1), logfile: /Users/alfredo/python/anchore-cli/.tox/py37/log/py37-5.log
================================== log start ===================================
/Users/alfredo/python/anchore-cli/36/bin/python3.6: No module named pip

=================================== log end ====================================
___________________________________ summary ____________________________________
ERROR:   py37: InvocationError for command /Users/alfredo/python/anchore-cli/.tox/py37/bin/python -m pip freeze (exited with code 1)
(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ tox -e py38
GLOB sdist-make: /Users/alfredo/python/anchore-cli/setup.py
py38 inst-nodeps: /Users/alfredo/python/anchore-cli/.tox/.tmp/package/1/anchorecli-0.6.1.dev0.zip
ERROR: invocation failed (exit code 1), logfile: /Users/alfredo/python/anchore-cli/.tox/py38/log/py38-5.log
================================== log start ===================================
/Users/alfredo/python/anchore-cli/36/bin/python3.6: No module named pip

=================================== log end ====================================
___________________________________ summary ____________________________________
ERROR:   py38: InvocationError for command /Users/alfredo/python/anchore-cli/.tox/py38/bin/python -m pip freeze (exited with code 1)


(36) rocoto[issue-31*] ~/python/anchore-cli ᓆ deactivate
rocoto[issue-31*] ~/python/anchore-cli ᓆ source 38/bin/activate
(38) rocoto[issue-31*] ~/python/anchore-cli ᓆ tox -e py38
GLOB sdist-make: /Users/alfredo/python/anchore-cli/setup.py
py38 recreate: /Users/alfredo/python/anchore-cli/.tox/py38
py38 installdeps: pytest
py38 inst: /Users/alfredo/python/anchore-cli/.tox/.tmp/package/1/anchorecli-0.6.1.dev0.zip
py38 installed: -e [email protected]:anchore/anchore-cli@a4f22e6c8f9f08bd9c157efb9b34869134251e15#egg=anchorecli,certifi==2019.11.28,chardet==3.0.4,Click==7.0,filelock==3.0.12,idna==2.8,packaging==20.1,pluggy==0.13.1,prettytable==0.7.2,py==1.8.1,pyparsing==2.4.6,python-dateutil==2.8.1,PyYAML==5.3,requests==2.22.0,six==1.14.0,toml==0.10.0,tox==3.14.3,urllib3==1.25.7,virtualenv==16.7.9
py38 run-test-pre: PYTHONHASHSEED='4195586144'
py38 run-test: commands[0] | pytest -q tests
........................................................................ [ 56%]
.......................................................                  [100%]
127 passed in 0.32s
___________________________________ summary ____________________________________
  py38: commands succeeded
  congratulations :)

[asottile]

my initial hunch is something to do with framework builds -- as I can't reproduce on linux or windows, will investigate more when I have access to macos

my guess is a fix similar to this

[asottile]

bisecting, it regressed in 43d65b177f6cb9a8757554bf8988f59018b92c0f CC @gaborbernat

[asottile]

this patch fixes it (as expected):

diff --git a/src/tox/session/__init__.py b/src/tox/session/__init__.py
index a2e0ca7..d79f39d 100644
--- a/src/tox/session/__init__.py
+++ b/src/tox/session/__init__.py
@@ -41,6 +41,10 @@ from .commands.show_env import show_envs
 def cmdline(args=None):
     if args is None:
         args = sys.argv[1:]
+    # this environment variable comes from framework builds (particularly on
+    # macos).  without removing it from the environment, subprocesses in
+    # virtualenvs will reference the incorrect site-packages
+    os.environ.pop('__PYVENV_LAUNCHER__', None)
     main(args)

however I don't think this is quite right (there's some care taken to do this elsewhere -- maybe I should do it in the popen call itself?) -- @gaborbernat any thoughts on where you'd prefer I stick this code?

[alfredodeza]

I can confirm the patch fixes the problem for me. I am baffled because I haven't installed pyenv or anything like that. I am going by @asottile 's comment that it is something coming with the Python sources I installed.

At some point I also used brew install python which got me Python 3.7, and I managed to get 3.6 installed that way. That also showed the same behavior. I was able to replicate with downloading from python.org directly.

Thanks for the patch! Would love a release now :)

[gaborbernat]

I generally don't like changing the global environment, instead would prefer to remove only where the popen is called, copying the global, pruning and passing on; so should be here https://github.com/tox-dev/tox/blob/master/src/tox/venv.py#L415-L434

[asottile]

the one advantage to a global patch is this would stop regressing every couple releases 🤷‍♂ (it has already regressed twice)

but I'll see if I can shimmy it into the right place

[maxwellmckinnon]

+1 to hitting this on macOS running tox
python3.7 and 3.8 were installed from python.org

tox.ini
envlist = {s}-py{37,38}

action: s-py37, msg: envreport
cwd: /Users/mm/dev/audioanalysis
cmd: /Users/mm/dev/audioanalysis/.tox/s-py37/bin/python -m pip freeze
/Users/mm/.venvs/TA3.8/bin/python3.8: No module named pip

[maxwellmckinnon]

Downgrading quite a few versions back eventually fixed it for me pip install tox==3.0.*

[asottile]

tox<3.8 should be sufficient -- the regression was introduced in 3.8.0

[dchimeno]

Same here.
Using tox<3.8 and it works.

[gaborbernat]

So this was an upstream issue, fixed at virtualenv level via https://github.com/pypa/virtualenv/pull/1749

[anentropic]

I am getting this issue with tox 3.20.0, with tox installed via pipx

if the fix is in the non-stdlib release of virtualenv how do we get the fix with tox?

[asottile]

upgrade virtualenv

[anentropic]

I had upgraded virtualenv but it still wasn't working

it's working now...

I had to:

1. pipx uninstall tox
2. pip install --upgrade virtualenv
3. pipx install tox

pipx creates a virtualenv for the packages it installs, so this was needed to force pipx to recreate the virtualenv for tox using the latest virtualenv package