Virtualenv: Include Standard Library scripts in virtualenv's scripts

[native-api]

It looks like 2to3 is currently not being redirected by Virtualenv. See https://travis-ci.community/t/2to3-command-not-found-in-virtualenv-in-bionic/4495 for a case when this was unexpected by the user -- since it's normally available on PATH for a Linux installation.

In the light of Py2 nearing EOL, the demand for it is going to rise!

---

• [X] Minimal reproducible example or detailed descriptions

$ which 2to3
/home/vmuser/.pyenv/shims/2to3

$ pyenv install 3.6.9
<...>

$ ~/.pyenv/versions/3.6.9/bin/python -m pip install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/db/9e/df208b2baad146fe3fbe750eacadd6e49bcf2f2c3c1117b7192a7b28aec4/virtualenv-16.7.2-py2.py3-none-any.whl (3.3MB)
    100% |████████████████████████████████| 3.3MB 1.3MB/s 
Installing collected packages: virtualenv
Successfully installed virtualenv-16.7.2

$ ~/.pyenv/versions/3.6.9/bin/python -m virtualenv test
Using base prefix '/home/vmuser/.pyenv/versions/3.6.9'
New python executable in /home/vmuser/test/bin/python
Installing setuptools, pip, wheel...
done.

$ . test/bin/activate

(test) $ which 2to3
/home/vmuser/.pyenv/shims/2to3

The expected behavior was for the last which to return a path inside virtualenv -- the same as for e.g. python and pip.

• [X] OS and pip list output
  Ubuntu Bionic Desktop x64

$ ~/.pyenv/versions/3.6.9/bin/python -m pip list
Package    Version
---------- -------
pip        18.1   
setuptools 40.6.2 
virtualenv 16.7.2 

[pradyunsg]

Does this still occur after doing pyenv rehash?

What does the shim execute when run before or after pyenv rehash?

I have a feeling that this has something to do with pyenv.

[native-api]

Does this still occur after doing pyenv rehash?

Yes.

What does the shim execute when run before or after pyenv rehash?

Since system Python is selected for which the apt package with 2to3 is not installed, it says

pyenv: 2to3: command not found

The `2to3' command exists in these Python versions:
<list>

in both cases.

I have a feeling that this has something to do with pyenv.

pyenv is irrelevant. The expected behavior was for the second which to return virtualenv's shim like it does for python. The bug is that virtualenv doesn't create one. (I thought this was obvious. Probably not.)

[pradyunsg]

Ah, okay... What's PATH after activation?

(I'm not on a machine where I can try reproducing this right now)

[native-api]

(test) vmuser@ubuntuvm:~$ echo $PATH
/home/vmuser/test/bin:/home/vmuser/.rbenv/shims:/home/vmuser/.rbenv/bin:/home/vmuser/.pyenv/shims:/home/vmuser/.pyenv/bin:/home/vmuser/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/vmuser/.rvm/bin
(test) vmuser@ubuntuvm:~$ ls /home/vmuser/test/bin
activate       activate_this.py  pip     python3
activate.csh   activate.xsh      pip3    python3.6
activate.fish  easy_install      pip3.6  python-config
activate.ps1   easy_install-3.6  python  wheel

[pradyunsg]

As a workaround, you can use python -m lib2to3 inside your virtualenv, with equivalent functionality.

[pradyunsg]

/cc other @pypa/virtualenv-committers if they think adding scripts from the standard library to the virtualenv when creating them, is a good idea.

I'm ambivalent.

[pfmoore]

I'm -0.5. It's extra clutter that's not been flagged as an issue until now, so I assume the need is rare. I'm not convinced that Python 2 nearing EOL makes a difference here. The extra scripts aren't on PATH in a non-virtual environment on Windows either, so it's not like the expectation that they are present is universal.

[gaborbernat]

I think this should work as general approach. All applications at host environment should be available at virtual environment level.

[pradyunsg]

I'm not convinced that Python 2 nearing EOL makes a difference here.

+1

That's not my concern here either. :)

The extra scripts aren't on PATH in a non-virtual environment on Windows either

Oh. Aren't these in the Scripts/ directory? My understanding was that the current python.org installers add that directory to PATH and thus they can be executed.

All applications at host environment should be available at virtual environment level.

Uhm... "applications" is a little vague here -- are you referring to the scripts or something else? Could you please clarify what you mean?

Note that if we do this, I'm a strong -1 on including all scripts from the host environment anyway and ambivalent on including the ones that backed by the standard library.

[pfmoore]

Oh. Aren't these in the Scripts/ directory?

No, all that's in Scripts are entry point wrappers for installed libraries (pip, easy_install and wheel in a base install). Looking a bit further, 2to3.py is in Tools/scripts (along with a fairly mixed bunch of 60+ other scripts), but that directory isn't put on PATH, even if the user selects "Add Python to your PATH".

Note that if we do this, I'm a strong -1 on including all scripts from the host environment anyway and ambivalent on including the ones that backed by the standard library.

At a minimum, I'd argue that we should not add more to PATH than a basic install does. Which means that even if we do this, we don't do it on Windows.

pradyunsg changed the title Include Standard Library scripts in virtualenv's bin/ Include Standard Library scripts in virtualenv's scripts 2 hours ago

Just noting that this is a slight scope change. The OP was only specifically interested in 2to3, and in particular that activating a virtualenv didn't shadow the system 2to3. Whether that's important I can't comment on (given that I've argued for this not happening on Windows, I've no relevant experience on Unix to say whether I feel this is a problem). But without knowing what other scripts are "Standard Library scripts" I can't comment if this is a significant point.

Hmm, pyvenv is probably another "Standard Library script", and I'm pretty sure we don't want to shadow the system version of that with one that runs venv from a virtualenv environment (as that just causes complications for no practical benefit).

[native-api]

The OP was... interested... in particular that activating a virtualenv didn't shadow the system 2to3.

(I'm not sure what this phrase puts in my mouth, it could be interpreted both ways.)

For clarity, I'm asking for virtualenv to create a shim for 2to3.

(Let me edit the original post with the expected behavior, I see that me omitting it is causing confusion.)

[native-api]

(Let me edit the original post with the expected behavior)

Done. I also clarified why the user expected it to be on PATH.

[pfmoore]

I'm not sure what this phrase puts in my mouth

Sorry for misrepresenting your comments and thanks for the clarification! (It looks like I did correctly understand what you were asking for, I just restated it badly...)

I also clarified why the user expected it to be on PATH.

The link you referenced is a bit confusing. The user seems to have a process that works on trusty and xenial, but not on bionic. It's not clear to me why that would be the case if the issue is in virtualenv. (Note: this is a digression - regardless, it's clearly the case that virtualenv doesn't copy 2to3.py, so the discussion of whether we should is valid even if the original issue is more subtle).

I'm asking for virtualenv to create a shim for 2to3.

Note that we'll never create a shim. All we'd do is copy the 2to3.py script. If a shim is needed (that's a pyvenv thing, I think) then it wouldn't be us that handled that.

[pradyunsg]

No, all that's in Scripts are entry point wrappers for installed libraries (pip, easy_install and wheel in a base install).

Oooooooooooooo. Okay.

Just noting that this is a slight scope change.

I don't think 2to3 is a special snowflake -- if we add that, I expect someone to come around asking for the rest. We should either fully do this or not at all. Being stuck midway isn't something we'd want here.

[pradyunsg]

(I hate the mobile UI)

[native-api]

It's "special" in that it's backed by the standard library AND is in PATH for a standard Linux installation.
There aren't actually many other tools like this. Other than 2to3, it's only pydoc, idle and pyvenv:

$ ls ~/.pyenv/versions/3.6.9/bin
2to3          easy_install-3.6  idle3.6  pip3.6  pydoc3.6  python3.6         python3.6m         python-config  virtualenv
2to3-3.6      idle              pip      pydoc   python    python3.6-config  python3.6m-config  pyvenv
easy_install  idle3             pip3     pydoc3  python3   python3.6-gdb.py  python3-config     pyvenv-3.6

[native-api]

(also python-config which is already being redirected)

[pradyunsg]

Ah, thanks for the clarification @native-api! Much appreciated. Along with @pfmoore's explanation of how this is a not-on-Windows issue, that helps my understanding. :)

I'm still ambivalent on including it.

[pfmoore]

Other than 2to3, it's only pydoc, idle and pyvenv

I'd be curious to know who chose that list. Is it specifically pyvenv? Ubuntu (bash on Windows, and a docker image with python3 installed) doesn't seem to have 2to3 by default, although the Python docker image has the same binaries you mentioned (plus pip, easy_install and wheel, which come from installed packages). It seems a bit distribution-dependent :-( At least the list of relevant binaries seems consistent though.

BTW, as I said above, even if we do make this change, I strongly advise not including pyvenv.

[native-api]

Ubuntu (bash on Windows, and a docker image with python3 installed) doesn't seem to have 2to3 by default

That's because in Ubuntu, 2to3 is moved to a separate package:

$ apt-file search 2to3 | grep -E '/2to3[^/]*$'
2to3: /usr/bin/2to3
<...>
python2.7: /usr/bin/2to3-2.7
<...>

By "standard Linux installation", I meant the logic that's intrinsic to Python's Linux build script, without distros' meddling -- e.g. if you install from source (which is what pyenv does).

[gpiks]

I could take this up and will try to work on this.

[gaborbernat]

Note, this would replace our current hack of adding pydoc as part of the activation script.