Pipx: Can not install pkg when specifying PyPy

[xsc27]

I use CPython and PyPy. I am not sure if this issue occurs with other Python builds.

❯ pipx install --python pypy3 youtube-dl

Traceback (most recent call last):
  File "/home/demo/.local/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/home/demo/.local/lib/python3.8/site-packages/pipx/main.py", line 590, in cli
    return run_pipx_command(parsed_pipx_args)
  File "/home/demo/.local/lib/python3.8/site-packages/pipx/main.py", line 162, in run_pipx_command
    return commands.install(
  File "/home/demo/.local/lib/python3.8/site-packages/pipx/commands/commands.py", line 62, in install
    venv.create_venv(venv_args, pip_args)
  File "/home/demo/.local/lib/python3.8/site-packages/pipx/venv.py", line 139, in create_venv
    pipx_pth.write_text(f"{shared_libs.site_packages}\n", encoding="utf-8")
  File "/home/demo/.pyenv/versions/3.8.1/lib/python3.8/pathlib.py", line 1248, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors) as f:
  File "/home/demo/.pyenv/versions/3.8.1/lib/python3.8/pathlib.py", line 1215, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/demo/.pyenv/versions/3.8.1/lib/python3.8/pathlib.py", line 1071, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/demo/.local/pipx/venvs/youtube-dl/site-packages/pipx_shared.pth'

[itsayellow]

It looks like venv.create_venv() is trying to save pipx_shared.pth to a non-existent directory.

It's trying to save to

/home/demo/.local/pipx/venvs/youtube-dl/site-packages/

When it should be something more like

/home/demo/.local/pipx/venvs/youtube-dl/lib/python3.7/site-packages/

Or something equivalent with a pypy in it.

[itsayellow]

I can confirm this.

verbose output:
```

pipx install --python pypy3 --verbose youtube-dl
pipx > (_package_name_from_spec:93): Determined package name: youtube-dl
pipx > (_package_name_from_spec:94): Package name determined in 0.0s
pipx > (run_subprocess:112): running pypy3 -m venv --without-pip /Users/mclapp/.local/pipx/venvs/youtube-dl
pipx > (run_subprocess:112): running /Users/mclapp/.local/pipx/venvs/youtube-dl/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /Users/mclapp/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))

pipx > (rmdir:18): removing directory /Users/mclapp/.local/pipx/venvs/youtube-dl
Traceback (most recent call last):
File "/usr/local/bin/pipx", line 11, in
load_entry_point('pipx', 'console_scripts', 'pipx')()
File "/Users/mclapp/git/pipx/src/pipx/main.py", line 574, in cli
return run_pipx_command(parsed_pipx_args)
File "/Users/mclapp/git/pipx/src/pipx/main.py", line 156, in run_pipx_command
include_dependencies=args.include_deps,
File "/Users/mclapp/git/pipx/src/pipx/commands/commands.py", line 62, in install
venv.create_venv(venv_args, pip_args)
File "/Users/mclapp/git/pipx/src/pipx/venv.py", line 139, in create_venv
pipx_pth.write_text(f"{shared_libs.site_packages}\n", encoding="utf-8")
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1235, in write_text
with self.open(mode='w', encoding=encoding, errors=errors) as f:
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1203, in open
opener=self._opener)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1058, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/mclapp/.local/pipx/venvs/youtube-dl/site-packages/pipx_shared.pth'

[itsayellow]

I made my own venv using pypy3, and it appears if you run import sysconfig; print(sysconfig.get_path('purelib')) it actually gives you a valid directory:

> venv/bin/python -c "import sysconfig; print(sysconfig.get_path('purelib'))"
/Users/mclapp/venv/site-packages

This is not the typical cpython site-packages directory path, but it does exist in my venv.

[itsayellow]

The problem seems to be that with --without-pip pypy3's venv does not create a site-packages directory at all:

> pypy3 -m venv --without-pip venv
> ls venv
bin/        include/    lib/        pyvenv.cfg
> rm -rf venv
> pypy3 -m venv venv   
> ls venv
bin/           include/       lib/           pyvenv.cfg     site-packages/

[itsayellow]

I think this is a bug with pypy and/or purelib. The top-level site-package directory seems spurious. Always site-packages exists in lib/python*/site-packages where it should be, it's just that purelib in pypy3 never reports that location:

> pypy3 -m venv venv
> ls venv/lib/python3.6
site-packages/
> rm -rf venv          
> pypy3 -m venv --without-pip venv
> ls venv/lib/python3.6
site-packages/

[uranusjr]

The top-level site-packages directory is a PyPy-specific thing; virtualenv has a specific layout setup for it as well. It is only reported if it exists:

$ pypy3 -m venv test2 --without-pip

$ test2/bin/python -c "import sys
for p in sys.path:
 print(p)
"

/usr/local/Cellar/pypy3/7.0.0/libexec/lib_pypy/__extensions__
/usr/local/Cellar/pypy3/7.0.0/libexec/lib_pypy
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/lib-tk
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/plat-darwin
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/plat-mac
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/plat-mac/lib-scriptpackages

$ mkdir test2/site-packages
$ test2/bin/python -c "import sys
for p in sys.path:
 print(p)
"

/usr/local/Cellar/pypy3/7.0.0/libexec/lib_pypy/__extensions__
/usr/local/Cellar/pypy3/7.0.0/libexec/lib_pypy
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/lib-tk
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/plat-darwin
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/plat-mac
/usr/local/Cellar/pypy3/7.0.0/libexec/lib-python/3/plat-mac/lib-scriptpackages
/private/tmp/test2/site-packages

Notice how the last command reports an empty directory.

IMO pipx should simply make sure the parent site-packages exists when it creates the pth file.

[gaborbernat]

Always site-packages exists in lib/python*/site-packages where it should be, it's just that purelib in pypy3 never reports that location:

This is not true. Python implementations/distributions are free to customize where they want their site-packages to be; both via the sysconfig or the distutils - https://docs.python.org/3/library/sysconfig.html#installation-paths (pip notably uses the later). One should be reading those configuration files to find out where the platform/pure library are to be. While true that the most popular python implementation often sets the site-package to lib/python*/site-packages this is not mandatory, just the case in general. PyPy always uses the <prefix> / site-packages folder. Furthermore, the virtual environment creation is not enforced to create the pure/platform folder. Only to add it to the sys.path if these exists. So if one does not install the seed packages (pip/setuptools/wheel) it's fine to not create it. Note virtualenv 20+ does offer the guarantee that the folder will be created (see https://github.com/pypa/virtualenv/blob/master/src/virtualenv/create/via_global_ref/builtin/via_global_self_do.py#L84), but the venv module will not.

PS. Note similarly the script folder may be customized, see https://github.com/pypa/virtualenv/blob/master/src/virtualenv/create/describe.py#L30-L40