Pipx: [windows] pipx seems to install a package correctly but claims that "symlink missing or pointing to unexpected location"

[sachinmaiya]

Describe the bug

Installed flake8 on wsl (windows subsystem for linux) via pipx. Installation completes successfully but pipx reports in red that "symlink missing or pointing to unexpected location". I notice the same on Windows native.

The reality is that the installation has succeeded and I am able to use the installed flake8.

I have tried with other packages and packages that I have created myself; I get a similar error.

Thank you for pipx, BTW; it has saved me quite a bit of bother. :)

How to reproduce

16-Mon/09:53 $ pipx install --verbose flake8
pipx > (run_pipx_command:134): Virtual Environment location is /home/snm/.local/pipx/venvs/flake8
pipx > (run:97): running /usr/bin/python3 -m venv --without-pip /home/snm/.local/pipx/venvs/flake8
pipx > (run:97): running /home/snm/.local/pipx/venvs/flake8/bin/python -m pip install flake8
Collecting flake8
  Using cached https://files.pythonhosted.org/packages/26/de/3f815a99d86eb10464ea7bd6059c0172c7ca97d4bdcfca41051b388a653b/flake8-3.7.8-py2.py3-none-any.whl
Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting mccabe<0.7.0,>=0.6.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting pycodestyle<2.6.0,>=2.5.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/0e/0c/04a353e104d2f324f8ee5f4b32012618c1c86dd79e52a433b64fceed511b/pycodestyle-2.5.0-py2.py3-none-any.whl
Collecting pyflakes<2.2.0,>=2.1.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/84/f2/ed0ffb887f8138a8fe5a621b8c0bb9598bfb3989e029f6c6a85ee66628ee/pyflakes-2.1.1-py2.py3-none-any.whl
Installing collected packages: entrypoints, mccabe, pycodestyle, pyflakes, flake8
Successfully installed entrypoints-0.3 flake8-3.7.8 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1
WARNING: You are using pip version 19.2.2, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
pipx > (_symlink_package_apps:554): ⚠️  Filenexists at /home/snm/.local/bin/flake8 and points to /home/snm/.local/bin/flake8, not /home/snm/.local/pipx/venvs/flake8/bin/flake8. Not modifying.
  installed package flake8 3.7.8, Python 3.6.8
    - flake8 (symlink missing or pointing to unexpected location)
done! ✨ 🌟 ✨

16-Mon/09:54 $ pipx list
venvs are in /home/snm/.local/pipx/venvs
apps are exposed on your $PATH at /home/snm/.local/bin
   package flake8 3.7.8, Python 3.6.8
    - flake8 (symlink missing or pointing to unexpected location)

Expected behavior

Perhaps, the error is being misreported?

[cs01]

File exists at /home/snm/.local/bin/flake8 and points to /home/snm/.local/bin/flake8, not /home/snm/.local/pipx/venvs/flake8/bin/flake8. Not modifying.

indicates the file already exists. It looks like it's been installed with pip3 install --user. What you can do is either remove the file and then try installing with pipx, or add the --force flag to pipx:

pipx install flake8 --force

Running which flake8 might also help determine which is being used, as there might be more than one on your $PATH.

[sachinmaiya]

You are right. I had flake8 installed via --user in WSL. I removed that installation (which flake8 prints nothing on the screen) and tried installing again in windows cmd; I still get the same report (barring the File exists error:

U:\>pipx install --verbose flake8
pipx > (run_pipx_command:134): Virtual Environment location is C:\Users\snm\.local\pipx\venvs\flake8
pipx > (run:97): running c:\users\snm\appdata\local\programs\python\python37-32\python.exe -m venv --without-pip C:\Users\snm\.local\pipx\venvs\flake8
pipx > (run:97): running C:\Users\snm\.local\pipx\venvs\flake8\Scripts\python.exe -m pip install flake8
Collecting flake8
  Using cached https://files.pythonhosted.org/packages/26/de/3f815a99d86eb10464ea7bd6059c0172c7ca97d4bdcfca41051b388a653b/flake8-3.7.8-py2.py3-none-any.whl
Collecting pycodestyle<2.6.0,>=2.5.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/0e/0c/04a353e104d2f324f8ee5f4b32012618c1c86dd79e52a433b64fceed511b/pycodestyle-2.5.0-py2.py3-none-any.whl
Collecting pyflakes<2.2.0,>=2.1.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/84/f2/ed0ffb887f8138a8fe5a621b8c0bb9598bfb3989e029f6c6a85ee66628ee/pyflakes-2.1.1-py2.py3-none-any.whl
Collecting mccabe<0.7.0,>=0.6.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Using cached https://files.pythonhosted.org/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Installing collected packages: pycodestyle, pyflakes, mccabe, entrypoints, flake8
Successfully installed entrypoints-0.3 flake8-3.7.8 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1
WARNING: You are using pip version 19.2.2, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
  installed package flake8 3.7.8, Python 3.7.4
  These apps are now globally available
    - flake8.exe
    - flake8 (symlink missing or pointing to unexpected location)
done!

[cs01]

Try running

pipx install --force --verbose flake8

[sachinmaiya]

I ran pipx uninstall flake8 first.
I think I get the same error and flake8 seems to work after the install as before. Note that I included "--no-cache-dir" but the result is the same without it.

>pipx install --force --verbose --pip-args '--no-cache-dir' flake8
pipx > (run_pipx_command:134): Virtual Environment location is C:\Users\snm\.local\pipx\venvs\flake8
pipx > (run:97): running c:\users\snm\appdata\local\programs\python\python37-32\python.exe -m venv --without-pip C:\Users\snm\.local\pipx\venvs\flake8
pipx > (run:97): running C:\Users\snm\.local\pipx\venvs\flake8\Scripts\python.exe -m pip install --no-cache-dir flake8
Collecting flake8
  Downloading https://files.pythonhosted.org/packages/26/de/3f815a99d86eb10464ea7bd6059c0172c7ca97d4bdcfca41051b388a653b/flake8-3.7.8-py2.py3-none-any.whl (70kB)
     |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 71kB 4.5MB/s
Collecting pyflakes<2.2.0,>=2.1.0 (from flake8)
  Downloading https://files.pythonhosted.org/packages/84/f2/ed0ffb887f8138a8fe5a621b8c0bb9598bfb3989e029f6c6a85ee66628ee/pyflakes-2.1.1-py2.py3-none-any.whl (59kB)
     |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 61kB 4.1MB/s
Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Downloading https://files.pythonhosted.org/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting pycodestyle<2.6.0,>=2.5.0 (from flake8)
  Downloading https://files.pythonhosted.org/packages/0e/0c/04a353e104d2f324f8ee5f4b32012618c1c86dd79e52a433b64fceed511b/pycodestyle-2.5.0-py2.py3-none-any.whl (51kB)
     |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 51kB ...
Collecting mccabe<0.7.0,>=0.6.0 (from flake8)
  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Installing collected packages: pyflakes, entrypoints, pycodestyle, mccabe, flake8
Successfully installed entrypoints-0.3 flake8-3.7.8 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1
WARNING: You are using pip version 19.2.2, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
  installed package flake8 3.7.8, Python 3.7.4
  These apps are now globally available
    - flake8.exe
    - flake8 (symlink missing or pointing to unexpected location)
done!

[cs01]

Confirmed this is failing in unit tests in the new Travis tests (thanks @jayvdb). Added a TODO to fix it if any windows devs feel like taking a shot at it.

https://github.com/pipxproject/pipx/blob/047a0be84ccfc70d40886eb9cfdef8b45bac5113/tests/test_install.py#L31

[itsayellow]

Some more information on this one:

venv_metadata_inspectory.py function get_apps() on Windows:
The correct binaries (i.e. those that end in .exe) are found in the RECORD metadata. (In the loop for line in dist.get_metadata_lines("RECORD"):.

All the incorrect entries on Windows (those with no .exe ending.) are coming from the 'console_scripts' section in the metadata. (In the loop for section in ["console_scripts", "gui_scripts"]:

[itsayellow]

Actually reading up on this, this might make sense.

The console_scripts section comes from pkg_resources.get_entry_map() which is the full map of the entry points for the package specified. For windows scripts, the entry point names will not correspond to the actual installed name, because a .exe is tacked on the end of Windows apps.

RECORD has to do with what/where things actually got installed:
https://www.python.org/dev/peps/pep-0376/#record

This makes me wonder if we should only pay attention to the RECORD items, since they are a list of things that actually got installed?

[uranusjr]

It would be tedious to figure out what entries in RECORD are actually console scripts, since a Python package can contain arbitrary files, including .exe. pkg_resources.get_entry_map() may be more reliable. It might be a good idea to filter entries in it against RECORD though.

[itsayellow]

Actually we already filter RECORD based on what's also in the venv bin directory, which seems pretty reliable.

But I just found the exception to RECORD. When I installed a package with the --editable option set, there was no RECORD file produced. So it looks like searching the entry_points is necessary to account for editable packages that may not have a RECORD file.

I think the simplest fix would be on Windows to add a check in the for loop for entry_points that looks in the venv bin_dir for both the entry_point name and the entry_point name with .exe on the end. We already have if-clauses to verify possible apps exist in the venv bin_dir in the other metadata search loops in get_apps().

[ardeaf]

What is the intended behavior? I'm on a windows system and have been dealing with just running app.exe instead of app from Git Bash, but would love if I could get rid of the exe via a symlink. Is that possible without using wsl? I'm not used to developing on a windows machine so not sure if it's the norm that all cli apps have exe at the end 🤷‍♂️

[uranusjr]

@ardeaf I am not on a Windows machine right now, but Windows automatically appends .exe when you call app (the actual behaviour is more complex but let’s ignore that). You shouldn’t need symlinks for that.

[ardeaf]

@uranusjr You're right! It works, it just doesn't auto complete for some reason. thanks. Though I should also mention that it mentions the same error -- that the symlink is missing or pointing to an unexpected location.

[itsayellow]

I've been experimenting with fixing this. Blech, the Windows stuff (or maybe setuptools stuff) turns out to be more complicated than I originally thought.

Most of the time it just means that you need to use the entrypoint name with a .exe tacked on the end.

EXCEPT. If you have an editable local-path-based package to install, there are 3 files that setuptools (I believe pip uses setuptools) creates for each entry point:

• <entrypoint_name>.exe
• <entrypoint_name>.exe.manifest
• <entrypoint_name>-script.py

I tried to only have pipx link the <entrypoint_name>.exe file into .local/bin, but when running it, it got upset, expecting <entrypoint_name>-script.py to be in the same path.

I then tried to get clever and stick a .pth file inside .local/bin that pointed to the venv bin directory so the .exe file could find its <entrypoint_name>-script.py companion. That did NOT work. It really wants the <entrypoint_name>-script.py file to be in the same directory.

Does anyone have any pointers to information on these three files that get created on Windows for every entry_point on an editable install? I kept googling and couldn't find anything that describes it.

[uranusjr]

I don’t think I’ve seen the manifest file (well I do for Windows development, but not Python entry points), but the script.py-.exe combination is an artifact from the “legacy” Python build format (eggs). You only see them if you use setup.py to install a package, which is almost never these days, except for pip install -e, which still uses it because the editable install is intentionally left out from PEP 517 due to resource constraints, and is still in pre-PEP discussion (words are there will be a PEP in the following months, but don’t hold your breath for that).

The legacy format of .exe, as you discovered, looks for a -script.py companion to actually invoke Python. It does not do anything else on its own. So AFAIK the only easy solution is to just hard code this logic in the discovery process to copy them both to the destination. Alternatively you can jus generate your own .exe a la pip that does not need the companion script file. pip uses distlib, and the API is pretty straightforward.

[uranusjr]

For reference, this is where setuptools creates the script/exe combo.

https://github.com/pypa/setuptools/blob/682b6511ac67e021b542e74ce30e13fe52bc2da9/setuptools/command/easy_install.py#L2063

[itsayellow]

I've been reading:
https://github.com/pypa/setuptools/blob/682b6511ac67e021b542e74ce30e13fe52bc2da9/setuptools/command/easy_install.py#L2243
https://setuptools.readthedocs.io/en/latest/history.html#id627
https://bitbucket.org/tarek/distribute/issues/143

I'm wondering if on Windows we should also copy over the .exe.manifest file into .local/bin to be safe? What do you think @uranusjr ?

[itsayellow]

I made another PR #341 that also adds the .exe.manifest file to PIPX_BIN_DIR if it exists in the venv bin_dir, just to be safe.

[uranusjr]

I believe the manifest file can have a positive effect to anti virus things, no harm to copy them.