Pipx: Installing local package fails

[rokroskar]

Describe the bug

pipx fails to install a local package.

How to reproduce

pipx install .
Cannot determine package name from spec '.'. Check package spec for errors.

pipx --version
0.15.1.3

Expected behavior

The package should be installed.

[cs01]

This is due to a quirk in pip's behavior. If you run pip list or pip freeze from a directory that is pip-installable, without actually installing the package, it will list it as installed. Maybe it's intentional, or maybe it's just because it gets put on Python's current path. In any case, this causes pipx to not be able to identify the package name. Fortunately, there is an easy workaround.

Install the package from any other directory. For example, try going up to the parent directory, then running

pipx install ./mypackage

[rokroskar]

Thanks for getting back to me @cs01 - I did try passing in an absolute path (though while in the directory of the package) and it failed in the same way. Backing up one directory works though, thanks for the tip!

I don't see my package listed though when I run pip freeze in its base directory - only when I run python -m pip freeze. Don't really understand why there's a difference, but there's that. :)

[rokroskar]

Curious - why doesn't pipx extract the package name from setup.py?

[cs01]

Since pip allows so many different installation types, getting package names is tricky, and implementations change over time. So instead of building parsing of setup.py and other files into pipx, pipx installs the package to a temporary virtual environment and compares the pip's package list before and after the installation. pipx abstracts away all the details of "how to install and list packages" to pip, since ultimately that's what does the installation anyway.

Because pip lists the package as installed before the installation, pipx does not detect any newly installed packages and hence cannot determine the package name.

[cs01]

If you run with the --verbose flag it might help make it more clear what's going on.

[cs01]

I don't see my package listed though when I run pip freeze in its base directory - only when I run python -m pip freeze. Don't really understand why there's a difference, but there's that. :)

pipx runs python -m pip freeze so that is probably why that's happening.

[rokroskar]

Sure I understand you don't want to build in all the logic and rely on pip to give you that info instead. Is there some way to make the hack you proposed automatic? Naively I expect the invocation of pipx to not be affected by the current working directory at all since the venvs are isolated anyway.

[cs01]

Is there some way to make the hack you proposed automatic?

Given how often this occurs (even to me), I agree it would be good to make this automatic. I am not sure if there is a pip API specifically for this. I suppose we could check if there is a setup.py, pyprojects.toml, etc. in the current working directory and print a warning.

[uranusjr]

If I understand the problem correctly, one possible hack is to use pip list --json --verbose and exclude the entry if location is the same as pwd.

[uranusjr]

Ah, checking location does not help, because pip would still “prefer” the package in cwd after pip install ., so the installed package is still not inspectable. I’ll try raising this issue to pip and see if it’s fixable there (I suspect yes, since it is “fixed” in pip list but not python -m pip list).

[itsayellow]

Another idea I had (which may or may not be a hack) would be if we notice '.' as the install directory, to change our cwd to one level up and then modify the path to specify the original directory by relative name.

[rokroskar]

I'm trying to understand how pipx does the install - if it makes a temporary directory for the venv, does it copy the cwd there? Why not just use absolute paths and avoid this problem altogether?

[cs01]

Why not just use absolute paths

Absolute paths won't fix it. It has to do with the current working directory. I think switching the cwd when running pip to be the venv root would fix this issue.

@rokroskar You can check out the source here https://github.com/pipxproject/pipx/blob/155cef9236aa0290578df7b46148f32c58bb5f6f/src/pipx/commands/commands.py#L85.

[rokroskar]

It has to do with the current working directory. I think switching the cwd when running pip to be the venv root would fix this issue.

ah ok, I see - I assumed you were doing that already - yes that certainly sounds like it could work!

[layday]

Another option is to invoke the Python interpreter in isolated mode to exclude the current working directory from the path.

https://github.com/pipxproject/pipx/blob/155cef9236aa0290578df7b46148f32c58bb5f6f/src/pipx/venv.py#L298

Replace "-m" with "-Im" above. This is not strictly equivalent to pip list because it also excludes _user_ packages, but pipx operates on the system level.

[itsayellow]

Isolated mode is very interesting, it has other good effects too.
https://docs.python.org/3/using/cmdline.html#id2

-I

Run Python in isolated mode. This also implies -E and -s. In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory. All PYTHON* environment variables are ignored, too. Further restrictions may be imposed to prevent the user from injecting malicious code.

New in version 3.4.

The -E option ignores all PYTHON* environment variables, which could allow us to stop cleaning them in util.py run commands possibly.

[layday]

(I never understood what those googly eyes mean.)

[itsayellow]

I take them to mean "interesting!" although that could just be me.

[cs01]

Agreed, kind of like "something that looks interesting and I will look into more".

[rokroskar]

@layday that seems like the simplest, least hacky suggestion!

[rokroskar]

I've tested this and it seems to fix this problem - @layday should I open a PR for the change?

[layday]

I don't know - I'm just a passer-by :)

[uranusjr]

The problem I have with -I is it has too many implications (ignoring all PYTHON environment variables). This might be a good idea anyway, but is a breaking change significant enough that needs to be discussed through (and probably a long testing period).

[rokroskar]

@uranusjr sure, it seems like a pretty significant flag - but to my naive eyes it looks like in this context, it would be used in a method that is only used to determine which package was installed (i.e. to get the package name). There are no other uses of list_installed_packages as far as I can tell.

[uranusjr]

Relevant: pypa/pip#2926

[pawamoy]

Any update on this :slightly_smiling_face:?

Isn't it a one character diff PR? Plus one or two tests of course.

[uranusjr]

The aforementioned pip issue will be fixed in the upcoming 20.1 (due today). We can revisit it soon. One way to help is to try this again with the new pip version and see if the issue resolves itself.

[rokroskar]

that's great news, will give it a go once the pip release is out - thanks @uranusjr !

[rokroskar]

@uranusjr just tested this now - pip-20.1 doesn't seem to fix the problem

[pawamoy]

@rokroskar it does for me. Just tried to install local package while in the package parent directory, and it worked.

[rokroskar]

Hmm strange...

❯ pip --version
pip 20.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
❯ pipx --version
0.15.1.3
❯ pipx install .
Cannot determine package name from spec '.'. Check package spec for errors.

[layday]

pipx doesn't use your system's pip - is the pipx version of pip, 20.1? pipx upgrade-all will upgrade the shared-library pip to the latest version.

[pawamoy]

Alternatively, just upgrade pipx's pip: ~/.local/pipx/shared/bin/pip install -U pip

[rokroskar]

Ah indeed - I forgot about the shared libraries, thanks. That does fix the problem!

[uranusjr]

So does this mean there’s nothing more pipx needs to do? The issue can be closed if that’s the case.

[rokroskar]

I wasn't sure if you want to keep it open until the updated pip is used by pipx by default.

[uranusjr]

The shared pip is updated automatically, see #396. The pip update will land in at most a month, but GitHub does not have a timer feature to auto close this when that happens. I’ll just close this.