Pip: Different pip install behavior depending on usage underscore/dot/dash in package name

[jt1]

• Pip version: 9.0.1
• Python version: 3.5.2
• Operating system: Ubuntu 16.04.3 LTS

Description:

Package name defined in setup.py is foo.bar. When running pip install --no-index --force-reinstall --use-wheel --upgrade foo.bar it works ok and package is uninstalled first. When running same command using package name foo_bar or foo-bar it seems to works but packages is not uninstalled first.

Why is it like that?
In my opinion upgrade behavior should be the same regardless of the used name of the package.

What I've run:

~/foo$ mkvirtualenv foo
Using base prefix '/usr'
New python executable in /home/qba/.virtualenvs/foo/bin/python3
Not overwriting existing python script /home/qba/.virtualenvs/foo/bin/python (you must use /home/qba/.virtualenvs/foo/bin/python3)
Installing setuptools, pip, wheel...done.
(foo) ~/foo$ tree
.
├── foo
│   ├── a.py
│   └── __init__.py
└── setup.py

1 directory, 3 files
(foo) ~/foo$ cat setup.py 
from distutils.core import setup

setup(
    name='foo.bar',
    version='0.1',
    packages=['foo',],
)
(foo) ~/foo$ pip wheel .
Processing /home/qba/foo
Building wheels for collected packages: foo.bar
  Running setup.py bdist_wheel for foo.bar ... done
  Stored in directory: /home/qba/foo
Successfully built foo.bar
(foo) ~/foo$ pip install --no-index --force-reinstall --use-wheel --upgrade --find-links=file:///home/qba/foo foo_bar
Collecting foo_bar
Installing collected packages: foo-bar
Successfully installed foo-bar
(foo) ~/foo$ pip install --no-index --force-reinstall --use-wheel --upgrade --find-links=file:///home/qba/foo foo_bar
Collecting foo_bar
Installing collected packages: foo-bar
Successfully installed foo-bar
(foo) ~/foo$ pip install --no-index --force-reinstall --use-wheel --upgrade --find-links=file:///home/qba/foo foo.bar
Collecting foo.bar
Installing collected packages: foo.bar
  Found existing installation: foo.bar 0.1
    Uninstalling foo.bar-0.1:
      Successfully uninstalled foo.bar-0.1
Successfully installed foo.bar-0.1
(foo) ~/foo$ pip install --no-index --force-reinstall --use-wheel --upgrade --find-links=file:///home/qba/foo foo_bar
Collecting foo_bar
Installing collected packages: foo-bar
Successfully installed foo-bar
(foo) ~/foo$ pip install --no-index --force-reinstall --use-wheel --upgrade --find-links=file:///home/qba/foo foo-bar
Collecting foo-bar
Installing collected packages: foo-bar
Successfully installed foo-bar

[pfmoore]

What is your actual use case here? It seems like you're simply exploring corner cases - is there a particular real world situation that this behaviour is affecting?

[jt1]

If you use alternative package name, pip will not crash but old package files will not be removed from virtualenv which can cause app crash after deploy (my case) :/

I agree that right package name should be always used, but if other names are allowed they should work in the same way.

[pfmoore]

I'd argue that --force-reinstall --upgrade and the fact that you're removing files from the package without changing the version is where things are going wrong here. It sounds like you should probably be using editable installs while you're developing your package. If you're not keen on editable installs, pip uninstall followed by pip install is a cleaner way of managing a package that's changing without changing the version number.

I'm not disagreeing that the inconsistencies are wrong, but I don't see them as a high priority issue unless they are being triggered by a more usual development workflow than the examples you're describing.

[jt1]

Right, but I am afraid similar inconsistencies are being triggered for more usual cases like pip install:

(foo) [18:03:16]~/foo$ pip install fluent-logger 
Collecting fluent-logger
  Downloading fluent_logger-0.9.1-py2.py3-none-any.whl
Collecting msgpack (from fluent-logger)
  Downloading msgpack-0.5.4-cp35-cp35m-manylinux1_x86_64.whl (314kB)
    100% |████████████████████████████████| 317kB 2.2MB/s 
Installing collected packages: msgpack, fluent-logger
Successfully installed fluent-logger-0.9.1 msgpack-0.5.4

(foo) [18:06:05]~/foo$ pip install fluent-logger 
Requirement already satisfied: fluent-logger in /home/qba/.virtualenvs/foo/lib/python3.5/site-packages
Requirement already satisfied: msgpack in /home/qba/.virtualenvs/foo/lib/python3.5/site-packages (from fluent-logger)

(foo) [18:06:09]~/foo$ pip install fluent.logger 
Collecting fluent.logger
  Using cached fluent_logger-0.9.1-py2.py3-none-any.whl
Requirement already satisfied: msgpack in /home/qba/.virtualenvs/foo/lib/python3.5/site-packages (from fluent.logger)
Installing collected packages: fluent.logger
Successfully installed fluent.logger

(foo) [18:06:13]~/foo$ pip install fluent_logger 
Requirement already satisfied: fluent_logger in /home/qba/.virtualenvs/foo/lib/python3.5/site-packages
Requirement already satisfied: msgpack in /home/qba/.virtualenvs/foo/lib/python3.5/site-packages (from fluent_logger)

[pfmoore]

OK, so (ignoring the parts that if I follow your example, are unnecessary):

$ pip install fluent-logger
$ pip install fluent.logger

triggers a second install, without a proper uninstall/upgrade. That does indeed sound like a bug.

[StephanErb]

I have filed https://github.com/pypa/pkg_resources/issues/13 for this, but I guess it will require some serious discussion. pkg_resources might be the wrong place to fix this.

[pradyunsg]

I feel InstallRequirement.name should switch to use packaging.utils.canonicalize_name instead.

/cc @pypa/pip-committers for thoughts.

[pfmoore]

I'll have a think about this once 10.0.0 is out of the door. Feel free to ping me then if I forget :-)

[pradyunsg]

Ping @pfmoore. :)

[pfmoore]

Sorry, I've been hugely busy on RL issues, so haven't had any time to think about this (and likely won't in the near future).

If your suggestion fixes

$ pip install fluent-logger
$ pip install fluent.logger

so that it doesn't do a second install, that's OK with me. I guess there may be other knock-on effects (would the output of pip freeze be affected?) but I don't know precisely what they'd be or how serious they might be without implementing it to find out.

[pradyunsg]

Sorry, I've been hugely busy on RL issues, so haven't had any time to think about this (and likely won't in the near future).

No issue. :)

[chrahunt]

I can confirm this is still an issue in 19.2.1.

[deveshks]

Hi @chrahunt , @pradyunsg , @pfmoore

Could I work on this issue?

The issue still exists in the pip 20.1.dev0

$ pip --version
pip 20.1.dev0 from /Users/devesh/pip/src/pip (python 3.8)

$ pip install fluent-logger
Collecting fluent-logger
  Using cached fluent_logger-0.9.6-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: msgpack<1.0.0 in ./.env/lib/python3.8/site-packages (from fluent-logger) (0.6.2)
Installing collected packages: fluent-logger
Successfully installed fluent-logger-0.9.6

$ pip install fluent.logger
Collecting fluent.logger
  Using cached fluent_logger-0.9.6-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: msgpack<1.0.0 in ./.env/lib/python3.8/site-packages (from fluent.logger) (0.6.2)
Installing collected packages: fluent.logger
Successfully installed fluent.logger

Also if I understand correctly, the fix will involve adding a line after https://github.com/pypa/pip/blob/master/src/pip/_internal/req/req_install.py#L428 to using packaging.utlis.canonicalize_name as indicated in https://github.com/pypa/pip/issues/5021#issuecomment-377768403

no_marker.name = canonicalize_name(no_marker.name)

I also ran the pip install twice and the second one fails:

$ pip install fluent.logger
Requirement already satisfied: fluent.logger in ./.env/lib/python3.8/site-packages (0.9.6)
Requirement already satisfied: msgpack<1.0.0 in ./.env/lib/python3.8/site-packages (from fluent.logger) (0.6.2)

I also checked the output of pip freeze as indicated in https://github.com/pypa/pip/issues/5021#issuecomment-388122953 and it contains fluent-logger==0.9.6 which is the correct name with a - for the package. What are other side-effects I should be aware of and verify when implementing this?

[uranusjr]

I’m likely missing something, but why is the first one containing a version part, but not the second one? I feel this would be the realy problem here; the dot-dash difference doesn’t really bother me (i.e. the package name does not need to be canonicalised), but the different between the output format does, since it indicates there’s some parsing logic wrong.

[deveshks]

Hi @uranusjr

This is because at https://github.com/pypa/pip/blob/e40d2670facb6cfc815f948c05ca73abb3200f50/src/pip/_internal/utils/misc.py#L637
req is fluent-logger when we run pip install fluent-logger, and its fluent.logger, when we run pip install fluent.logger.

Now since the working set only has an entry for key fluent-logger, the dist is None when we do the second one , and so the installed_version is not added to the package name at https://github.com/pypa/pip/blob/e40d2670facb6cfc815f948c05ca73abb3200f50/src/pip/_internal/commands/install.py#L424-L428

To resolve this, we can canonicalise the name of the package when doing working_set.find(req), or fix it before when we canonicalise the package name while looking for it

[uranusjr]

I see, thanks for the explaination. The fix you proposed sounds like the correct solution to me. Let’s get this fixed!

Side: I was wondering why the message uses such a unique format, rather than more robust formats like fluent-logger (0.9.6) or fluent-logger==0.9.6. And it turns out this was introduced way back in #2221. I wonder if it’s possible to change the output, since using dash here would result in some ambiguity (also why the wheel uses underscore instead). The packaging scene has also changed quite a lot since then; I’d say the == format has since been taken for granted.

cc @pfmoore and @dstufft since you were involved at the time.

[deveshks]

I see, thanks for the explaination. The fix you proposed sounds like the correct solution to me. Let’s get this fixed!

Hi @uranusjr , I assume you meant using
no_marker.name = canonicalize_name(no_marker.name) to correct the package name and correctly look up if it is already installed or not. I will create a PR for the same