Pip: Package names containing a dot don't work under DevPI

[grgsrs]

• Pip version: 8.1.2
• Python version: 2.7.11
• Operating System: Linux
  

  Description:

When asked for a package name containing a dot version 8.1.2 of pip rewrites the package name with a dash instead (e.g. "foo.bar" becomes "foo-bar"). This seems to work fine on PyPI, but not on DevPI (we're using devpi-server-2.1.4). Downgrading to version 8.1.1 of pip resolves the problem.

What I've run:

$ virtualenv ~/v27
New python executable in /home/myname/v27/bin/python
Please make sure you remove any previous custom paths from your /home/myname/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
~
$ ~/v27/bin/pip list
pip (8.1.2)
setuptools (21.0.0)
wheel (0.29.0)
~
$ ~/v27/bin/pip -v -v -v install zope.interface
Collecting zope.interface
  1 location(s) to search for versions of zope.interface:
  * http://devpi.our.domain.com/our_company/prod/+simple/zope-interface/
  Getting page http://devpi.our.domain.com/our_company/prod/+simple/zope-interface/
  Starting new HTTP connection (1): webproxy.our.domain.com
  "GET http://devpi.our.domain.com/our_company/prod/+simple/zope-interface/ HTTP/1.1" 200 139
  Analyzing links from page http://devpi.our.domain.com/our_company/prod/+simple/zope-interface/
  Could not find a version that satisfies the requirement zope.interface (from versions: )
Cleaning up...
No matching distribution found for zope.interface
Exception information:
Traceback (most recent call last):
  File "/home/myname/v27/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/myname/v27/lib/python2.7/site-packages/pip/commands/install.py", line 310, in run
    wb.build(autobuilding=True)
  File "/home/myname/v27/lib/python2.7/site-packages/pip/wheel.py", line 750, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/myname/v27/lib/python2.7/site-packages/pip/req/req_set.py", line 370, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/myname/v27/lib/python2.7/site-packages/pip/req/req_set.py", line 522, in _prepare_file
    finder, self.upgrade, require_hashes)
  File "/home/myname/v27/lib/python2.7/site-packages/pip/req/req_install.py", line 268, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/myname/v27/lib/python2.7/site-packages/pip/index.py", line 491, in find_requirement
    'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for zope.interface

[grgsrs]

I tried setting up a local installation of devpi-server 3.1.0 and the same problem still occurs.

[fzlee]

I met this issue as well, I checked the source code, every time pip finding packages, the package name is passed to a function called canonicalize_name.

That function is located at /pip/_vendor/packaging/utils.py, and it will replace dot with dash in package names.

Regarding to zope.interface, seems both versions (zope.interface and zope-interface) are available in https://pypi.python.org/simple/, that's why we can download zope.interface via pypi.python.org.

I doubt if converting dot to dash is intentional. anyway , a change might need on either pip or devpi-server

[neilvana]

We've seen it too. I hope this gets fixed soon.

[chadawagner]

Same here with python 3.4

[rberrelleza]

I'm seeing the same issue with libraries hosted on artifactory, so this goes beyond devpi and pypi.

[dstufft]

The tl;dr is that these tools will need to be updated to match the normalization scheme as defined in PEP 503. This can be implemented using the packaging library's canonicalize_name function.

The longer explanation is that originally the simple API was not well defined but PyPI had all of the URLs using the form of the name that the author originally entered (e.g. Django not django) and the various tools (pip / easy_install) would make a request using exactly what the use typed (e.g. pip install django requested it as 'django' not Django). This generally worked fine on PyPI because PyPI would redirect anything that matched what became the PEP 503 normalization rules to the correct thing. Obviously a static mirror like bandersnatch couldn't do this redirect so if pip/easy_install couldn't find a name it would fall back to requesting the top level API that lists every single package on PyPI.

This worked but it meant that people were regularly downloading a several mb file from Mirrors because they happened to use a slightly different but equivalent version of the name. It also caused extra http requests even in the PyPI case to redirect people to the correct place.

To remedy this I wrote PEP 503 to define the simple API and instead of making the author provided name the cannonical name I made the normalized name the cannonical name which meant that we could do the normalization everywhere without having to worry about redirects (except for old versions of the tools) and without the nasty several mb fallback. This was implemented in pip with version 8.

However due to an oversight the '.' Was not correctly getting normalized by pip (and a lot of other tools) but because a lot of them mis-implemented then similarly it worked. However with pip 8.1.2 we updated one of our dependencies which correctly implemented this and that caused pip to start correctly normalizing the '.' but until these tools like devpi get updated things aren't going to quite work.

As a transitional thing tools should probably support both the erroneously normalized and the correctly normalized form.

Sent from my iPhone

On May 10, 2016, at 11:00 PM, Gregory Saunders [email protected] wrote:

When asked for a package name containing a dot version 8.1.2 of pip rewrites the package name with a dash instead (e.g. "foo.bar" becomes "foo-bar"). This seems to work fine on PyPI, but not on DevPI (we're using devpi-server-2.1.4). Downgrading to version 8.1.1 of pip resolves the problem.

[mrmachine]

This is pretty bad. Every single installation of devpi and similar tools will now need to be upgraded immediately. Who knows, some projects might not be able to use the latest (fixed) version of such a tool for other reasons, and the fix likely won't be backported to every old version of those tools. Perhaps some kind of deprecation timeline would have been possible, to give the authors of those tools time to at least create a fixed version?

[mrmachine]

For anyone using tox on Travis CI, which creates a virtualenv and installs the latest pip (even though Travis CI environment itself uses pip 6.0.7, here's how we worked around this problem.

Add a pip.sh script to your project:

#!/bin/bash
pip install '<8.1.2'
pip install "$@"

Add to your tox.ini:

install_command = {toxinidir}/pip.sh {opts} {packages}

This will downgrade the version of pip inside the virtualenv created by tox, before installing the dependencies.

[pfmoore]

Well, PEP 503 was published 8 months ago, so tools like devpi have had some time to implement it. It's a shame they've been slow in doing so, but I'm not sure what we can do about that (other than raising an issue on devpi).

[fschulze]

Still running tests, but it looks like a devpi-common 2.0.9 release will be enough to fix this.

[RonnyPfannschmidt]

@pfmoore well, imho that bugfix should have been a major release (from my pov its a breaking change of interaction with external systems)

[MarkusH]

I bisected this problem to 8e236dd6a09bd2f70f9d4fc886da8c354d4c58f2

[pfmoore]

@RonnyPfannschmidt Major release for what? I believe from what @dstufft said the support was added in pip in the version 8 (a major release) - there was a bug around the dot character in 8.0 itself, that bugfix was in a minor release but that seems perfectly fine to me. Or do you mean a devpi major release?

[RonnyPfannschmidt]

@pfmoore im aware that the code change is small, but the behavior change is breaking

as such my opinion is, that a major release might have been better so people notice that the update might break something

as for devpi, i don't know if the change on the server side is breaking or just cosmetic
basally if it breaks users, major, if it doesn't patch

[pfmoore]

@RonnyPfannschmidt Well, the behaviour change was in 8. It just didn't _work_ properly. Would you have been happier if pip 8 had rewritten foo.bar to foo-bar and we'd had this issue then? I get what you're saying, we should have called pip 8.1.2 version 9.0.0. But that's a hard call - how were we to know that devpi (and other tools) hadn't already implemeted PEP 503? How were we to know that canonicalising dots was a bigger deal than (say) normalising case? Honestly, though, I don't think it's worth worrying about now, it's history.

Users who were _really_ carefully following the feature progression would have seen that PEP 503 was in pip 8, and that devpi hadn't implemented the server side of it, and pinned to pip < 8.0 until they could get a devpi with PEP 503 support. It's hardly surprising if users missed this but that's the "correct" process.

[dstufft]

Whether or not this change should have triggered a major or minor or path release would be a more interesting question if anybody had noticed that pip 8 didn't correctly normalize the . to - and if anybody had noticed that when upgrading one of the dependencies in pip that _did_ correctly normalize the . to a - that pip had now started to do it correctly.

[dstufft]

IOW, nobody made a conscious choice to change this behavior in 8.1.2, it was a consequence of the fact that I expected any changes like this to have already happened, so I didn't look real close and our dependencies updating to PEP 503 behavior.

[RonnyPfannschmidt]

@pfmoore my understanding is that the intent was to change behavior in pip 8 and the fix life-cycle accidentally deferred that to 8.1.2

i'd prefer to only see that kind of change for major version jumps (as the intent was)

i'm well aware the current problem is unintended and i'm pretty sure i'd have done the release just as @dstufft did

this is more of a "ouch this is an accident, it should be major due to breakage we observe"

[mrmachine]

@pfmoore semantic correctness doesn't always trump pragmatic real world concerns. Unfortunately, I am not at all surprised that a tonne of projects that interact with pypi had no idea that pip 8 introduced this change or that there was a bug in it. Widely used projects do often consider the actual real world impact on fixing "bugs" that result in backwards incompatible breakage for 3rd party code. I'm not sure if it would have been possible, but perhaps making this a soft change in 8.0 and monitoring the tools to find problems like this before enabling it could have prevented unexpected breakage.

[dstufft]

To be clear it was warning since pip 6.0 in the general case, it was an oversight that it wasn't correctly converting a . to a - (but it was converting other characters) and pip 8.0 was when that behavior was actually converted from a warning to an error (again, still with the oversight that . wasn't getting converted to -) and it wasn't until 8.1.2 that we happened to inadvertently correct that oversight because one of the libraries we use did it correctly.

Unfortunately we don't have any metrics into how people are invoking pip, what the outcomes of their commands are, or anything like that which we could use to judge impact (if we had even realized this wasn't working correctly in the first place or that we had inadvertently fixed it) so impact tends to be judged based on gut feeling mostly and testing against PyPI itself. Prior to PEP 503 the definition of what a repository was, was literally "whatever PyPI does" so all the non PyPI implementations attempted to copy that to varying degrees of success and typically they didn't re-evaluate that when PyPI changed until pip install <thing> actually breaks for them.

Typically I would roll this back and issue a 8.1.3 and send this specific change through some kind of deprecation process... _however_, the dependency we updated and the changes that caused it to actually happen are fixing other important bugs caused by the new marker support. Rolling that back will re-break some packages that are currently not able to be installed by pip 8 -> 8.1.1. It ends up being a choice between breaking those packages, or breaking things like devpi and bandersnatch, and I have to fall along the axis of fixing the things we meant to fix in 8.1.2 and letting the breaking change stay, largely because those packages don't have much of a recourse for fixing it otherwise and this breaking change can be fixed in devpi/bandersnatch/etc instead.

[dstufft]

See also, Postel's Tarpit.

[mrmachine]

@dstufft thanks for the additional explanation. That all makes sense and sounds like the right thing has been done in the end, it was just unfortunate that tools like devpi had not been updated already. I'm not developing those tools, I'm only using them, so your explanation adds some clarity to the situation for me personally.

[fschulze]

devpi fix is out, see https://groups.google.com/d/msg/devpi-dev/-MgysUfZ36Q/JnqqMscrAAAJ

[dstufft]

@fschulze thanks for the quick fix :)

[cjw296]

@dstufft - sadly, no one except the few dedicated souls like your good self actually read, let alone understand PEPs (for this one you'd have had to be on the pypa lists (distutils-sig, still?) or maybe python-dev? to even find out about the PEP - is that a realistic expectation for everyone who runs any python package hosting infrastructure?).

The rest of us react to stuff changing as we find out about it, most often when things break.

What makes me sad is that this change means you can no longer host a poor man's python package repository using a filesystem and apache, unless you can guarantee no packages include dots in their names...

[cjw296]

@fschulze - any chance of a point release fix for devpi-server 2.x too?

[fschulze]

@cjw296 you can, by either depending on pip >= 8.1.2 or adding a rewrite rule to your webserver. Store the files with dashes and rewrite anything with [_.]+ to _.

And if you had read the link, you would have seen that devpi-common 2.0.9 fixes the issue for server 2.6.x as well.

[dstufft]

@cjw296 I expect that the people who are writing things like devpi and such will be on distutils-sig and to read and understand PEPs related to the thing they're working on yes. I believe that at least some of them are though it's completely reasonable to have missed the nuance in the normalization procedure since it's really a slight difference in a regex.

That being said, you can still trivially host a "poor man's repository" in one of two ways:

• When you create the directory layout on disk, ensure that you do the normalization as described in PEP 503 (it's written as a Python function for you already even!) or using packaging.utils.canonicalize_name and stick the files inside of that.
• Stick all the files in one big directory and use -f instead of -i.

[fschulze]

And I'm sure only developers have been affected by this pip change anyway, because in production everyone pins all their packages including pip, right? (SCNR)

[tobias-9]

Firstly, having NEVER experienced a warning in regard to "." in package names, this effective pulling the plug is pretty bad.

Secondly, while PyPI still has packages using the "." pip should allow for it. Goto

https://pypi.python.org/pypi/zope.interface

vs

https://pypi.python.org/pypi/zope-interface

Notice where you redirect in the 2nd case.

I must say I believe PEP 503's forcing of "-" to be the effective "separator" in package names to be a cause of confusion, if not simply wrong. Reasons:

• "-" is used to separate "major" parts of the package file - eg, name, version, py versions; notice the "." separators in the versioning scheme after all!
• The "." in package names is also better aligned with how the package will actually be used:
  from zope.interface import registry
• PEP 426 allows for "." (and "-" and "_"); let it be the author's choice if it's accepted character (same for lowercasing)

[cjw296]

@dstufft - fair enough. I must echo @tobias-9's comment though: I've never seen any warning from pip relating to dots in package names... where would I look?

[fschulze]

The normalization helped make the code in devpi a lot cleaner and simpler and was the reason the actual fix was now so simple, the only part that took long, was testing.

[cjw296]

@fschulze - hard to pin pip sometimes, when it's pip that installs everything else ;-)

[dstufft]

To better word my last response, I don't mean to suggest that the devpi developers are _not_ reading or understanding the relevant PEPs, just that they're the group of people that I expect to do that, not any random person :)

[dstufft]

@tobias-9

You're still allowed to use a . in a package name, the change is purely one of the API that pip uses to talk to a repository server as far as what URL pip is going to request from that server. In packages the -, _, and . characters are considered equivalent and we normalize runs of those characters to a single - (so -_-_-_ is normalized to -). This normalization makes it _easier_ to implement a repository server, since (when everything supports it), there is only one canonical representation in the API for what URL you need to respond at with the relevant data no matter if the author used . or _ or -.

The breakage is simply fixing a bug in the normalization procedure where . wasn't correctly being normalized and it just so happens that the bug existed in pip _and_ some of the non-PyPI repository implementations (I suspect if that wasn't true, it would have been noticed earlier) and both sides of the equation need to agree on the normalization rules for it to actually work. You'll see that devpi is now functioning correctly since it and pip now agree again on what the normalization rules are with no loss of functionality (other than a blip of downtime here).

As far as warnings go, the warning was for packages that were not installable using the normalized form. As I said, it was intended that . be normalized too which would have caused a warning for say, zope.interface, HOWEVER, due to the bug that was inadvertently fixed in 8.1.2 that warning wasn't being triggered for things that used a . (as opposed to the other things we normalize).

[tobias-9]

@dstufft Ok, I guess that is part of my confusion then, thanks for clearing up.

Still I must say normalizing to "-" instead of "." is confusing based on the same reason of having to Python split the string "zope.interface-4.1.3-py3.3-win-amd64.egg" vs "zope-interface-4.1.3-py3.3-win-amd64.egg". Just looking at the strings it's much easier to distinguish the first vs the second. The second would make me wonder whether "interface" has some special meaning in the packaging world.

Regardless, thanks for the great work! (despite this bit of a headache), Also thanks to @fschulze for the quick devpi "fix"!

[dstufft]

To be totally clear, this normalization only applies to the simple API, like: /simple/zope-interface/ and it doesn't apply to the file itself. The reasoning for normalizing to - instead of _ or . is largely historical, the rules were adapted from what setuptools was doing and made a bit more... aggressive in ensuring that there was only one canonical form of any one project name but otherwise we left them alone. Historically setuptools normalized to - and so we just kept that since it didn't really matter that much.

For file names, we tend to _not_ normalize the filename itself, other than in the newer formats we "escape" - by converting it to _ for the reason you mentioned above.

[tobias-9]

@dstufft Got it!

[pfmoore]

Specifically the wheel filename format relies on the project name never having a - in it (for the parsing reason @tobias-9 noted. IMO, it's a mistake that we use different canonicalisation rules in different places (e.g. for the project name in the simple interface and the project name in the wheel format), but it's not one that's likely to get changed now (and honestly, it's not one that will affect any but a small minority of people).

[dstufft]

@pfmoore I mean, we don't really canonicalize the name inside of a filename at all, we just escape it for characters that aren't safe or valid. Tools like pip internally canonicalize that name though when we compare it for the name of the package we're looking for.

[dstufft]

I'm going to close this issue since DevPI is fixed and there's nothing actionable here on pip's end. Thanks all!

[hpk42]

FWIW we are actually working on a fix for the fix released today now, things got more complicated. We'll probably need to do a full devpi-server-4.0 i.e. require an export/import cycle as things stand.

[hpk42]

also, it seems that pypi.python.org/simple/X has changed behaviour -- if X is "zope.interface" it says "Links for zope.interface" and if X is "zope-interface" then "Links for zope-interface". At some point in the past pypi would instead redirect to the canonical normalized name. Is this intentional?

[dstufft]

I think the redirect is just finnicky because PyPI is old and busted.

[farvour]

It would appear Gemfury is still not PEP-503 compatible and is having trouble with dot package names. We have rolled back to 8.1.1 to address on our end, and I have submitted a ticket to Gemfury as well about it.

[lisimia]

Hey guys, just ran into this, should/can we update the release notes to be very explicit about this breaking change?

[dstufft]

We obviously can't replace the release notes in the already released artifact, but if you want to submit a PR to master adding the changelog item we can update them for the website and the next release.

[etc0de]

This seems to be breaking pypiserver ( https://pypi.python.org/pypi/pypiserver ) as well... even the latest version. Might wanna check back with the devs to get it working for everyone again

[pavelrad]

Faced this problem with pip 8.1.2 and my company's devpi repo (devpi-server-2.1.5.post3
devpi-web-2.2.3).
Here is the cleaned terminal output of pip configured to use our local index:

$ pip install pip==8.*
Successfully installed pip-8.1.2
$ pip install backports.shutil_get_terminal_size
Collecting backports.shutil_get_terminal_size
  Could not find a version that satisfies the requirement backports.shutil_get_terminal_size (from versions: )
No matching distribution found for backports.shutil_get_terminal_size
$ pip install pip==7.*
Successfully installed pip-7.1.2
$ pip install backports.shutil_get_terminal_size
Collecting backports.shutil-get-terminal-size
  Downloading https://xxxxxxxx/root/pypi/+f/5ca/283ed87cdba75/backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl
Installing collected packages: backports.shutil-get-terminal-size
Successfully installed backports.shutil-get-terminal-size-1.0.0

Looks like it's a problem with pip 8., 'cause with 7. everything is ok.

[pavelrad]

pip 8.1.1 doesn't have this problem.

[chadawagner]

see http://doc.devpi.net/latest/announce/server-4.0.html

[pavelrad]

@chadawagner thank you!