Pip: Behavior change between 8.1.2 and 9.0.1 of pip breaks --extra-index-url in requirements files

[sc68cal]

• Pip version: 9.0.1
• Python version: Python 2.7.13
• Operating system: Debian Stretch

Description:

When moving from Debian Jesse to Stretch, the newer version of pip gets picked up, and ends up breaking our docker build. Our requirements.txt file contains the following, and worked successfully when using pip version 8.1.2

grpcio

--extra-index-url <SANITIZED>
wrigley==0.0.4
junos-eznc
python-consul
retry

Moving to 9.0.1 results in the following error

Collecting wrigley==0.0.4 (from -r requirements.txt (line 4))
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 342, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 404 Client Error: Not Found for url: https://pypi.python.org/simple/wrigley/

[sc68cal]

We are currently using this workaround

```diff
diff --git a/Dockerfile b/Dockerfile
index badcf1c..359ec28 100644
--- a/Dockerfile
+++ b/Dockerfile
+RUN pip install --extra-index-url wrigley==0.0.4
diff --git a/requirements.txt b/requirements.txt
index 6a3c94d..649c90a 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,7 @@
grpcio
-
---extra-index-url
+## extra-index-url is commented out - and have been moved to a line in the Dockerfile
+# --extra-index-url
wrigley==0.0.4
junos-eznc
python-consul

[zaufi]

It looks like the issue I've faced (#4605), but I use extra-index-url in /etc/pip.conf

[pradyunsg]

Indeed. I'm closing #4605 in favour of this issue.

[SiegristJ]

sc68cal: By any chance are you using the PyPI URL 'https://pypi.python.org/pypi' instead of 'https://pypi.python.org/simple'. Switching to simple resolved the --extra-index-url for me with pip 9.0.1 on CentOS 7.

Yesterday, a link pointing to this helped resolve the problem: https://www.python.org/dev/peps/pep-0503/

[sc68cal]

@SiegristJ - you can see the contents of our requirements.txt file - and our Dockerfile simply has pip install -r requirements.txt - we are not setting the PyPI URL

[mishas]

Some more information about the bug:

If you'll look at the original exception that @sc68cal posted, you can see that it use version 2.12.4 of the requests package (/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl).

It seems to be the problem. updating to version 2.17 (or higher) seems to fix that problem.

It seems like the bug is actually in the requests package, and is already fixed, BUT:
Installing pip using apt-get install python-pip on ubuntu and debian is very common, and it brings in the wrong requests dependency. Even upgrading requests (using pip install --upgrade requests) doesn't work, as pip installed using apt-get only uses requests that comes with it.

I think, that even though, as stated above, the bug is not in pip, but in requests, it is worth patching it here as well for the benefit of everyone using pip from apt-get.

For now, a work-around of installing pip using easy_install -U pip works.

[pfmoore]

If this is fixed in a newer version of requests, we'll pick that up when we re-vendor that version. I note that at the moment we are vendoring requests 2.14.2 in master, but 2.18.4 is released on PyPI, so we need to revendor before the next pip release. We don't use the system-installed version of requests, so upgrading that won't have any effect (by design).

If Ubuntu/Debian packaging complicates matters beyond this (I don't quite understand what you're saying on that point - it seems like you expect their version to have unbundled requests and so for upgrading requests to work for you) then you'll have to take that up with them.

[mishas]

I think re-vendoring requests with the next version of pip should work. I didn't know if you or debian/ubuntu are in charge of that - and from your answer it seems like it's you.

That should fix it.

[pfmoore]

The pip devs do the revendor, but the packaging that Debian/Ubuntu do to make the apt package isn't anything to do with us (and I think they unbundle our vendored packages, so the revendoring might not matter in that case), so I can't comment on when apt-get will be updated.

[dstufft]

Debian/Ubuntu debundle our vendored requests, and then more or less re-vendor the version of requests that ships with Debian/Ubuntu. To fix python-pip their version of requests will need to be fixed and then python-pip will need rebuilt.

[badicsalex]

I've ran into this issue, and as I've seen it's because the exception type in index.py:HTMLPage.get_page is simply wrong. It is named HTTPError, yes, but "pip._vendor.requests.exceptions.HTTPError" error is thrown, but "requests.exceptions.HTTPError" is catched, and they are of course different.
Rewriting the code to import HTTPError with SSLError, and writing simply HTTPError in the except, instead of requests.HTTPError solves the issue.

I don't know if this is useful information for you, but this same bug was reported on Ubuntu too: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1579196

Or at least I think this is the same bug as originally reported in this thread, otherwise I'm sorry, and will open a different one.

(note: I think something is horribly broken around vendorization/wheels that actually causes this issue)

[cameel]

Hey guys, has this ever been fixed? I would assume so from the comments but this issue is still open and surprisingly I'm observing the same behavior between 0.18.0 and 0.18.1. On 0.18.0 pip correctly ignores the HTTP 404 for packages that don't exist in the extra registry and finds them on PyPI. On 0.18.1 it fails as soon as it sees the error, no matter whether the package exists on PyPI or not.

This happens only when using --no-cache-dir flag.

The environment is similar (Debian Stretch, 8.4) but I'm using Python 3.6.8 from testing.

[pradyunsg]

I think you mean 18.0 and 18.1 version numbers?

[cameel]

Oh, yeah. Thanks for the correction. 18.0 and 18.1 of course. I've updated the comment.

[artdgn]

Seeing same problem during docker build (
Exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main status = self.run(options, args) File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run session=session, autobuilding=True File "/usr/local/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build assert building_is_possible AssertionError The command '/bin/sh -c pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt' returned a non-zero code: 2
Both removing --no-cache-dir or forcing pip==18.0 solved the issue.

[xavfernandez]

@artdgn your comment is related to #6158 not this one.