Pip: Credential prompt after --extra-index-url redirects not allowed by requests

[sergei-maertens]

Environment

• pip version: 19.0
• Python version: 3.6
• OS: Debian 8.11

Description

We have a private PyPI running, which has basic auth protection. The CI user has a ~/.config/pip/pip.conf file containing the --extra-index-url option with username & password in the URL for basic auth.

Since pip 19.0, there is a prompt for the username and password for this index URL, which wasn't the case with 18.1. In a CI context where we cannot provide input, this is of course not convenient.

Expected behavior
PIP/requests should use the credentials found in the URL in the pip.conf and not prompt for them.

How to Reproduce

1. Have a private PyPI
2. Configure this with basic auth username/password in the URL in pip.conf
3. pip install some-package - where some-package lives in the private PyPI

Output, from our CI

$ pip install -r requirements/jenkins.txt
User for <redacted>: Exception:
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 269, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 196, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 639, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 610, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 743, in _get_pages
    page = _get_html_page(location, session=self.session)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 229, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/index.py", line 177, in _get_html_response
    "Cache-Control": "max-age=0",
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/download.py", line 403, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 653, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/var/lib/jenkins/workspace/ispnext-backend-develop/backend/env/lib/python3.6/site-packages/pip/_internal/download.py", line 197, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)

[dstufft]

Unverified, but tagging as a 19.0 issue as this seems like a major regression.

[sergei-maertens]

Thank you. Just confirmed it on my local machine as well (similar setup), which is

Python 3.6 in virtualenv
pip 19.0 (just updated with pip install pip -U)
OS: Arch Linux

I can pin everything to 18.1, but smells like a major regression indeed :)

[sergei-maertens]

Building my Docker image now, this also happens when I run:

pip install \
    --extra-index-url=${PRIVATE_PYPI} \
    -r requirements/production.txt

so with the explicit CLI option instead of using pip.conf

[cjerdonek]

Some ideas:

@sergei-maertens, does your URL perform any redirects?

If you add -vv to your pip install invocation, do you get any more information? Can you share the additional logs?

Lastly, how does the traceback compare if you use 18.1 and change the password so it's wrong?

[sergei-maertens]

@cjerdonek there was a redirect, indeed! tthat seems to have been it!

In my pip.conf I had https://user:[email protected], after changing that to https://user:[email protected]/simple, the prompt is no longer there.

So, a combination of misconfiguration and regression in robustness of how to handle that? :-)

Log output, for completeness:

~ pip install pip==18.1 -vv
Created temporary directory: /tmp/pip-ephem-wheel-cache-8v35yud4
Created temporary directory: /tmp/pip-req-tracker-kxd48baw
Created requirements tracker '/tmp/pip-req-tracker-kxd48baw'
Created temporary directory: /tmp/pip-install-aifmfhg0
Looking in indexes: https://pypi.org/simple, https://[redacted]:****@pypi.[redacted].net
Collecting pip==18.1
  2 location(s) to search for versions of pip:
  * https://pypi.org/simple/pip/
  * https://[redacted]:****@pypi.[redacted].net/pip/
  Getting page https://pypi.org/simple/pip/
  Looking up "https://pypi.org/simple/pip/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/pip/
    ...
    [truncated]
    ...
  Getting page https://[redacted]:[redacted]@pypi.[redacted].net/pip/
  Looking up "https://pypi.[redacted].net/pip/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.[redacted].net:443
  https://pypi.[redacted].net:443 "GET /pip/ HTTP/1.1" 302 0
  Status code 302 not in (200, 203, 300, 301)
  Starting new HTTP connection (1): pypi.[redacted].net:80
  http://pypi.[redacted].net:80 "GET /simple/pip/ HTTP/1.1" 301 184
  Looking up "https://pypi.[redacted].net/simple/pip/" in the cache
  Request header has "max_age" as 0, cache bypassed
  https://pypi.[redacted].net:443 "GET /simple/pip/ HTTP/1.1" 401 194
User for pypi.[redacted].net: 

[cjerdonek]

Great! 👍

I asked about redirects because pip's vendored requests library was upgraded since the last release, and requests's release notes mention a couple recent changes about how requests will now strip the authorization info in the presence of certain redirects like https -> http (which will show up as not supplying a username / pass -- hence a prompt). And indeed, that's what it looks like is happening from your logs.

[cjerdonek]

@dstufft Do you think anything is actionable here in the short or long term?

[sergei-maertens]

This actually made me spot another possible issue vaguely related, the redirect to http instead of https shouldn't happen, but this is probably because of SSL-termination at the nginx reverse-proxy. Not relevant for the progress of this issue, but might help others.

[pradyunsg]

requests's release notes mention a couple recent changes about how requests will now strip the authorization info in the presence of certain redirects like https -> http

Interesting.

pip's got request ~2.20.0~ 2.21.0, so it won't have the fix for the "unintended Authorization header stripping". Maybe we should update the version vendored?

[cjerdonek]

It should have that, right, because it's in a lower version 2.20.1? https://github.com/requests/requests/blob/master/HISTORY.md#2201-2018-11-08

[cjerdonek]

Changing this to reflect that we can probably do something to make this situation easier to resolve / know what went wrong.

[pradyunsg]

@cjerdonek Anything actionable here for 19.0.2?

[cjerdonek]

No..

[cjerdonek]

This actually made me spot another possible issue vaguely related, the redirect to http instead of https shouldn't happen,

I believe this is the same as #3174.