Pip: Pip 10 basic auth failure when installing build dependencies

[zware]

• Pip version: 10.0.0
• Python version: 3.6.5
• Operating system: Linux (Alpine, Gentoo). Reproduced once on macOS; could not a second time.

Description:

Attempting to install a package with build dependencies from a private index with basic auth, without a ~/.netrc file fails with EOFError: EOF when reading a line on line 189 of _internal/download.py (handle_401(), username = six.moves.input("User for %s: " % parsed.netloc)).

What I've run:

$ docker pull python:3.6-alpine
...
$ docker run -it -e PIP_INDEX_URL=https://zware:[email protected]/myindex python:3.6-alpine /bin/sh
/ # python3.6 -m venv venv
/ # ./venv/bin/python -m pip install -U pip
<pip 10 downloaded from private index, replaces pip 9.0.3>
/ # ./venv/bin/python -m pip install aiohttp  # "--no-binary aiohttp" may be necessary on non-Alpine platforms
< downloads from private index >
  Installing build dependencies ... error
  Complete output from command /venv/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-me9p5l31 <full URL to setuptools on my private index> <full URL to wheel on my private index>:
  Looking in indexes: <my index, with basic auth in URL>
  Collecting setuptools==39.0.1 from <full URL to setuptools on my private index>
  User for <my index netloc>: Exception:
  Traceback (most recent call last):
    File "/venv/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
      status = self.run(options, args)
    File "/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 291, in run
      resolver.resolve(requirement_set)
    File "/venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 103, in resolve
      self._resolve_one(requirement_set, req)
    File "/venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
      self.require_hashes
    File "/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 308, in prepare_linked_requirement
      progress_bar=self.progress_bar
    File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 837, in unpack_url
      progress_bar=progress_bar
    File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 674, in unpack_http_url
      progress_bar)
    File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 869, in _download_http_url
      stream=True,
    File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
      return self.request('GET', url, **kwargs)
    File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 397, in request
      return super(PipSession, self).request(method, url, *args, **kwargs)
    File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
      resp = self.send(prep, **send_kwargs)
    File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 625, in send
      r = dispatch_hook('response', hooks, r, **kwargs)
    File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
      _hook_data = hook(hook_data, **kwargs)
    File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 189, in handle_401
      username = six.moves.input("User for %s: " % parsed.netloc)
  EOFError: EOF when reading a line

-------------------------------------------
Command "/venv/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-me9p5l31 <full URL to setuptools on my private index> <full URL to wheel on my private index>" failed with error code 2 in None

[lopter]

Hi,

I have been running in this as well, I believe a regression was introduced in 10.0.0 in that regard. The problem doesn't show up with pip 9.0.3.

[mgu]

same for me, the basic auth in the index URL is not used when downloading the packages

[zware]

Since it may be helpful to others encountering this: I've successfully worked around the issue by adding the credentials to ~/.netrc and leaving the user:pass out of the index URL.

I can also confirm what @lopter said about pip <= 9.0.3 not being affected.

_Edit: Left out an all-important =; pip == 9.0.3 is not affected, only 10.0.0._

[mgu]

https://github.com/pypa/pip/blob/master/src/pip/_internal/index.py#L833
basic auth has been removed in resp.url (if I change resp.url to url, it works much better :) )

[lopter]

https://github.com/pypa/pip/blob/master/src/pip/_internal/index.py#L833
basic auth has been removed in resp.url (if I change resp.url to url, it works much better :) )

Interesting, I just looked at the history and nothing interesting shows up for that file.

[RemiCardona]

Like everyone else here, I found that ~/.netrc "solves" the issue.

However, there must be some caching involved somewhere because I can only reliably reproduce using Debian's sbuild (we use dh-virtualenv) which recreates a fresh chroot for every package build. Outside this fresh chroot, once the auth is accepted, the ~/.netrc is no longer necessary. That's the part that worries me most. (IOW, can credentials be extracted from this cache?)

[austinbutler]

Same issue on OS X. I don't have the issue using the same repository in Ubuntu 18.04 in Docker, though.

[pradyunsg]

Interesting, I just looked at the history and nothing interesting shows up for that file.

That's probably because that file has been moved.

/src/pip/_internal/index.py@master#L833

3070 seems to be the last PR to modify it.

[fbauzac]

I'm hitting the same issue on Ubuntu 16.04 when running virtualenv
with my company's PyPI server which requires authentication; I've put
my username in pip.conf but I'd like to avoid also putting my password
there:

~/.pip/pip.conf:

[global]
index-url = https://fbauzac@MYSERVER/...

$ dpkg -l python-pip python-virtualenv python-minimal
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                           Version                      Architecture                 Description
+++-==============================================-============================-============================-==================================================================================================
ii  python-minimal                                 2.7.12-1~16.04               amd64                        minimal subset of the Python language (default version)
ii  python-pip                                     8.1.1-2ubuntu0.4             all                          alternative Python package installer
ii  python-virtualenv                              15.0.1+ds-3ubuntu1           all                          Python virtual environment creator

$ virtualenv /tmp/venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /tmp/venv/bin/python2
Not overwriting existing python script /tmp/venv/bin/python (you must use /tmp/venv/bin/python2)
Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /tmp/venv/bin/python2 - setuptools pkg_resources pip wheel:
  Collecting setuptools
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 328, in run
    wb.build(autobuilding=True)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/wheel.py", line 748, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file
    finder, self.upgrade, require_hashes)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 442, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 400, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 545, in _get_pages
    page = self._get_page(location)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 648, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 757, in get_page
    "Cache-Control": "max-age=600",
  File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 582, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 173, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line
User for MYSERVER:
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 2363, in <module>
    main()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 719, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 988, in create_environment
    download=download,
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 918, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 812, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /tmp/venv/bin/python2 - setuptools pkg_resources pip wheel failed with error code 2

I've checked with strace:

A process with pid 15337 is spawned:
execve("/usr/bin/python2", ["/usr/bin/python2", "/usr/lib/python3/dist-packages/virtualenv.py", "/tmp/venv"], [/* 64 vars */]
Isn't it odd that python2 is running a file under /usr/lib/python3? /usr/lib/python3/dist-packages/virtualenv.py has exactly the same contents as /usr/lib/python2.7/dist-packages/virtualenv.py on my system though.
It creates pipes [3, 4], [6, 7] and [8, 9].
Then forks -> creates subprocess with pid 15339
the subprocess 15339 sets up fd 3 as being its stdin, and fd 7 as both its stdout and stderr (dup2 syscalls).
Then the parent process 15337 writes a small Python program to fd 4, so it goes to 15339's stdin:

import sys
import pkgutil
import tempfile
import os

import pip

#cert_data = pkgutil.get_data("pip._vendor.requests", "cacert.pem")
cert_data = None
if cert_data is not None:
    cert_file = tempfile.NamedTemporaryFile(delete=False)
    cert_file.write(cert_data)
    cert_file.close()
else:
    cert_file = None

try:
    args = ["install", "--ignore-installed"]
    if cert_file is not None:
        args += ["--cert", cert_file.name]
    args += sys.argv[1:]

    sys.exit(pip.main(args))
finally:
    if cert_file is not None:
        os.remove(cert_file.name)

Then it closes fd 4.
then reads from fd 6, for the purpose (I guess) to print the output to the user in case the subprocess fails. Note that it reads one byte at a time, which may be slow.

The subprocess 15339 is execve("/tmp/venv/bin/python2", ["/tmp/venv/bin/python2", "-", "setuptools", "pkg_resources", "pip", "wheel"], [/* 71 vars */] At some point, it requests authentication by printing a prompt "User for MYSERVER: " to stdout.
But then it exits, supposedly because it could not read from its stdin (as the parent has closed the other end of the pipe: fd 4)

The related Python code is virtualenv.py function install_wheel() which puts a script into the stdin of the subprocess:

call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)

My solution is to patch pip/download.py in pip 8.1.1:

diff --git a/pip/download.py b/pip/download.py
index bbef9ea..546df79 100644
--- a/pip/download.py
+++ b/pip/download.py
@@ -145,7 +145,10 @@ class MultiDomainBasicAuth(AuthBase):
         if username is None:
             username, password = self.parse_credentials(parsed.netloc)

-        if username or password:
+        if username:
+            if not password:
+                password = getpass.getpass("Password for {}: ".format(parsed.netloc))
+
             # Store the username and password
             self.passwords[netloc] = (username, password)

The equivalent patch against pip-master is:

diff --git a/src/pip/_internal/download.py b/src/pip/_internal/download.py
index 2bbe176..b470cca 100644
--- a/src/pip/_internal/download.py
+++ b/src/pip/_internal/download.py
@@ -169,7 +169,10 @@ class MultiDomainBasicAuth(AuthBase):
             netrc_auth = get_netrc_auth(req.url)
             username, password = netrc_auth if netrc_auth else (None, None)

-        if username or password:
+        if username:
+            if not password:
+                password = getpass.getpass("Password for {}: ".format(parsed.netloc))
+
             # Store the username and password
             self.passwords[netloc] = (username, password)

[chrahunt]

Can anyone still reproduce this with the latest pip (19.2.3 as of this writing)?

[fbauzac]

Hello,

I have tried several versions of virtualenv.

Virtualenv embeds one version of pip (or two, in the case of e.g. 16.7.6).

It looks like virtualenv only uses its embedded pip, never the pip from the python environment. Indeed, upgrading pip itself to version 19.3 does not help.

My case is fixed with virtualenv 16.1.0+:
16.0.0 (containing pip 10.0.1): fail
16.1.0 (containing pip 18.1): success

Thanks a lot!

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.