Attempting to install a package with build dependencies from a private index with basic auth, without a ~/.netrc
file fails with EOFError: EOF when reading a line
on line 189 of _internal/download.py
(handle_401()
, username = six.moves.input("User for %s: " % parsed.netloc)
).
$ docker pull python:3.6-alpine
...
$ docker run -it -e PIP_INDEX_URL=https://zware:[email protected]/myindex python:3.6-alpine /bin/sh
/ # python3.6 -m venv venv
/ # ./venv/bin/python -m pip install -U pip
<pip 10 downloaded from private index, replaces pip 9.0.3>
/ # ./venv/bin/python -m pip install aiohttp # "--no-binary aiohttp" may be necessary on non-Alpine platforms
< downloads from private index >
Installing build dependencies ... error
Complete output from command /venv/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-me9p5l31 <full URL to setuptools on my private index> <full URL to wheel on my private index>:
Looking in indexes: <my index, with basic auth in URL>
Collecting setuptools==39.0.1 from <full URL to setuptools on my private index>
User for <my index netloc>: Exception:
Traceback (most recent call last):
File "/venv/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
status = self.run(options, args)
File "/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 291, in run
resolver.resolve(requirement_set)
File "/venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 103, in resolve
self._resolve_one(requirement_set, req)
File "/venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
self.require_hashes
File "/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 308, in prepare_linked_requirement
progress_bar=self.progress_bar
File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 837, in unpack_url
progress_bar=progress_bar
File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 674, in unpack_http_url
progress_bar)
File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 869, in _download_http_url
stream=True,
File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 397, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 625, in send
r = dispatch_hook('response', hooks, r, **kwargs)
File "/venv/lib/python3.6/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
_hook_data = hook(hook_data, **kwargs)
File "/venv/lib/python3.6/site-packages/pip/_internal/download.py", line 189, in handle_401
username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line
-------------------------------------------
Command "/venv/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-me9p5l31 <full URL to setuptools on my private index> <full URL to wheel on my private index>" failed with error code 2 in None
Hi,
I have been running in this as well, I believe a regression was introduced in 10.0.0 in that regard. The problem doesn't show up with pip 9.0.3.
same for me, the basic auth in the index URL is not used when downloading the packages
Since it may be helpful to others encountering this: I've successfully worked around the issue by adding the credentials to ~/.netrc
and leaving the user:pass
out of the index URL.
I can also confirm what @lopter said about pip <= 9.0.3 not being affected.
_Edit: Left out an all-important =
; pip == 9.0.3 is not affected, only 10.0.0._
https://github.com/pypa/pip/blob/master/src/pip/_internal/index.py#L833
basic auth has been removed in resp.url
(if I change resp.url
to url
, it works much better :) )
https://github.com/pypa/pip/blob/master/src/pip/_internal/index.py#L833
basic auth has been removed in resp.url (if I change resp.url to url, it works much better :) )
Interesting, I just looked at the history and nothing interesting shows up for that file.
Like everyone else here, I found that ~/.netrc
"solves" the issue.
However, there must be some caching involved somewhere because I can only reliably reproduce using Debian's sbuild
(we use dh-virtualenv
) which recreates a fresh chroot for every package build. Outside this fresh chroot, once the auth is accepted, the ~/.netrc
is no longer necessary. That's the part that worries me most. (IOW, can credentials be extracted from this cache?)
Same issue on OS X. I don't have the issue using the same repository in Ubuntu 18.04 in Docker, though.
Interesting, I just looked at the history and nothing interesting shows up for that file.
That's probably because that file has been moved.
/src/pip/_internal/index.py@master#L833
I'm hitting the same issue on Ubuntu 16.04 when running virtualenv
with my company's PyPI server which requires authentication; I've put
my username in pip.conf but I'd like to avoid also putting my password
there:
~/.pip/pip.conf:
[global]
index-url = https://fbauzac@MYSERVER/...
$ dpkg -l python-pip python-virtualenv python-minimal
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============================================-============================-============================-==================================================================================================
ii python-minimal 2.7.12-1~16.04 amd64 minimal subset of the Python language (default version)
ii python-pip 8.1.1-2ubuntu0.4 all alternative Python package installer
ii python-virtualenv 15.0.1+ds-3ubuntu1 all Python virtual environment creator
$ virtualenv /tmp/venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /tmp/venv/bin/python2
Not overwriting existing python script /tmp/venv/bin/python (you must use /tmp/venv/bin/python2)
Installing setuptools, pkg_resources, pip, wheel...
Complete output from command /tmp/venv/bin/python2 - setuptools pkg_resources pip wheel:
Collecting setuptools
Exception:
Traceback (most recent call last):
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 328, in run
wb.build(autobuilding=True)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file
finder, self.upgrade, require_hashes)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 442, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 400, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 545, in _get_pages
page = self._get_page(location)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 648, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 757, in get_page
"Cache-Control": "max-age=600",
File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 582, in send
r = dispatch_hook('response', hooks, r, **kwargs)
File "/tmp/venv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/hooks.py", line 31, in dispatch_hook
_hook_data = hook(hook_data, **kwargs)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 173, in handle_401
username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line
User for MYSERVER:
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv.py", line 2363, in <module>
main()
File "/usr/lib/python3/dist-packages/virtualenv.py", line 719, in main
symlink=options.symlink)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 988, in create_environment
download=download,
File "/usr/lib/python3/dist-packages/virtualenv.py", line 918, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 812, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /tmp/venv/bin/python2 - setuptools pkg_resources pip wheel failed with error code 2
I've checked with strace:
A process with pid 15337 is spawned:
execve("/usr/bin/python2", ["/usr/bin/python2", "/usr/lib/python3/dist-packages/virtualenv.py", "/tmp/venv"], [/* 64 vars */]
Isn't it odd that python2 is running a file under /usr/lib/python3? /usr/lib/python3/dist-packages/virtualenv.py has exactly the same contents as /usr/lib/python2.7/dist-packages/virtualenv.py on my system though.
It creates pipes [3, 4], [6, 7] and [8, 9].
Then forks -> creates subprocess with pid 15339
the subprocess 15339 sets up fd 3 as being its stdin, and fd 7 as both its stdout and stderr (dup2 syscalls).
Then the parent process 15337 writes a small Python program to fd 4, so it goes to 15339's stdin:
import sys
import pkgutil
import tempfile
import os
import pip
#cert_data = pkgutil.get_data("pip._vendor.requests", "cacert.pem")
cert_data = None
if cert_data is not None:
cert_file = tempfile.NamedTemporaryFile(delete=False)
cert_file.write(cert_data)
cert_file.close()
else:
cert_file = None
try:
args = ["install", "--ignore-installed"]
if cert_file is not None:
args += ["--cert", cert_file.name]
args += sys.argv[1:]
sys.exit(pip.main(args))
finally:
if cert_file is not None:
os.remove(cert_file.name)
Then it closes fd 4.
then reads from fd 6, for the purpose (I guess) to print the output to the user in case the subprocess fails. Note that it reads one byte at a time, which may be slow.
The subprocess 15339 is execve("/tmp/venv/bin/python2", ["/tmp/venv/bin/python2", "-", "setuptools", "pkg_resources", "pip", "wheel"], [/* 71 vars */]
But then it exits, supposedly because it could not read from its stdin (as the parent has closed the other end of the pipe: fd 4)
The related Python code is virtualenv.py function install_wheel() which puts a script into the stdin of the subprocess:
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
My solution is to patch pip/download.py in pip 8.1.1:
diff --git a/pip/download.py b/pip/download.py
index bbef9ea..546df79 100644
--- a/pip/download.py
+++ b/pip/download.py
@@ -145,7 +145,10 @@ class MultiDomainBasicAuth(AuthBase):
if username is None:
username, password = self.parse_credentials(parsed.netloc)
- if username or password:
+ if username:
+ if not password:
+ password = getpass.getpass("Password for {}: ".format(parsed.netloc))
+
# Store the username and password
self.passwords[netloc] = (username, password)
The equivalent patch against pip-master is:
diff --git a/src/pip/_internal/download.py b/src/pip/_internal/download.py
index 2bbe176..b470cca 100644
--- a/src/pip/_internal/download.py
+++ b/src/pip/_internal/download.py
@@ -169,7 +169,10 @@ class MultiDomainBasicAuth(AuthBase):
netrc_auth = get_netrc_auth(req.url)
username, password = netrc_auth if netrc_auth else (None, None)
- if username or password:
+ if username:
+ if not password:
+ password = getpass.getpass("Password for {}: ".format(parsed.netloc))
+
# Store the username and password
self.passwords[netloc] = (username, password)
Can anyone still reproduce this with the latest pip (19.2.3 as of this writing)?
Hello,
I have tried several versions of virtualenv.
Virtualenv embeds one version of pip (or two, in the case of e.g. 16.7.6).
It looks like virtualenv only uses its embedded pip, never the pip from the python environment. Indeed, upgrading pip itself to version 19.3 does not help.
My case is fixed with virtualenv 16.1.0+:
16.0.0 (containing pip 10.0.1): fail
16.1.0 (containing pip 18.1): success
Thanks a lot!
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.
Most helpful comment
Since it may be helpful to others encountering this: I've successfully worked around the issue by adding the credentials to
~/.netrc
and leaving theuser:pass
out of the index URL.I can also confirm what @lopter said about pip <= 9.0.3 not being affected.
_Edit: Left out an all-important
=
; pip == 9.0.3 is not affected, only 10.0.0._