Pipenv: Environment variables in Pipfile does not resolve

Created on 20 Jun 2018  ·  12Comments  ·  Source: pypa/pipenv

When I add a second source in my Pipfile with a private pypi registry, and then try to run pipenv update (or anything else which tries to install from it), I'm getting an error.

I've found similar issue, both of which are closed:
https://github.com/pypa/pipenv/issues/2351
https://github.com/pypa/pipenv/issues/1906

If I set the username and password as part of the source url directly in the Pipfile it works as expected, also when I create a ~/.netrc file with the machine, login and password.

Expected result

I expect the packages defined in Pipfile to be installed

Actual result
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${PYPI_AUTH}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${PYPI_AUTH}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:
  privatepackage

Finding the best candidates:
User for pypi.domain.io: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.domain.io
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 85, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 74, in main
    system=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 65, in resolve
    allow_global=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 442, in resolve_deps
    pre,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 352, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 157, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 142, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

Steps to replicate
Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${PYPI_AUTH}@pypi.domain.io/pypi"
verify_ssl = true
name = "private-pypi"

[packages]
privatepackage = {version = "*", index = "private-pypi"}

[requires]
python_version = "3.6"
Command

env PYPI_AUTH="user:password" pipenv update

Needs More Information Possible Bug
Source
picture ilons

Most helpful comment

Don't mind netrc, that was merely a distraction of mine it seems.
My problem is that pipenv does not read env variables as expected.

picture ilons on 22 Jun 2018
👍2

All 12 comments

Did you spell the variable wrong? PUPI_AUTH?

uranusjr picture uranusjr on 20 Jun 2018

No, I didn't, it sneaked in when I created the issue.

I just did a second verification on it to be sure, and I can still reproduce, with the exception that I don't get it to work with netrc (as I thought I did initially).

When I put the user/pass directly in the Pipfile it works as expected.

Reproduced with:
mkdir test; cd test

Pipfile:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${PYPI_AUTH}@pypi.domain.io/pypi"
#url = "https://user:[email protected]/pypi"
#url = "https://pypi.domain.io/pypi"
verify_ssl = true
name = "private-pypi"

[packages]
six = "*"
privatepackage = {version = "*", index = "private-pypi"}

[requires]
python_version = "3.6"
Running:

PYPI_AUTH="user:pass" && pipenv update || echo "${PYPI_AUTH}"

Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
elf.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

user:pass
ilons picture ilons on 21 Jun 2018

Pipenv doesn’t parse netrc files, if that’s relevant; it only reads environment variables

techalchemy picture techalchemy on 22 Jun 2018

Don't mind netrc, that was merely a distraction of mine it seems.
My problem is that pipenv does not read env variables as expected.

ilons picture ilons on 22 Jun 2018
👍2

Based on the error I think it is residing the variable but downstream there is an expectation of these being separate and for security reasons we only interpolate at the last possible moment. The lack of a separator is potentially throwing things off. Can you try passing these in separately as ${USER}:${PASS}?

techalchemy picture techalchemy on 22 Jun 2018

I can reproduce this issue on release 2018.05.18. However, it works when embedding the credential in source URL inside Pipfile directly like that in https://github.com/pypa/pipenv/issues/2389#issue-334035601

Update: I can reproduce this issue on release 2018.06.25 (released 4 hours ago) too.

PS: ~/.netrc works when no authentication specified for index-url for pip9 since requests supports it

I've spent some time to diagnostic this problem via running pipenv lock --verbose and found out the problem would be pipenv/utils.py where it uses index-urls from Pipfile.lock directly without evaluating it via os.path.expandvars(), so when it performs GET request against pypi source, it doesn't send a header named Authorization at all.

shawnzhu picture shawnzhu on 25 Jun 2018

@shawnzhu the approach we recommend and have recommended is ${USER}:${PASS} in your source header, otherwise these values are not properly split when they are parsed by urlparse. I am still waiting to hear whether using this approach provides resolution or not -- without that info, I'd say that we are not really looking further at this (which is why you can reproduce whatever the issue is)

techalchemy picture techalchemy on 26 Jun 2018
👍1

Passing them separately still gives the same issue:

Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${USER}:${PASS}@pypi.domain.io/pypi"
#url = "https://${PYPI_AUTH}@pypi.domain.io/pypi"
#url = "https://user:[email protected]/pypi"
#url = "https://pypi.domain.io/pypi"
verify_ssl = true
name = "private-pypi"

[packages]
six = "*"
privatepackage = {version = "*", index = "private-pypi"}

[requires]
python_version = "3.6"
Command

USER=user && PASS="pass" && pipenv update --verbose || echo "Got ${USER}:${PASS}"

Output
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${USER}:${PASS}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${USER}:${PASS}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:
  privatepackage
  six

Finding the best candidates:
User for pypi.domain.io: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.domain.io
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 85, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 74, in main
    system=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 65, in resolve
    allow_global=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 442, in resolve_deps
    pre,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 352, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 157, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 142, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

Got user:pass
ilons picture ilons on 27 Jun 2018

You should update your pipenv version, but I also can't replicate this at all --

 /t/test2  cat Pipfile
[[source]]
url = "https://${FAKE_PYPI}/simple"
verify_ssl = false
name = "kenneth-mirror"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"

 /t/test2  set -gx FAKE_PYPI pypi.kennethreitz.org
 /t/test2  pipenv install six
Creating a virtualenv for this project...
Pipfile: /tmp/test2/Pipfile
Using /home/hawk/.pyenv/versions/3.6.4/bin/python3.6m (3.6.4) to create virtualenv...
⠋Running virtualenv with interpreter /home/hawk/.pyenv/versions/3.6.4/bin/python3.6m
Using base prefix '/home/hawk/.pyenv/versions/3.6.4'
New python executable in /home/hawk/.virtualenvs/test2-h6PFdzq5/bin/python3.6m
Also creating executable in /home/hawk/.virtualenvs/test2-h6PFdzq5/bin/python
Installing setuptools, pip, wheel...done.
Setting project for test2-h6PFdzq5 to /tmp/test2

Virtualenv location: /home/hawk/.virtualenvs/test2-h6PFdzq5
Installing six...
Looking in indexes: https://pypi.kennethreitz.org/simple, https://pypi.org/simple
Collecting six
  Downloading https://pypi.kennethreitz.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.11.0

Adding six to Pipfile's [packages]...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (b69ba7)!
Installing dependencies from Pipfile.lock (b69ba7)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternativaly, run a command inside the virtualenv with pipenv run.

This works with pipenv update as well. We have tests running in CI that test against this. I can't replicate the behavior :/

techalchemy picture techalchemy on 28 Jun 2018

@techalchemy Do you expect the changes made in https://github.com/pypa/pipenv/pull/2484 will fix my issue as well?

ilons picture ilons on 5 Jul 2018

Those are released so if it helps it will work in the newest release

techalchemy picture techalchemy on 5 Jul 2018

So I got around doing as suggested and updated pipenv version, and did a few tests to see where this started working.

Fedora 28, Python 3.6.5, pip 9.0.3:
11.10.0: Failed
11.10.1, 11.10.2, 11.10.3, 11.10.4, 2018.5.18, 2018.6.25, 2018.7.1: Working

Shame on me that I did not update to newest version before testing.

It works well in the past couple of releases.

ilons picture ilons on 10 Jul 2018
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jacebrowning picture jacebrowning  ·  3Comments
hynek picture hynek  ·  3Comments
bgjelstrup picture bgjelstrup  ·  3Comments
erinxocon picture erinxocon  ·  3Comments
fbender picture fbender  ·  3Comments