pipenv install with private source doesn't use provided environment variables

Created on 4 Apr 2018 · 15Comments · Source: pypa/pipenv

Expected result

Have configured a private source which require username and password to authenticate. (It's a privately hosted https://jfrog.com/artifactory/ pypi repo.)

[[source]]
url = "http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@my-private-repo"
verify_ssl = false
name = "artifactory"

When running ARTIFACTORY_USERNAME=myuser ARTIFACTORY_PASSWORD=mypass pipenv install from the same directory as the pipfile on a project, I would expect the env-variables in the private source to use the values I provided.

I've also tried to provide env-variables in other way:

Setting env variable inline on the command (as shown in the example command above)
Exporting env variables
Putting env variables into a '.env' file that look like this:

ARTIFACTORY_USERNAME=myuser
ARTIFACTORY_PASSWORD=mypass

All of the above give me the same result error result (below).

When I provide the username and password directly inline in the pipfile it works fine, but when I use environment variables instead it fails with the following error:

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@my-private-repo --trusted-host my-private-repo

                          ROUND 1                           
Current constraints:
  aiopubsub==1.0.0
  asynctest
  pytest==3.3.2

Finding the best candidates:
  found candidate aiopubsub==1.0.0 (constraint was ==1.0.0)
User for artifactory.firmglobal.com: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): artifactory.firmglobal.com
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 82, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 71, in main
    clear=do_clear,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 63, in resolve
    verbose=verbose,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 425, in resolve_deps
    pre,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 336, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 116, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 101, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 722, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 831, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

/usr/local/lib/python2.7/dist-packages/pipenv/utils.py:1230: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/pipenv-ip4PTc-requirements'>
  warnings.warn(warn_message, ResourceWarning)

I assume it's just a confusing error message to tell me that the authentication fails, since I get the exact same error message if I put in an invalid username or password directly in the pipfile private source description.

Steps to replicate

pipfile which use source that require authentication. Use env variables for credentials:

[[source]]
url = "http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@private-repo"
verify_ssl = false
name = "my repo"

Run pipenv install where you provide credentials:
ARTIFACTORY_USERNAME=user ARTIFACTORY_PASSWORD=pass pipenv install

What am I doing wrong?

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python2.7/dist-packages/pipenv'

Python location: '/usr/bin/python'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.5: /usr/bin/python3.5m
3.5: /usr/bin/python3.5
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
2.7.12: /usr/bin/python
2.7.12: /usr/bin/python2
3.5.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.4.0-116-generic',
 'platform_system': 'Linux',
 'platform_version': '#140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018',
 'python_full_version': '2.7.12',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

LC_NUMERIC
MANDATORY_PATH
CONSCRIPT_OPTS
_LXSESSION_PID
XDG_GREETER_DATA_DIR
SPARK_HOME
PROJECT_HOME
LC_CTYPE
PYTHONDONTWRITEBYTECODE
XDG_CURRENT_DESKTOP
LC_PAPER
LOGNAME
XDG_SEAT
PATH
XDG_VTNR
QT_PLATFORM_PLUGIN
PYTHONUNBUFFERED
VIRTUALENVWRAPPER_SCRIPT
ZSH
DISPLAY
XDG_SESSION_DESKTOP
LANG
CONFIRMIT_CFG_DIR
TERM
SHELL
XDG_SESSION_PATH
XAUTHORITY
LANGUAGE
SHLVL
QT_QPA_PLATFORMTHEME
TERMINATOR_UUID
PIP_REQUIRE_VIRTUALENV
WINDOWID
_
SSH_AGENT_PID
HOME
LC_MONETARY
XDG_CONFIG_HOME
SAL_USE_VCLPLUGIN
CONSCRIPT_HOME
XDG_RUNTIME_DIR
WORKON_HOME
VIRTUALENVWRAPPER_PROJECT_FILENAME
LC_ADDRESS
SSH_AUTH_SOCK
VIRTUALENV_PYTHON
GDMSESSION
VIRTUALENVWRAPPER_WORKON_CD
CONFIRMIT_CFG_HOME
IBUS_DISABLE_SNOOPER
XDG_SEAT_PATH
PIP_PYTHON_PATH
XDG_SESSION_ID
DBUS_SESSION_BUS_ADDRESS
ORBIT_SOCKETDIR
VIRTUALENVWRAPPER_HOOK_DIR
LC_IDENTIFICATION
DESKTOP_SESSION
LSCOLORS
XDG_CONFIG_DIRS
DEFAULTS_PATH
XDG_SESSION_TYPE
OLDPWD
LS_COLORS
GDM_LANG
LC_TELEPHONE
LC_MEASUREMENT
PWD
COLORTERM
LC_NAME
XDG_MENU_PREFIX
LC_TIME
LESS
PAGER
XDG_DATA_DIRS
USER

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/andreas/.conscript/bin:/opt/spark-2.2.1/bin:/opt/flink/bin:/home/andreas/activator-dist-1.3.12/bin:/home/andreas/bin:/home/andreas/bin:/home/andreas/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL: /usr/bin/zsh
LANG: en_US.UTF-8
PWD: /path-to-project

Contents of Pipfile ('/path-to-project/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "http://$USERNAME:$PASSWORD@my-private-repo"
verify_ssl = false
name = "artifactory"

[dev-packages]
pytest = "==3.3.2"
asynctest = "*"
aiopubsub = "==1.0.0"

[packages]
aio-pika = "==1.8.1"
deepdiff = "==3.3.0"
pika = "*"
async-timeout = "*"
async-generator = "*"
"e1839a8" = {path = ".", editable = true}

[requires]
python_version = "3.6"

Dependency Resolution Category Type

Source

AndreasPresthammer

Most helpful comment

Found an issue https://github.com/pypa/pipenv/issues/2037

Gr1N on 23 Apr 2018

❤1 👍1

All 15 comments

Dotenv are only loaded with run and shell, not install (and other commands). Yes, I know it’s weird, but that’s a design decision I don’t have the authority change ¯\_(ツ)_/¯ You’ll need to load your environs in other forms.

uranusjr on 4 Apr 2018

It's fine that the .env approach doesn't work, but shouldn't the other ones work?

AndreasPresthammer on 4 Apr 2018

I am experiencing the same issue, it's downloading packages but failing to update/create Pipfile.lock

JoeyBG on 6 Apr 2018

Yes if these are set outside of .env this should work just fine.

Tested this against current master and it is definitely working, so if it isn't for you try master

techalchemy on 9 Apr 2018

Hi. It's not working with latest code, at least in my setup.

Here is my Pipfile

[[source]]
url = "https://${PYPI_USERNAME}:${PYPI_PASSWORD}@pypi.xpertsea.com/simple/"
verify_ssl = true
name = "pypi"

[packages]
dpath = "*"
scipy = "*"

[dev-packages]
mypy = "*"
pylint = "*"

[requires]
python_version = "3.6"

Here is what I get using pipenv lock :

Locking [dev-packages] dependencies…
candidates = self.finder.find_all_candidates(req_name)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

Drilling down the code it appears that at line 1061 in core.py when venv_resolve_deps is called, it'll eventually use the Project object that has been explicitly told to not expand venvs when parsed, so the call to PyPI fails with invalid credentials and produce this weird error. It actually happens down the line in utils.py, around line 345 where the actual PyPI call is.

I'm just not sure where the best place is to actually fix this issue without causing another bug later down the row..

JoeyBG on 9 Apr 2018

gotcha, so the issue occurs during locking because of project.py:

    @property
    def _lockfile(self):
        """Pipfile.lock divided by PyPI and external dependencies."""
        pfile = pipfile.load(self.pipfile_location, inject_env=False)
        lockfile = json.loads(pfile.lock())
        for section in ('default', 'develop'):
            lock_section = lockfile.get(section, {})
            for key in list(lock_section.keys()):
                norm_key = pep423_name(key)
                lockfile[section][norm_key] = lock_section.pop(key)
        return lockfile

techalchemy on 10 Apr 2018

I don't know if it's psosible to address this without also patching the piptools resolver debug logging functionality to prevent it from logging the interpolated index urls to the console...

techalchemy on 10 Apr 2018

I am also affected by this problem. Are there any news?

eolo999 on 12 Apr 2018

@eolo999 not yet unfortunately. We will update here when we get this sorted

techalchemy on 12 Apr 2018

@techalchemy when you plan to release new version of pipenv with this fix?

Gr1N on 22 Apr 2018

@Gr1N finishing up an automated suite that handles rebuilding vendored dependencies with licenses (due to some legal issues) => see #2035 -- I want to get this included because it adjusts for some SSL changes

Ideally this will be out today, at the very least I can put it up in prerelease for testing purposes once builds are passing

techalchemy on 22 Apr 2018

sorry for the holdup

techalchemy on 22 Apr 2018

pip install --upgrade --pre pipenv -- if things are working I'll move it out of prerelease tomorrow

techalchemy on 23 Apr 2018

@techalchemy version 11.10.1.dev1 works fine for me thanks! Also as I can see you need to updated docs:

Luckily - pipenv will hash your Pipfile before expanding environment variables (and, helpfully, will substitute the environment variables again when you install from the lock file - so no need to commit any secrets! Woo!)

It's not true anymore, as I see in Pipfile.lock env vars not hashed and it's great to my mind:

        ...
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "artifactory",
                "url": "${EXTRA_PYPI_INDEX}",
                "verify_ssl": true
            }
        ]
        ...

Gr1N on 23 Apr 2018

Found an issue https://github.com/pypa/pipenv/issues/2037

Gr1N on 23 Apr 2018

❤1 👍1

Was this page helpful?

0 / 5 - 0 ratings