pipenv doesn't respect pip.conf

[hrbonz]

I'm using devpi as a private pypi repository where I can proxy cache pypi packages and add my own in-house packages.

My pip.conf looks like the following:

[global]
index_url = https://pypi.priv.xxx/prod/+simple/
[search]
index = https://pypi.priv.xxx/prod/

Couldn't find another issue mentioning this problem.

[kennethreitz42]

These entries need to go into your Pipfile.

[hrbonz]

@kennethreitz I don't think I've made my case really clear so let me try to convince you with some use cases I've run into while getting started with pipenv.
First of, I'm just starting to dive into pipenv, how it works and its code. I know that you can specify a source in Pipfile, I also saw you can use named indexes, that looks similar to what can be done in ~/.pypirc.

Creating a new pipenv project/environment

$ mkdir foobar
$ cd foobar
$ pipenv install --verbose requests
⠋New python executable in /home/xxx/.local/share/virtualenvs/foobar-JdBU33Mf/bin/python         
Installing setuptools, pip, wheel...done.                                                      

Virtualenv location: /home/xxx/.local/share/virtualenvs/foobar-JdBU33Mf                         
Installing requests… 
⠙Installing u'requests'
$ "/home/xxx/.local/share/virtualenvs/foobar-JdBU33Mf/bin/pip" install   --verbose   "requests"
-i https://pypi.python.org/simple --exists-action w
Collecting requests
  1 location(s) to search for versions of requests:
  * https://pypi.python.org/simple/requests/
  Getting page https://pypi.python.org/simple/requests/
[...]
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
Cleaning up...

Adding requests to Pipfile's [packages]…
  PS: You have excellent taste! ✨ 🍰 ✨
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (76e6d4)!

Install a package from an additional index

When installing a package that pypi doesn't know, it won't work right away (it does work with pip or pip-tools as they use pip.conf settings).

$ pipenv install --verbose palantir
Installing palantir…
⠋Installing u'palantir'
$ "/home/hr/.local/share/virtualenvs/foobar-JdBU33Mf/bin/pip" install   --verbose   "palantir" -i https://pypi.python.org/simple --exists-action w
Collecting palantir
  1 location(s) to search for versions of palantir:
[...]
Error:  An error occurred while installing palantir!
  Could not find a version that satisfies the requirement palantir (from versions: )
No matching distribution found for palantir

If Pipfile gets set with a new index and the package is associated with this index (following https://docs.pipenv.org/advanced.html#specifying-package-indexes), trying to install the package from the command line will call pypi first anyway:

$ pipenv install --verbose palantir
Installing palantir… 
⠋Installing u'palantir'                                                                        
$ "/home/hr/.local/share/virtualenvs/foobar-JdBU33Mf/bin/pip" install   --verbose   "palantir"
-i https://pypi.python.org/simple --exists-action w                                            
⠋$ "/home/hr/.local/share/virtualenvs/foobar-JdBU33Mf/bin/pip" install   --verbose   "palantir"
 -i https://pypi.priv.xxx/prod/+simple/ --exists-action w                
Collecting palantir                         
  1 location(s) to search for versions of palantir:
[...]
Successfully installed palantir-1.1.5
Cleaning up...

Adding palantir to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Use a requirement file generated by pip-tools

When using pip-tools, the requirements file will start with the index used to install packages:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
--index-url https://pypi.priv.xxx/prod/+simple/

bcrypt==3.0.0
[...]

This setting is not respected when using the requirement file (starting with a clean project):

$ pipenv install -r requirements.txt
Requirements file provided! Importing into Pipfile…
Pipfile.lock (c23e27) out of date, updating to (3c7b08)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
CRITICAL:pip.index:Could not find a version that satisfies the requirement palantir==1.1.5 (from versions: )
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.

Observations

I feel that there is some sort of "gap" in the handling of indexes and the UX:

• a user can't specify an index (by name or resource) while installing through the command line which forces one to create an entry for the index in Pipfile and associate every package with an index. That's not very convenient in the case of a bunch of packages from pypi and only one from a private index.
• when installing a package from the command line, what is the default index that should be used?
• when installing a package from a custom index, it is not possible to specify it on the command line.
• when an index is specified in a requirement file, it is ignored.

I'm diving in the code to provide a PR to cover the problems exposed above:

• on initialization of a project, check pip.conf for custom indexes
• define a default index in Pipfile to be used when none is specified
• provide an index by name or resource on the command line
• respect index provided in a requirement file and add it to Pipfile

If an index is used either in a requirement file or on the command line but is unknown to Pipfile then an entry should be added with an automatic name similar to the venv naming used in pipenv.

Why do I care so much? Living in China, pypi is not always available or fast (numerous timeouts or dead slowness) so having an index like devpi that caches and allows me to mix my private packages is a double win. That makes our dev, testing, docker builds, etc way faster.

As an example, this is a pretty classic behavior happening when timeouts get involved:

Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches requests==2.17.3,==2.18.4
Tried: 0.2.0, 0.2.0, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.3.0, 0.3.0, 0.3.1, 0.3.1, 0.3.2, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.4.0, 0.4.0, 0.4.1, 0.4.1, 0.5.0, 0.5.0, 0.5.1, 0.5.1, 0.6.0, 0.6.0, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 0.6.3, 0.6.3, 0.6.4, 0.6.4, 0.6.5, 0.6.5, 0.6.6, 0.6.6, 0.7.0, 0.7.0, 0.7.1, 0.7.1, 0.7.2, 0.7.2, 0.7.3, 0.7.3, 0.7.4, 0.7.4, 0.7.5, 0.7.5, 0.7.6, 0.7.6, 0.8.0, 0.8.0, 0.8.1, 0.8.1, 0.8.2, 0.8.2, 0.8.3, 0.8.3, 0.8.4, 0.8.4, 0.8.5, 0.8.5, 0.8.6, 0.8.6, 0.8.7, 0.8.7, 0.8.8, 0.8.8, 0.8.9, 0.8.9, 0.9.0, 0.9.0, 0.9.1, 0.9.1, 0.9.2, 0.9.2, 0.9.3, 0.9.3, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.10.2, 0.10.2, 0.10.3, 0.10.3, 0.10.4, 0.10.4, 0.10.6, 0.10.6, 0.10.7, 0.10.7, 0.10.8, 0.10.8, 0.11.1, 0.11.1, 0.11.2, 0.11.2, 0.12.0, 0.12.0, 0.12.1, 0.12.1, 0.13.0, 0.13.0, 0.13.1, 0.13.1, 0.13.2, 0.13.2, 0.13.3, 0.13.3, 0.13.4, 0.13.4, 0.13.5, 0.13.5, 0.13.6, 0.13.6, 0.13.7, 0.13.7, 0.13.8, 0.13.8, 0.13.9, 0.13.9, 0.14.0, 0.14.0, 0.14.1, 0.14.1, 0.14.2, 0.14.2, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.0.2, 1.0.2, 1.0.3, 1.0.3, 1.0.4, 1.0.4, 1.1.0, 1.1.0, 1.2.0, 1.2.0, 1.2.1, 1.2.1, 1.2.2, 1.2.2, 1.2.3, 1.2.3, 2.0.0, 2.0.0, 2.0.0, 2.0.0, 2.0.1, 2.0.1, 2.0.1, 2.0.1, 2.1.0, 2.1.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.2.0, 2.2.0, 2.2.1, 2.2.1, 2.2.1, 2.2.1, 2.3.0, 2.3.0, 2.3.0, 2.3.0, 2.4.0, 2.4.0, 2.4.0, 2.4.0, 2.4.1, 2.4.1, 2.4.1, 2.4.1, 2.4.2, 2.4.2, 2.4.2, 2.4.2, 2.4.3, 2.4.3, 2.4.3, 2.4.3, 2.5.0, 2.5.0, 2.5.0, 2.5.0, 2.5.1, 2.5.1, 2.5.1, 2.5.1, 2.5.2, 2.5.2, 2.5.2, 2.5.2, 2.5.3, 2.5.3, 2.5.3, 2.5.3, 2.6.0, 2.6.0, 2.6.0, 2.6.0, 2.6.1, 2.6.1, 2.6.1, 2.6.1, 2.6.2, 2.6.2, 2.6.2, 2.6.2, 2.7.0, 2.7.0, 2.7.0, 2.7.0, 2.8.0, 2.8.0, 2.8.0, 2.8.0, 2.8.1, 2.8.1, 2.8.1, 2.8.1, 2.9.0, 2.9.0, 2.9.0, 2.9.0, 2.9.1, 2.9.1, 2.9.1, 2.9.1, 2.9.2, 2.9.2, 2.9.2, 2.9.2, 2.10.0, 2.10.0, 2.10.0, 2.10.0, 2.11.0, 2.11.0, 2.11.0, 2.11.0, 2.11.1, 2.11.1, 2.11.1, 2.11.1, 2.12.0, 2.12.0, 2.12.0, 2.12.0, 2.12.1, 2.12.1, 2.12.1, 2.12.1, 2.12.2, 2.12.2, 2.12.2, 2.12.2, 2.12.3, 2.12.3, 2.12.3, 2.12.3, 2.12.4, 2.12.4, 2.12.4, 2.12.4, 2.12.5, 2.12.5, 2.12.5, 2.12.5, 2.13.0, 2.13.0, 2.13.0, 2.13.0, 2.14.0, 2.14.0, 2.14.0, 2.14.0, 2.14.1, 2.14.1, 2.14.1, 2.14.1, 2.14.2, 2.14.2, 2.14.2, 2.14.2, 2.15.1, 2.15.1, 2.15.1, 2.15.1, 2.16.0, 2.16.0, 2.16.0, 2.16.0, 2.16.1, 2.16.1, 2.16.1, 2.16.1, 2.16.2, 2.16.2, 2.16.2, 2.16.2, 2.16.3, 2.16.3, 2.16.3, 2.16.3, 2.16.4, 2.16.4, 2.16.4, 2.16.4, 2.16.5, 2.16.5, 2.16.5, 2.16.5, 2.17.0, 2.17.0, 2.17.0, 2.17.0, 2.17.1, 2.17.1, 2.17.1, 2.17.1, 2.17.2, 2.17.2, 2.17.2, 2.17.2, 2.17.3, 2.17.3, 2.17.3, 2.17.3, 2.18.0, 2.18.0, 2.18.0, 2.18.0, 2.18.1, 2.18.1, 2.18.1, 2.18.1, 2.18.2, 2.18.2, 2.18.2, 2.18.2, 2.18.3, 2.18.3, 2.18.3, 2.18.3, 2.18.4, 2.18.4, 2.18.4, 2.18.4

[eromoe]

@kennethreitz
Nope. For real example, I live in China. The speed to pypi.python.org is usually lower than 50k/s , I have to set a global china mirror. Or I would see a lot of

  File "d:\python27\lib\site-packages\pipenv\patched\pip\_vendor\requests\packages\urllib3\response.
py", line 324, in read
    flush_decoder = True
  File "d:\python27\lib\contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "d:\python27\lib\site-packages\pipenv\patched\pip\_vendor\requests\packages\urllib3\response.
py", line 246, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip._vendor.requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='pypi.py
thon.org', port=443): Read timed out.

So, you mean every time I use pipenv need to write a pip.conf[Pipfile] for it ? It is unacceptable for me.

[dAnjou]

These entries may contain credentials which certainly won't go into the Pipfile which is supposed to go into the project repo.

[erinxocon]

@eromoe @hrbonz So is the problem that pipenv resolves pypi first instead of private servers?

[eromoe]

For me, it is. I really need a global mirror settings.

pip.conf is follow:

1. Firstly the site-wide file is read, then
2. The per-user file is read, and finally
3. The virtualenv-specific file is read.

respect pip.conf means pip user can switch to pipenv seamlessly.

[tuukkamustonen]

I like the idea of being able to put those configs in Pipfile (to easily share the config to other devs), but pipenv definitively must also respect pip.conf (resort to it for anything not defined in Pipfile).

[hrbonz]

@erinxocon this is one of the issues I identified. I've been pretty busy with work but hope to push first PR before next week.

[galuszkak]

Hi, @kennethreitz thanks for awesome project.

I have question, will you reconsider Your position on that issue after arguments brought into that discussion?

I have another very similar use case with pip.conf and having credentials in seperate pip.conf file is valid for having predictable build on CI/CD pipeline and local dev machine.

@hrbonz for now I found solution to use just $PIP_INDEX_URL env variables from pip until pip.conf file will be supported by pipenv. Those $PIP_VARIABLE could be sourced for now from .env file.

[mmoya]

I think that combining pypa/pip#3728 and custom indexes would be _the_ solution for private indexes without exposing credentials. Also commented in #1406.

[manselmi]

Here's one workaround if all you need are the index name and URL (e.g. using devpi). You can invoke this shell function whenever you want to create a Pipfile that has your custom index name and URL but is otherwise pristine.

pipenv_init() {
  # pipenv issue #856: pipenv doesn't respect pip.conf
  # https://github.com/pypa/pipenv/issues/856

  # This function accepts one optional argument: the path to the pipenv
  # executable. If not set or empty, defaults to PATH lookup.
  local PIPENV="${1:-pipenv}"
  if ! command -v -- "${PIPENV}" > /dev/null; then
    >&2 printf '%s\n' 'pipenv not found'
    return 1
  fi

  # Check for required environment variables.
  if [[ -z "${PIPENV_INDEX_NAME-}" ]] || [[ -z "${PIPENV_INDEX_URL-}" ]]; then
    >&2 printf '%s\n' 'PIPENV_INDEX_{NAME,URL} env vars must be set and not empty'
    return 1
  fi

  # Create fresh Pipfile and virtualenv.
  #
  # While doing so, move requirements.txt out of the way so pipenv
  # doesn't attempt to populate the virtualenv before we have a chance
  # to modify Pipfile.
  # https://github.com/pypa/pipenv/blob/v9.0.3/pipenv/cli.py#L308-L330
  # https://github.com/pypa/pipenv/blob/v9.0.3/pipenv/project.py#L117-L119
  # https://github.com/pypa/pipenv/blob/v9.0.3/pipenv/project.py#L231-L240
  # https://github.com/pypa/pipenv/blob/v9.0.3/pipenv/utils.py#L1112-L1124
  local TEMP_REQUIREMENTS_TXT
  "${PIPENV}" --rm || true
  rm -f -- Pipfile Pipfile.lock
  if [[ -f requirements.txt ]]; then
    TEMP_REQUIREMENTS_TXT="$(mktemp)"
    mv -- requirements.txt "${TEMP_REQUIREMENTS_TXT}"
  fi
  "${PIPENV}" install
  if [[ -n "${TEMP_REQUIREMENTS_TXT}" ]]; then
    mv -- "${TEMP_REQUIREMENTS_TXT}" requirements.txt
  fi
  rm -- Pipfile.lock

  # Within Pipfile's `[[source]]` section, set `name` to
  # `${PIPENV_INDEX_NAME}` and `url` to `${PIPENV_INDEX_URL}`.
  local TEMP_PIPFILE="$(mktemp)"
  < Pipfile \
      sed \
      -e '/^\[\[source\]\]$/,/^\[/ { s|^\(name = \).*|\1"'"${PIPENV_INDEX_NAME}"'"| ; s|^\(url = \).*|\1"'"${PIPENV_INDEX_URL}"'"| ; }' \
    > "${TEMP_PIPFILE}"
  mv -- "${TEMP_PIPFILE}" Pipfile
}

[ninrod]

gentleman, seriously. I have a simple use case that directly is affected by this issue.

I have a project that uses pipenv. I write this project both at home and at work.
At home, I have no problem. At work, I have to use the internal pipy index.

As the Pipfile is commited and pushed to git, I cannot keep changing it as I hop from home to work. I would like to have an external configuration that signals to pipenv that I have to use another index.

putting this config inside the Pipfile does not work.

[GhostofGoes]

Any updates on this?

[techalchemy]

@hrbonz @ninrod @GhostofGoes fixed in #1769 and #1809 -- environment variables in Pipfiles are now expanded at runtime

Thanks all for your patience, our highest priority has been the core functionality of the codebase, so features like this one tend to slip through the cracks. Always happy to discuss contributions for items we aren't currently prioritizing however!

[umrashrf]

The only alternative I can think until this gets fixed is to override the PyPi URL so all traffic is forwarded to the internal mirror of PyPi (for example, Artifactory or whatnot). Since PyPi is restricted internally anyway.