Pipenv: support for multiple sources

[ghost]

I have tried to define two sources in Pipfile, something like this:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://myprivatepypi.com/simple"
verify_ssl = true
name = "internal"

[requires]
python_version = "2.7"

[dev-packages]
"coverage" = {version="*", index="pypi"}

[packages]
"my-private-package" = {version="*", index="internal"}

However, when I try pipenv install I run into an error, which includes a 404, saying that pipenv is looking for 'my-private-package' in https://pypi.python.org/simple.

https://github.com/kennethreitz/pipenv/blob/1faec7f251780e318264dee0699e6936e0c2000b/pipenv/project.py#L412-L413

Seems like the code might just be grabbing the first source in the Pipfile?

[kennethreitz42]

let me take a look

[kennethreitz42]

are you using the latest version?

[kennethreitz42]

this is confirmed as working on the latest version.

[kennethreitz42]

please ensure you're using the latest version and enclose the full traceback if so. Closing until then.

[ghost]

Thanks for taking a look. Here's the full set of commands:

egustavson /home/egustavson/projects/dea-core 
(master)$ pipenv --version
pipenv, version 8.0.5


egustavson /home/egustavson/projects/dea-core 
(master)$ cat Pipfile 
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypicloud.test.netflix.net:7002/pypi"
verify_ssl = true
name = "netflix"

[requires]
python_version = "2.7"

[dev-packages]
"nflx-kragle-scripts" = {version="*", index="netflix"}
"nflx-cloudsol-python-libs" = {version="*", index="netflix"}
"coverage" = {version="*", index="pypi"}

[packages]
"nflx-kragle" = {version="*", index="netflix"}
"nflx-kragle-quinto" = {version="*", index="netflix"}


egustavson /home/egustavson/projects/dea-core 
(master)$ cat Pipfile.lock
cat: Pipfile.lock: No such file or directory


egustavson /home/egustavson/projects/dea-core 
(master)$ pipenv install
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
404 Client Error: Not Found (nflx-cloudsol-python-libs does not exist) for url: https://pypi.python.org/simple/nflx-cloudsol-python-libs/

[kennethreitz42]

Ah, interesting.

[kennethreitz42]

Do as the warning suggests, and use install --skip-lock until we get to the bottom of this.

[ghost]

I tried it and got a stacktrace:

egustavson /home/egustavson/projects/dea-core 
(master)$ pipenv install --skip-lock
Installing dependencies from Pipfile…
Traceback (most recent call last):▉▉▉▉ 0/2 — 00:00:00
  File "/home/egustavson/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 1721, in install
    do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent, deploy=deploy)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 1238, in do_init
    skip_lock=skip_lock, verbose=verbose, concurrent=concurrent)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 794, in do_install_dependencies
    index=index
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pipenv/cli.py", line 1260, in pip_install
    if [x for x in requirements.parse(package_name.split('--hash')[0])][0].vcs:
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/requirements/parser.py", line 50, in parse
    yield Requirement.parse(line)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/requirements/requirement.py", line 220, in parse
    return cls.parse_line(line)
  File "/home/egustavson/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/requirements/requirement.py", line 198, in parse_line
    pkg_req = Req.parse(line)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3038, in parse
    req, = parse_requirements(s)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2986, in parse_requirements
    yield Requirement(line)
  File "/home/egustavson/.local/venvs/pipenv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2995, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'--truste'"
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/2 — 00:00:00

[kennethreitz42]

try latest, v8.0.6. I think I fixed it.

[ghost]

Seems to be working now, thanks!

[kennethreitz42]

excellent news!

[kennethreitz42]

i think you're our first customer using that feature haha

[helderco]

Hey @kennethreitz, second customer here :)

Been giving pipenv another shot since yesterday, but install hangs on some packages. I've tried removing my second source (we have internal, private packages), and no longer hangs.

$ pipenv --version
pipenv, version 8.2.0

Tried in Mac and Ubuntu.

[helderco]

One thing I did differently than @egustavson is I only specified my private index and left pypi untouched:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypi.example.org/simple"
verify_ssl = true
name = "internal"

[packages]
django = "~=1.11.5"
mysqlclient = "*"
django-environ = "*"
django-mfcookie = {version="*", index="internal"}

[helderco]

Tried this and still hangs:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypi.example.org/simple"
verify_ssl = true
name = "internal"

[packages]
django =  {version="~=1.11.5", index="pypi"}
mysqlclient =  {version="*", index="pypi"}
django-environ = {version="*", index="pypi"}
django-mfcookie = {version="*", index="internal"}

Here's the output:

$ pipenv --three install --verbose
Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…
Using /usr/local/bin/python3 to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/python3.6
Also creating executable in /Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A
Installing dependencies from Pipfile.lock…
Installing 'pytz==2017.2 --hash=sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d  --hash=sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9  --hash=sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9  --hash=sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c  --hash=sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67  --hash=sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9  --hash=sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043  --hash=sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4  --hash=sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-nD8Lmf-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-nD8Lmf-requirement.txt --require-hashes -i https://pypi.example.org/simple --exists-action w
Installing 'django-environ==0.4.4 --hash=sha256:e9c171b9d5f540e6f3bc42866941d9cd0bd77fb110a7c13a7c4857a2c08cfa40  --hash=sha256:ee2f8405d83137e3328b26b3de01bd715b5395fca22feb919dcc905fb6099cfa'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-aWT2Lj-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Installing 'six==1.11.0 --hash=sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb  --hash=sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-E_7UsF-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-E_7UsF-requirement.txt --require-hashes -i https://pypi.example.org/simple --exists-action w
Installing 'mysqlclient==1.3.12 --hash=sha256:b3b1a7e4468180afb79289b54069d9499242946a4cedf3928cbf6b2a13800016  --hash=sha256:d56e379c03efad746e84705cbb97401f60d1f98b05e11a27f2d9c2d043936974  --hash=sha256:371df79d000af56b4e540b7ce2120d1c9afb04b751bfce25a1eb609c50fd10ff  --hash=sha256:1e85e48b167e2af3bb08f273fdbd1ad6401cbe75057fa6513f97387dc7b282dc  --hash=sha256:2d9ec33de39f4d9c64ad7322ede0521d85829ce36a76f9dd3d6ab76a9c8648e5'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-GU9uWO-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w
Installing 'django-mfcookie==0.3.3'▉▉▉ 4/6 — 00:00:00
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-zn2odJ-requirement.txt -i https://pypi.example.org/simple --exists-action w
Installing 'django==1.11.5 --hash=sha256:89162f70a74aac62a53f975128faba6099a7ef2c9d8140a41ae9d6210bda05cd  --hash=sha256:1836878162dfdf865492bacfdff0321e4ee8f1e7d51d93192546000b54982b29'
$ "/Users/helder/.local/share/virtualenvs/pip-XBl5Wj6A/bin/pip" install   --verbose --no-deps  -r /var/folders/2v/nmb819xx3wsb7hcqvd68230w0000gn/T/pipenv-szL0JQ-requirement.txt --require-hashes -i https://pypi.python.org/simple --exists-action w

[kennethreitz42]

Confirmed with this Pipfile:

[[source]]

url = "http://pypi.home.kennethreitz.org/simple"
verify_ssl = false
name = "home"

[[source]]

url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "test"


[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"




[packages]

flask = {version="*", index="home"}
"cs18-sidecar" = {version="*", index="test"}
django = {version="*", index="pypi"}


[dev-packages]

[kennethreitz42]

Hmm, everything should be working as expected:

['flask -i http://pypi.home.kennethreitz.org/simple --trusted-host pypi.home.kennethreitz.org', 'cs18-sidecar -i https://testpypi.python.org/pypi', 'django -i https://pypi.python.org/simple']

digging in...

[techalchemy]

@kennethreitz I have some code on this as of yesterday which started to handle the cross-index resolution problem. It worked actually but it wrote the wrong info to the lockfile. I did note that we currently write no info to the lockfile

[techalchemy]

About indexes that is

[kennethreitz42]

@techalchemy please focus on the issues i have assigned to you :)

[kennethreitz42]

Those are the areas i need help with currently :)

[techalchemy]

I think we are supporting this properly now, closing!

[jonlil]

I think we are supporting this properly now, closing!

@techalchemy - can you elaborate why you think this is supported now?

[techalchemy]

@jonlil this is supported because the example Pipfile can be installed, locked, etc without issue:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://testpypi.python.org/pypi"
verify_ssl = true
name = "testpypi"

[packages]
pipenv-test-private-package = {version = "*", index = "testpypi"}
requests = "*"