Pipenv: Pipenv install fails to install sub dependencies from private repos

[siddartha1992]

Describe you environment

1. OSX (MAC)
2. Python version: 2.7.10
3. Pipenv version: 8.2.6

Issue

1) setup.py inside invocations repo -> installing CaaS package :

install_requires=[
    'CaaS>=1.0'
  ],

  dependency_links=[
    'https://<private_repo>#egg=CaaS-1.0'
  ],

2) Installing invocations in a virtualenv
python setup.py install

3) Valdiating CaaS installed correctly.

(test_1) c4b301cf5d25:invocations quj291$ pip freeze
CaaS==1.0

So far everything works.

4) Created a Pipfile

[requires]
python_version = '2.7'

[packages]
invocations = { git = 'git://<private-repo>/invocations',  ref = 'master' }

5) Fails to install invocations because CaaS package cannot be found: pipenv install

Collecting CaaS>=1.0 (from invocations)

  Could not find a version that satisfies the requirement CaaS>=1.0 (from invocations) (from versions: )
No matching distribution found for CaaS>=1.0 (from invocations)

6) Tried pipenv install --verbose

Collecting CaaS>=1.0 (from invocations)
  1 location(s) to search for versions of CaaS:
  * https://pypi.python.org/simple/caas/
  Getting page https://pypi.python.org/simple/caas/
  Looking up "https://pypi.python.org/simple/caas/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/caas/ HTTP/1.1" 404 33
  Status code 404 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/caas/: 404 Client Error: Not Found (caas does not exist) for url: https://pypi.python.org/simple/caas/ - skipping
Cleaning up...

Tries to get CaaS from pypi instead of private github repo's inside dependency_links of invocation's setup.py

Is this expected? How can I install CaaS package?

Thanks!

[techalchemy]

You need to include the editable flag for VCS repositories to resolve sub dependencies I believe

I.e. invocations = { git = 'git://<private-repo>/invocations', ref = 'master', editable = true}

Note that pipenv even issues a warning when you do this:

[siddartha1992]

1) Updated Pipfile

[requires]
python_version = '2.7'

[packages]
invocations = { git = '<private_repo>/invocations', ref = 'master', editable = true }

2) Output of pipenv install --verbose

c4b301cf5d25:docker-server-10-20 quj291$ pipenv install  --verbose
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches CaaS>=1.0
Tried: (no version found at all)

3) Ran with skip-lock option --skip-lock: pipenv install --skip-lock --verbose

Collecting CaaS>=1.0 (from invocations)
  1 location(s) to search for versions of CaaS:
  * https://pypi.python.org/simple/caas/
  Getting page https://pypi.python.org/simple/caas/
  Looking up "https://pypi.python.org/simple/caas/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/caas/ HTTP/1.1" 404 33
  Status code 404 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/caas/: 404 Client Error: Not Found (caas does not exist) for url: https://pypi.python.org/simple/caas/ - skipping
Cleaning up...

and

  Could not find a version that satisfies the requirement CaaS>=1.0 (from invocations) (from versions: )
No matching distribution found for CaaS>=1.0 (from invocations)

4) pipenv graph

invocations==0.1.5
  - boto [required: >=2.46, installed: 2.48.0]
  - CaaS [required: >=1.0, installed: ?]
  - invoke [required: >=0.15, installed: 0.21.0]
  - PyYAML [required: >=3.12, installed: 3.12]
  - simplejson [required: >=3.10, installed: 3.11.1]

Its still downloading CaaS from https://pypi.python.org/simple/caas/ instead of dependency_links in setup.py of invocations package.

Thanks!

[siddartha1992]

Updated the title to reflect the problem.

• boto, invoke, PyYAML, and simplejson (public repos) are all dependencies of invocations and they were installed without any issues.

• CaaS (private repo) wasn't able to install. pipenv tried to install this package from https://pypi.python.org/simple/caas/ instead of what's set in dependency_links in setup.py

[techalchemy]

Oh I see what you're saying, I either misunderstood or we solved one problem but uncovered another. I have an idea about where this might be happening but it will be messy to untangle since I don't have an example repository. Essentially, we aren't currently resolving anything in setup.py since we rely on pip to do that resolution. Out of curiosity, what is the output of pipenv run pip freeze?

[techalchemy]

P.S. Thanks for the update and the thorough report, it makes things much easier

[siddartha1992]

I have abstracted some additonal packages from Pipfile and setup.py, but here is the complete output of pipenv run pip freeze:

c4b301cf5d25:docker-server-10-20 quj291$ pipenv run pip freeze
ansible==2.4.0.0
asn1crypto==0.23.0
bcrypt==3.1.4
boto==2.48.0
cffi==1.11.2
cryptography==2.1.1
enum34==1.1.6
idna==2.6
invoke==0.21.0
ipaddress==1.0.18
Jinja2==2.9.6
MarkupSafe==1.0
paramiko==2.3.1
pyasn1==0.3.7
pycparser==2.18
PyNaCl==1.1.2
PyYAML==3.12
simplejson==3.11.1
six==1.11.0

Just a hunch, but might be related to : https://github.com/kennethreitz/pipenv/issues/571

[techalchemy]

Do you have underscores in your repo name?

[siddartha1992]

No, but it has hyphens. (referring to private repo)

[erinxocon]

Last check, hyphens through errors in repo links I think. Marking as a bug for fixing. Thanks for the through report @siddartha1992! And thanks for the ongoing help @techalchemy

[techalchemy]

@erinxocon the original issue was related to pep 440 normalization where we were replacing underscores with dashes. That has been an ongoing issue and has come up in a few places, but we don't do any manipulation that I know of with dashes. We have had a number of issues with VCS urls related to pip-tools resolving and may need /cc @vphilippon on this but not sure if this is in his wheelhouse. I've done a bunch of work on that code, just not sure how far to take the design decisions there vs resolving them in piptools

[techalchemy]

@siddartha1992 as a point of clarification on this issue, did pipenv fail to _install_ the dependencies or did it fail to _resolve_ the dependencies? The output you provided doesn't make it completely clear but there is a big difference as to the cause of the problem

[kennethreitz42]

known, designed this way

[manihamidi]

@siddartha1992 did the removal of hyphens fix your problem here? I don't have any hyphens or underscores and am facing similar difficulties.

[techalchemy]

@manihamidi I believe the issue with hyphens and underscores was resolved awhile ago, but this particular issue may be resolved in master— can you give that a try?

[manihamidi]

@techalchemy thanks for the prompt reply. See the repos PrivateA and PrivateB on my account page that I used to test this (PrivateA depends on PrivateB):

• installed pipenv from master:

$ pip install -U -e git+https://github.com/kennethreitz/pipenv.git#egg=pipenv-dev
...
Successfully installed pipenv

• pip installed PrivateA with --process-dependency-links: it installed both privateliba and privatelibb

$ pip install -e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0 --process-dependency-links
...
Successfully installed privateliba privatelibb-1.0.0

• tried pipenv hoping it would do the same (without --process-dependency-links as there is no such flag recognized): it did not work :disappointed:

$ pipenv install -e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0 --verbose
...
Could not find a version that satisfies the requirement privatelibb (from privateliba) (from versions: )
No matching distribution found for privatelibb (from privateliba)

I totally acknowledge that perhaps this isn't even the right paradigm for doing things and would love some feedback as to what is.

At the same time, there seems to also be acknowledgement (evinced by the backlash against the deprecation of dependency_links) that the use-case i'm looking for is not so insane/uncommon:

What is the "Human" way of getting automatic installation of sub-dependencies from github repos?

Thanks again for reading.

Detailed Console Responses:

OS: Windows 7
CONSOLE: Git bash
PYTHON: 2.7 (anaconda distribution)

Using pip:

$ pip install -e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0 --process-dependency-links

Obtaining privateliba from git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0
  Updating d:\scripts\appusingaandb\src\privateliba clone
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting privatelibb (from privateliba)
  Cloning https://github.com/manihamidi/privateB.git to c:\users\mani\appdata\local\temp\pip-build-yi1emy\privatelibb
Installing collected packages: privatelibb, privateliba
  Running setup.py install for privatelibb: started
    Running setup.py install for privatelibb: finished with status 'done'
  Running setup.py develop for privateliba
Successfully installed privateliba privatelibb-1.0.0

With pipenv:

$ pipenv install -e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0 --verbose

it doesn't work (as @siddartha1992 originally posted):

Installing -e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0…
Installing u'-e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0'
$ "C:\\Users\\Mani\\.virtualenvs\\AppUsingAandB-AVt2bpl0\\Scripts\\pip.exe" install  --src C:\Users\Mani\.virtualenvs\AppUsingAandB-AVt2bpl0\src --verbose   -e "git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0" -i https://pypi.python.org/simple --exists-action w
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
Obtaining privateliba from git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0
  Running command git config --get-regexp remote\..*\.url
  remote.origin.url https://github.com/manihamidi/privateA.git
  Clone in c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\src\privateliba exists, and has correct URL (https://github.com/manihamidi/privateA.git)
  Running command git rev-parse HEAD
  a98982f8827c03ec2642cc01083e219d207f371c
  Updating c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\src\privateliba clone
  Running command git version
  git version 2.9.0.windows.1
  Running command git fetch -q --tags
  Running command git show-ref
  a98982f8827c03ec2642cc01083e219d207f371c refs/heads/master
  a98982f8827c03ec2642cc01083e219d207f371c refs/remotes/origin/HEAD
  a98982f8827c03ec2642cc01083e219d207f371c refs/remotes/origin/master
  Running command git reset --hard -q a98982f8827c03ec2642cc01083e219d207f371c
  Running setup.py (path:C:\Users\Mani\.virtualenvs\AppUsingAandB-AVt2bpl0\src\privateliba\setup.py) egg_info for package privateliba
    Running command python setup.py egg_info
    WARNING: '' not a valid package name; please use only .-separated package names in setup.py
    running egg_info
    writing requirements to privateliba.egg-info\requires.txt
    writing privateliba.egg-info\PKG-INFO
    writing top-level names to privateliba.egg-info\top_level.txt
    writing dependency_links to privateliba.egg-info\dependency_links.txt
    reading manifest file 'privateliba.egg-info\SOURCES.txt'
    writing manifest file 'privateliba.egg-info\SOURCES.txt'
  Source in c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\src\privateliba has version 1.0.0, which satisfies requirement privateliba from git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0
Collecting privatelibb (from privateliba)
  1 location(s) to search for versions of privatelibb:
  * https://pypi.python.org/simple/privatelibb/
  Getting page https://pypi.python.org/simple/privatelibb/
  Looking up "https://pypi.python.org/simple/privatelibb/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/privatelibb/ HTTP/1.1" 404 40
  Status code 404 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/privatelibb/: 404 Client Error: Not Found (privatelibb does not exist) for url: https://pypi.python.org/simple/privatelibb/ - skipping
Cleaning up...
Exception information:
Traceback (most recent call last):
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\req\req_set.py", line 554, in _prepare_file
    require_hashes
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\req\req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "c:\users\mani\.virtualenvs\appusingaandb-avt2bpl0\lib\site-packages\pip\index.py", line 514, in find_requirement
    'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for privatelibb (from privateliba)

Error:  An error occurred while installing -e git+https://github.com/manihamidi/privateA.git#egg=privateliba-1.0.0!
  Could not find a version that satisfies the requirement privatelibb (from privateliba) (from versions: )
No matching distribution found for privatelibb (from privateliba)

[techalchemy]

@manihamidi I believe you can set the environment variable PIP_PROCESS_DEPENDENCY_LINKS=1 or something like that to have pip recognize this if you need it, can you see if that helps?

Also thanks for the super-detailed info, makes it really clear what you're trying to do and why its not working!

[manihamidi]

That worked! Thank you so much @techalchemy! :handshake:

Should this be added to the documentation somewhere? I'd be happy to do it.

More generally speaking, is the approach i'm taking kosher? Or is it "better practice" to create a private index and use some other flags (which?) to point pipenv to, for the private dependencies? I ask this as I realize the specification of "concrete" dependencies in setup.py is frowned upon by the experts.

[techalchemy]

@manihamidi it's not really pipenv documentation so it probably doesn't belong in our docs, it's just a pip environment variable. In general anything you can pass to pip on the command line can be set as an environment variable if you uppercase it and replace dashes with underscores.

You can setup a private index if you like, then you can specify it in your pipfile