Pipenv: Cannot install a pipenv app with a dependency, both hosted on github

[irgendsontyp]

I created two small test projects on GitHub to understand how pipenv works. One project is a simple app that uses the other project as dependency. The app project is located here: pipenv-app. The dependency project is located here: pipenv-dependency.

When trying to install the app via $ pipenv install -e git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app --verbose in a virtualenv created via $ pipenv --three, I get this output:

```Installing -e git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app…
⠋Installing '-e git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app'
$ "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/bin/pip" install --src /home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/src --verbose -e "git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app" -i https://pypi.python.org/simple --exists-action w
Obtaining pipenv-app from git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app
Cloning https://github.com/irgendsontyp/pipenv-app to /home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/src/pipenv-app
Running command git clone -q https://github.com/irgendsontyp/pipenv-app /home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/src/pipenv-app
Running setup.py (path:/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/src/pipenv-app/setup.py) egg_info for package pipenv-app
Running command python setup.py egg_info
running egg_info
creating pipenv_app.egg-info
writing pipenv_app.egg-info/PKG-INFO
writing dependency_links to pipenv_app.egg-info/dependency_links.txt
writing requirements to pipenv_app.egg-info/requires.txt
writing top-level names to pipenv_app.egg-info/top_level.txt
writing manifest file 'pipenv_app.egg-info/SOURCES.txt'
reading manifest file 'pipenv_app.egg-info/SOURCES.txt'
writing manifest file 'pipenv_app.egg-info/SOURCES.txt'
Source in /home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/src/pipenv-app has version 1.0.0, which satisfies requirement pipenv-app from git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app
Collecting pipenv-dependency (from pipenv-app)
1 location(s) to search for versions of pipenv-dependency:

• https://pypi.python.org/simple/pipenv-dependency/
  Getting page https://pypi.python.org/simple/pipenv-dependency/
  Looking up "https://pypi.python.org/simple/pipenv-dependency/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/pipenv-dependency/ HTTP/1.1" 404 46
  Status code 404 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/pipenv-dependency/: 404 Client Error: Not Found (pipenv-dependency does not exist) for url: https://pypi.python.org/simple/pipenv-dependency/ - skipping
  Cleaning up...
  Exception information:
  Traceback (most recent call last):
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
  status = self.run(options, args)
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
  wb.build(autobuilding=True)
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
  self.requirement_set.prepare_files(self.finder)
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
  ignore_dependencies=self.ignore_dependencies))
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
  require_hashes
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
  self.link = finder.find_requirement(self, upgrade)
  File "/home/maxx/.local/share/virtualenvs/pipenv-test-rA_KzhnY/lib/python3.6/site-packages/pip/index.py", line 514, in find_requirement
  'No matching distribution found for %s' % req
  pip.exceptions.DistributionNotFound: No matching distribution found for pipenv-dependency (from pipenv-app)

Error: An error occurred while installing -e git+https://github.com/irgendsontyp/pipenv-app#egg=pipenv-app!
Could not find a version that satisfies the requirement pipenv-dependency (from pipenv-app) (from versions: )
No matching distribution found for pipenv-dependency (from pipenv-app)
```

What am I doing wrong?

OS: Arch Linux
Python -V: Python 3.6.4
Pipenv --version: pipenv, version 9.0.1

[kennethreitz42]

Your dependency is not on PyPi.

[pmlk]

@irgendsontyp add the following to your setup.py:

setup(
    ...
    dependency_links=[
       "git+https://github.com/irgendsontyp/pipenv-dependency.git@master#egg=pipenvdependency-1.0.0",
    ],
   ...
)

Then try this:

$ cd pipenv-app
$ export PIP_PROCESS_DEPENDENCY_LINKS=1
$ pipenv install -e .

@irgendsontyp, inspired by your two mini-projects I sort of recreated them. Check out PipenvDependency and PipenvApp. Hope you don't mind that I stole your naming convention. Let me know if this works for you.

[irgendsontyp]

@pmlk Thanks for pointing out that using dependency_links could solve my issue.

I'm glad that I could inspire you with the small project as well as my naming convention.

Meanwhile I have read about that dependency_links entry in setup.py, included it in my project and it works very fine. But I still have doubts whether this is the right way to solve the git dependency issue because the guys here talk about "bad practice" and "deprecated" regarding dependency_links. But it seems there is no other way of defining dependencies within setup.py.

Also I still did not get the point of Pipfile: Why does one need Pipfile if there is also the possibility to use setup.py for handling dependencies?

[uranusjr]

@irgendsontyp I believe the “modern” way to install dependency via Git is through PEP 508 URL-based dependency lookup. Remove the dependency_links entry, and change the install_requires entry from

install_requires=[
    'pipenv-app',
],

to

install_requires=[
    'pipenv-app @ git+https://github.com/irgendsontyp/pipenv-app',
],

I’ve never actually used it myself, to be honest, but it should work.

[uranusjr]

Donald Stufft (the main person behind all these packaging stuff) published a great blog post on why the Python ecosystem needs two ways to specify dependencies. It was written a few years ago, so he was describing requirements.txt instead of Pipfile, but the idea is the same. The post also describes why exactly dependency_links is bad practice.

https://caremad.io/posts/2013/07/setup-vs-requirement/

[pmlk]

@uranusjr I tried your approach, but I can't get it to work. I suspect it's related to version numbers not being supplied.

install_requires=[
    'PipenvDependency @ git+https://github.com/pmlk/pipenv-dependency.git',
]

$ pip install -e .
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting PipenvDependency@ git+https://github.com/pmlk/pipenv-dependency.git (from PipenvApp==0.0.0)
  Could not find a version that satisfies the requirement PipenvDependency@ git+https://github.com/pmlk/pipenv-dependency.git (from PipenvApp==0.0.0) (from versions: )
No matching distribution found for PipenvDependency@ git+https://github.com/pmlk/pipenv-dependency.git (from PipenvApp==0.0.0)

I tried supplying the required version in several ways:

'PipenvDependency==0.0.0 @ git+https...#egg=PipenvDependency-0.0.0'

which yields a parse error:

$ pip install -e .
    Complete output from command python setup.py egg_info:
    error in PipenvApp setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'@ git+ht'"

Any help would be greatly appreciated!

EDIT: Never mind, it seems it will be supported by pip version 10.

[irgendsontyp]

@uranusjr

I believe the “modern” way to install dependency via Git is through PEP 508 URL-based dependency lookup

That seems to contradict the blog post you mentioned. If I got it right, the post explicitly says that having URLs in setup.py is a bad practice because developers that want to fork and modify one or more of the dependent libraries would also have to modify setup.py in order to remove the URL and thus making it an abstract dependency (only name + version, but no location). This leads me to the assumption that the right way to have dependencies for example on GitHub or somewhere else and not on pypi is to have a requirements.txt (pip) or Pipfile (pipenv) which include the specific locations of the dependencies or use --find-links= to specify the location for the dependencies and have setup.py contain the abstract dependencies, i.e. include package name and version number but omit the location where they can be found.

What do you think?

[uranusjr]

@irgendsontyp You are absolutely correct :)

What I meant was that, if you really want to have your dependency only available via Git, per-dependency URL is the way to go forward. It is still not a good idea (due to the reasons you mentioned), but is better than dependency_links.

The best practice is always to make your dependency available on pypi.org (or pypi.python.org), or, for non-public dependencies (closed-source code), spin up your own private PyPI server, make your package available there instead, and instruct your users to add a custom PyPI mirror to their Pip configuration.