Pipenv: Installation error from requirements.txt with -e

[philfreo]

Still having errors, with pipenv 7.6.9, trying to port a project. Here's a reduced/simplified case showing a reproducible error:

$ cat requirements.txt 
Flask==0.9
-e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=Flask-SQLAlchemy
flask-migrate==1.8.0

$ pipenv install
Creating a virtualenv for this project…
⠋New python executable in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/bin/python2.7
Also creating executable in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
No package provided, installing all dependencies.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:04
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.
Requirement already satisfied: Flask>=0.9 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: SQLAlchemy>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: Werkzeug>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: Jinja2>=2.4 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.9->flask-sqlalchemy)
Installing collected packages: flask-sqlalchemy
  Found existing installation: Flask-SQLAlchemy 2.1

Exception:
Traceback (most recent call last):
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/req/req_install.py", line 703, in uninstall
    '(at %s)' % (link_pointer, self.name, dist.location)
AssertionError: Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/src/flask-sqlalchemy does not match installed location of flask-sqlalchemy (at /Users/philfreo/Sites/foo/pipenv-problem)

  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

[kennethreitz42]

Please provide your pipfile.

[kennethreitz42]

Using this Pipfile, confirmed your bug:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

Flask = "==0.9"
"Flask-SQLAlchemy" = {ref = "1556d1822c46019e518707af0adc034ad4363ab4", git = "ssh://[email protected]/closeio/flask-sqlalchemy.git", editable = true}
"Flask-Migrate" = "==1.8.0"

[kennethreitz42]

fixed the bug you're encountering.

Now you'll see this error message:

Collecting flask==0.9
  Using cached Flask-0.9.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/ly/_zv29c116tjghvp91w46rs9c0000gn/T/pip-build-37ja8tfr/flask/setup.py", line 62
        print "Audit requires PyFlakes installed in your system."
                                                                ^
    SyntaxError: Missing parentheses in call to 'print'

    ----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/ly/_zv29c116tjghvp91w46rs9c0000gn/T/pip-build-37ja8tfr/flask/

[kennethreitz42]

all is working well for me on python2 on master.

[philfreo]

I'm still getting the same error on v7.7.3. Starting point: same requirements.txt above, no Pipfile yet, and I ran pipenv --rm first to get a fresh start.

$ pipenv install
Creating a virtualenv for this project…
⠋New python executable in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/bin/python2.7
Also creating executable in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
No package provided, installing all dependencies.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:05
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.
Requirement already satisfied: Flask>=0.9 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: SQLAlchemy>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: Werkzeug>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: Jinja2>=2.4 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.9->flask-sqlalchemy)
Installing collected packages: flask-sqlalchemy
  Found existing installation: Flask-SQLAlchemy 2.1

Exception:
Traceback (most recent call last):
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/req/req_install.py", line 703, in uninstall
    '(at %s)' % (link_pointer, self.name, dist.location)
AssertionError: Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/src/flask-sqlalchemy does not match installed location of flask-sqlalchemy (at /Users/philfreo/Sites/foo/pipenv-problem)

  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

[kennethreitz42]

looks like you have a problem with your python installation.

[kennethreitz42]

And yes you do have a pipfile.

[kennethreitz42]

try running pipenv install twice.

[philfreo]

I meant that I didn't start with a Pipfile. After running the above, here's what was created:

[pipenv-problem] cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

Flask = "==0.9"
"Flask-SQLAlchemy" = {ref = "1556d1822c46019e518707af0adc034ad4363ab4", git = "ssh://[email protected]/closeio/flask-sqlalchemy.git", editable = true}
"Flask-Migrate" = "==1.8.0"

Running pipenv install multiple times still results in the error above each time.

Regarding my python installation, I'm currently pyenv via brew. I can try getting rid of pyenv?

[kennethreitz42]

everything's working fine for me, with that pipfile (on python 2).

[kennethreitz42]

do a 'pipenv lock' and then 'pipenv install'

[kennethreitz42]

and make sure you have the latest version, please

[kennethreitz42]

there was a bug in the previous versions, causing "flask-sqlalchemy" to be the lockfile twice. that has been fixed.

[kennethreitz42]

i suspect that is the cause of your issue.

[philfreo]

lock completes without error, install does not.

# starting out with a clean slate:
$ rm -rf Flask_SQLAlchemy.egg-info; rm Pipfile Pipfile.lock
$ pipenv --rm
No virtualenv has been created for this project yet!

$ pipenv --version
pipenv, version 7.7.3

$ pipenv lock
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!

$ pipenv install
No package provided, installing all dependencies.
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:00
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.
Requirement already satisfied: Flask>=0.9 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: SQLAlchemy>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: Werkzeug>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: Jinja2>=2.4 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.9->flask-sqlalchemy)
Installing collected packages: flask-sqlalchemy
  Found existing installation: Flask-SQLAlchemy 2.1

Exception:
Traceback (most recent call last):
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/lib/python2.7/site-packages/pip/req/req_install.py", line 703, in uninstall
    '(at %s)' % (link_pointer, self.name, dist.location)
AssertionError: Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-problem-qZknlxgQ/src/flask-sqlalchemy does not match installed location of flask-sqlalchemy (at /Users/philfreo/Sites/foo/pipenv-problem)

[kennethreitz42]

$ pipenv --rm
$ pipenv install

[kennethreitz42]

Why are you installing flask-sqlalchemy from source anyway? any why is it editable?

[philfreo]

That worked! So, it seems like the following steps are _all_ required in this order even in a _fresh_ setup with nothing but the above requirements.txt:

pipenv lock
pipenv --rm 
pipenv install

I'm needed to fork flask-sqlalchemy in order to relax its requirements from Flask >= 0.10 to be 0.9 instead. flask-sqlalchemy (at least for my purposes) actually works fine with Flask 0.9, and I have a large application that would be non-trivial to update from Flask 0.9 to Flask 0.10. I'm only using -e because I haven't noticed a way to specify a specific GitHub Commit SHA otherwise.

If I simply remove the -e prefix from my requirements.txt then I get the following error:

$ pipenv lock
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches Flask==0.9,>=0.10,>=0.9
Tried: 0.1, 0.2, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.5.2, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.9, 0.10, 0.10.1, 0.11, 0.11, 0.11.1, 0.11.1, 0.12, 0.12, 0.12.1, 0.12.1, 0.12.2, 0.12.2

(Note: this is just a reduced requirements.txt that shows the problem; my actual application has a much larger requirements.txt that I'm hoping to convert to pipenv)

[kennethreitz42]

no, we fixed the bug. you just had an old lockfile present, which was causing those steps to be necessary.

glad we fixed your bug!

[philfreo]

No, the error happens even when there's no lockfile present. Look at this:

$ ls -al              
total 8
drwxr-xr-x   4 philfreo  _www   136 Sep 23 17:41 .
drwxr-xr-x  97 philfreo  _www  3298 Sep 23 17:29 ..
drwxr-xr-x   9 philfreo  _www   306 Sep 23 17:41 .git
-rw-r--r--   1 philfreo  _www   151 Sep 23 17:41 requirements.txt

$ cat requirements.txt 
Flask==0.9
-e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=Flask-SQLAlchemy
flask-migrate==1.8.0

$ pipenv --version
pipenv, version 7.7.3

$ pipenv --rm
No virtualenv has been created for this project yet!

$ pipenv install
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
No package provided, installing all dependencies.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:00
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.
Requirement already satisfied: Flask>=0.9 in /Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: SQLAlchemy>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: Werkzeug>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: Jinja2>=2.4 in /Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.9->flask-sqlalchemy)
Installing collected packages: flask-sqlalchemy
  Found existing installation: Flask-SQLAlchemy 2.1

Exception:
Traceback (most recent call last):
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/lib/python2.7/site-packages/pip/req/req_install.py", line 703, in uninstall
    '(at %s)' % (link_pointer, self.name, dist.location)
AssertionError: Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-test2-NLsCCm1N/src/flask-sqlalchemy does not match installed location of flask-sqlalchemy (at /Users/philfreo/Sites/foo/pipenv-test2)

  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

[kennethreitz42]

I will investigate.

[kennethreitz42]

cannot reproduce

[kennethreitz42]

idk how you got a ./src folder in there, but it shouldn't be there.

[kennethreitz42]

it seems to be what's causing all the trouble.

[kennethreitz42]

try this in a fresh directory

[philfreo]

[_es] mkdir pipenv-test3
[_es] cd pipenv-test3
[pipenv-test3] cp ../pipenv-test2/requirements.txt .

[pipenv-test3] cat requirements.txt 
Flask==0.9
-e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=Flask-SQLAlchemy
flask-migrate==1.8.0

[pipenv-test3] git init .
Initialized empty Git repository in /Users/philfreo/Sites/_es/pipenv-test3/.git/

[pipenv-test3] ls -al
total 8
drwxr-xr-x   4 philfreo  _www   136 Sep 23 17:51 .
drwxr-xr-x  98 philfreo  _www  3332 Sep 23 17:51 ..
drwxr-xr-x   9 philfreo  _www   306 Sep 23 17:51 .git
-rw-r--r--   1 philfreo  _www   151 Sep 23 17:51 requirements.txt

[pipenv-test3] pipenv --version
pipenv, version 7.7.3

[pipenv-test3] pipenv install
Creating a virtualenv for this project…
⠋New python executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/bin/python2.7
Also creating executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
No package provided, installing all dependencies.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:05
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.
Requirement already satisfied: Flask>=0.9 in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: SQLAlchemy>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: Werkzeug>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: Jinja2>=2.4 in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.9->flask-sqlalchemy)
Installing collected packages: flask-sqlalchemy
  Found existing installation: Flask-SQLAlchemy 2.1

Exception:
Traceback (most recent call last):
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/lib/python2.7/site-packages/pip/req/req_install.py", line 703, in uninstall
    '(at %s)' % (link_pointer, self.name, dist.location)
AssertionError: Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-test3-FjinwMF_/src/flask-sqlalchemy does not match installed location of flask-sqlalchemy (at /Users/philfreo/Sites/_es/pipenv-test3)

  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

[philfreo]

Not sure if it helps, but after doing the above I also end up with the following:

[pipenv-test3] ls -l
total 24
drwxr-xr-x  8 philfreo  _www   272 Sep 23 17:52 Flask_SQLAlchemy.egg-info
-rw-r--r--  1 philfreo  _www   304 Sep 23 17:52 Pipfile
-rw-r--r--  1 philfreo  _www  3940 Sep 23 17:52 Pipfile.lock
-rw-r--r--  1 philfreo  _www   151 Sep 23 17:51 requirements.txt

I would have expected the Flask_SQLAlchemy.egg-info dir to be inside the virtualenv. That's how it worked with my traditional pip/virtualenv/requirements.txt setup, though maybe this is intentional in pipenv for -e.

[kennethreitz42]

that is to be expected.

[kennethreitz42]

though we can improve that maybe

[philfreo]

And to be clear, I don't have any reason to be using -e other than "There doesn't seem to be another working way to install a specific github commit SHA of a forked package"

[kennethreitz42]

why don't you show me your full pipfile

[kennethreitz42]

and i'll show you what i'd do

[philfreo]

Here's what's pipenv generates from the requirements.txt in the OP here.

[pipenv-test3] cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

Flask = "==0.9"
"Flask-SQLAlchemy" = {ref = "1556d1822c46019e518707af0adc034ad4363ab4", git = "ssh://[email protected]/closeio/flask-sqlalchemy.git", editable = true}
"Flask-Migrate" = "==1.8.0"

[kennethreitz42]

I meant your "full" one, the production one

[kennethreitz42]

Here:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

Flask = "*"
"Flask-SQLAlchemy" = {ref = "1556d1822c46019e518707af0adc034ad4363ab4", git = "ssh://[email protected]/closeio/flask-sqlalchemy.git"}
"Flask-Migrate" = "==1.8.0"

[kennethreitz42]

you probably want editable = true though, as then all the sub-dependencies get pulled into the lockfile.

[kennethreitz42]

otherwise, they do not.

[philfreo]

Our application has a hard requirement on Flask 0.9 though (non-trivial upgrade for us).

Production requirements.txt is a 201 lines long with all packages frozen to a specific version. Unfortunately I don't think I can share the whole thing publicly for security reasons. I created this as a simplified case – it's a subset of production file – of the problem we ran into with the full app.

[kennethreitz42]

ah, well if i can be of any help when that time comes, let me know.

[kennethreitz42]

to be clear: editable == sub-deps get resolved/locked.

[kennethreitz42]

not editable == everything works as expected, but no dep resolution, no sub-dep locking.

[philfreo]

See the problem here is Flask-Sqlalchemy requires Flask >=0.9.

Yeah I know -- but since there isn't any _actual_ issue for us running Flask-Sqlalchemy with Flask 0.9 (basically, their setup.py has a stricter requirement than necessary), shouldn't my fork work?

[kennethreitz42]

You've managed to magically avoid a dependency mismatch with this specific set of dependencies, somehow. Must be an implementation detail of the way pip-tool's resolver is written. You got lucky, basically :)

[philfreo]

Put another way: it's not unusual for us to fork a pypi public package to make some tweak -- some of them completely unrelated to dependencies. Seems like this only way to install those forks is with -e? This seems like one example of a larger pattern that I would think should work?

[kennethreitz42]

do you run your own pypi index?

[philfreo]

No

[kennethreitz42]

then yes.

[kennethreitz42]

otherwise, we have another solution:

https://pipenv.readthedocs.io/en/latest/advanced.html#specifying-package-indexes

[kennethreitz42]

-e is the way to go. i'll see about installing the egg-links into the venv instead of littering them in the project directory.

[philfreo]

Ok. I don't understand this stuff super well, but it still seems like a bug that I must do lock, then --rm, then install for this requirements.txt. It's a fine workaround to my problem (thanks for helping figure that out) but still seems like a pipenv bug or usability issue to me?

I'll see if I can use this workaround on my production app too.

[kennethreitz42]

okay i got a patch in for installing the .egg-info's into venv/src/

[kennethreitz42]

I think your bug is becuase you have an egg-info laying around somewhere that should be there — to be clear, using this pipfile in a fresh directory works great for me.

[kennethreitz42]

https://github.com/kennethreitz/pipenv/commit/e20a58db70bd4050a1c21378400fc2f1a76aefc5

[philfreo]

I showed an ls -al above that shows there's no egg-info in the project directory (and used pipenv --rm). Or are you saying I might have one somewhere else, like my system wide python directory?

[kennethreitz42]

yes.

[kennethreitz42]

this is why i recommend installing pipenv with pipsi https://pipenv.readthedocs.io/en/latest/advanced.html#fancy-installation-of-pipenv

[kennethreitz42]

okay i just cut release 7.7.4, which contains that patch. see if that improves things for ya.

[philfreo]

Thanks. I'll try to cleanup everything and will give that a shot.

Meanwhile, I get a different issue when trying things on my production app. I opened #673 with details.

[kennethreitz42]

thanks for the patience

[philfreo]

With 7.7.4 pipenv lock still installs Flask_SQLAlchemy.egg-info in the project directory. Is that expected?

However once I have a lockfile, deleting Flask_SQLAlchemy.egg-info and then running pipenv install does _not_ install it in the project directory (installs in the virtualenv instead), so that's nice!

[kennethreitz42]

may have to pass additional args to pip tools

[philfreo]

I uninstalled pipenv from pip, installed pipsi and installed pipenv via pipsi (following this), and tried it again from scratch.

Even then, I still had to do the lock/--rm/install dance to get it to work. From scratch:

[_es] mkdir pipenv-test4
[_es] cd pipenv-test4
[pipenv-test4] cp ../pipenv-test3/requirements.txt .

[pipenv-test4] which python
/Users/philfreo/.pyenv/shims/python

[pipenv-test4] python --version
Python 2.7.12

[pipenv-test4] pipenv --version
pipenv, version 7.7.4

[pipenv-test4] which pipenv
/Users/philfreo/.local/bin/pipenv

[pipenv-test4] pipsi --version
pipsi, version 0.9, python /Users/philfreo/.local/venvs/pipsi/bin/python2.7

[pipenv-test4] pipenv install
Creating a virtualenv for this project…
⠋New python executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/bin/python2.7
Also creating executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
No package provided, installing all dependencies.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:04
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.
Requirement already satisfied: Flask>=0.9 in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: SQLAlchemy>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages (from flask-sqlalchemy)
Requirement already satisfied: Werkzeug>=0.7 in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: Jinja2>=2.4 in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages (from Flask>=0.9->flask-sqlalchemy)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.9->flask-sqlalchemy)
Installing collected packages: flask-sqlalchemy
  Found existing installation: Flask-SQLAlchemy 2.1

Exception:
Traceback (most recent call last):
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/lib/python2.7/site-packages/pip/req/req_install.py", line 703, in uninstall
    '(at %s)' % (link_pointer, self.name, dist.location)
AssertionError: Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/src/flask-sqlalchemy does not match installed location of flask-sqlalchemy (at /Users/philfreo/Sites/_es/pipenv-test4)

  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00


[pipenv-test4] ls -l
total 24
drwxr-xr-x  8 philfreo  _www   272 Sep 23 19:05 Flask_SQLAlchemy.egg-info
-rw-r--r--  1 philfreo  _www   304 Sep 23 19:05 Pipfile
-rw-r--r--  1 philfreo  _www  3940 Sep 23 19:05 Pipfile.lock
-rw-r--r--  1 philfreo  _www   151 Sep 23 19:04 requirements.txt

[pipenv-test4] rm -rf Flask_SQLAlchemy.egg-info 

[pipenv-test4] pipenv --rm
Removing virtualenv (/Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc)…

[pipenv-test4] pipenv install
Creating a virtualenv for this project…
⠋New python executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/bin/python2.7
Also creating executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc
No package provided, installing all dependencies.
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:04
To activate this project's virtualenv, run the following:
 $ pipenv shell

[kennethreitz42]

Egg-link /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/src/flask-sqlalchemy where is that coming from?

[philfreo]

I don't know... I tried to show how I'm starting fresh. Maybe related to these lines:

Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:04
An error occured while installing -e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy! Will try again.
Installing initially–failed dependencies…
Obtaining flask-sqlalchemy from git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=flask-sqlalchemy
  Skipping because already up-to-date.

[philfreo]

What is the "error"/"failure" that causes it to retry? And maybe that's where the path you're asking about gets installed?

[kennethreitz42]

Wonder why it fails the first time.

[kennethreitz42]

I cannot reproduce this on my machine.

[kennethreitz42]

okay, i think i found the source of the problem.

Try v7.7.5.

[kennethreitz42]

and delete any egg-info you see first

[philfreo]

It works! 🎆 👍 🍻 🙌

[pipenv-test4] pipenv --rm
No virtualenv has been created for this project yet!

[pipenv-test4] pipenv --version
pipenv, version 7.7.5

[pipenv-test4] ls -al
total 8
drwxr-xr-x   4 philfreo  _www   136 Sep 23 19:54 .
drwxr-xr-x  99 philfreo  _www  3366 Sep 23 19:04 ..
drwxr-xr-x   9 philfreo  _www   306 Sep 23 19:54 .git
-rw-r--r--   1 philfreo  _www   151 Sep 23 19:04 requirements.txt

[pipenv-test4] pipenv install
Creating a virtualenv for this project…
⠋New python executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/bin/python2.7
Also creating executable in /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/philfreo/.local/share/virtualenvs/pipenv-test4-KJKuC2dc
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
No package provided, installing all dependencies.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Flask==0.9
Flask-Migrate==1.8.0
-e git+ssh://[email protected]/closeio/flask-sqlalchemy.git@1556d1822c46019e518707af0adc034ad4363ab4#egg=Flask-SQLAlchemy
No handlers could be found for logger "pip.vcs.git"
Updated Pipfile.lock!
Installing dependencies from Pipfile.lock…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 13/13 — 00:00:04
To activate this project's virtualenv, run the following:
 $ pipenv shell

[philfreo]

Thanks so much for all the help / fixes so far!! I'll see what kind of progress I can make on converting the production app now.

[kennethreitz42]

✨🍰✨