Pipenv: Pipenv doesn't install dependencies of top-level dependencies from pre-existing Pipfile

[tribals]

Here is the story: I'm migrating from old requirements.txt to a brand new Pipfile. First time I run pipenv install, it said me that there is requirements.txt and it will convert it. OK, I thought. Next, it installed dependencies.

Then I tried to play with my project, run some code, and it fails due to missing imports. "It is strange", I thought. Then I run pipenv graph and find that some of dependencies of direct dependencies of my project was not installed.

I did a bunch of experiments combining different options, pythons (2.7, 3.6),Pipfile's (and .lock's) and still can't figure out why pipenv doesn't install all the dependencies.

Then I do stupid thing: I grabbed list of top-level deps, removed Pipfile{,.lock} and requirements.txt and installed all of them step-by-step with pipenv install <package>. Funny, but it worked.

Describe your environment

1. Solus
2. Python version: Python 2.7.14
3. Pipenv version: pipenv, version 10.1.0

Expected result

I expect that when I install deps from pre-existing Pipfile, pipenv install all of them, including deps of top-level deps.

Actual result

Some of deps of top-level deps are not installed at all.

Steps to replicate

Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

cryptography = "*"
APScheduler = "==3.0.5"
"pathlib2" = "*"


[dev-packages]



[requires]

python_version = "2.7"

Then:

$ pipenv install --two
$ pipenv graph | egrep 'installed: \?'

Trying to import:

$ python -m pathlib2
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "<venv>/lib/python2.7/site-packages/pathlib2.py", line 45, in <module>
    from scandir import scandir as os_scandir
ImportError: No module named scandir

[tribals]

Also, it seems like even if I manually re-installed package to fix dependencies problem, such originally missing deps _does not locked in Pipfile.lock_. Which means that the main goal of pipenv - reproducible, deterministic reestablishment of application environment - not achieved at all.

$ pipenv install
$ pipenv graph | egrep 'installed: \?'
  - futures [required: Any, installed: ?]
  - enum34 [required: Any, installed: ?]
  - ipaddress [required: Any, installed: ?]
  - scandir [required: Any, installed: ?]
$ pipenv uninstall apscheduler; pipenv install apscheduler
$ pipenv graph | egrep 'installed: \?' # seems like now dependency is installed
  - enum34 [required: Any, installed: ?]
  - ipaddress [required: Any, installed: ?]
  - scandir [required: Any, installed: ?]
$ grep futures Pipfile.lock || echo ALARM! Dependency is not locked!
ALARM! Dependency is not locked!

[tribals]

Why @kennethreitz you closed this?

[abulka]

I'm also getting this problem - as described above, with many sub-dependencies not being installed, with ? symbols in the graph. My environment is brew python 3.6.4, pipenv, version 11.3.1, mac high sierra. Here are some samples from pipenv graph

django-allauth==0.35.0
  - Django [required: >=1.11, installed: 1.11.8]
    - pytz [required: Any, installed: 2018.3]
  - python3-openid [required: >=3.0.8, installed: ?]
  - requests [required: Any, installed: 2.18.4]
    - certifi [required: >=2017.4.17, installed: ?]
    - chardet [required: >=3.0.2,<3.1.0, installed: ?]
    - idna [required: >=2.5,<2.7, installed: ?]
    - urllib3 [required: >=1.21.1,<1.23, installed: ?]
  - requests-oauthlib [required: >=0.3.0, installed: ?]
django-extensions==2.0.5
  - six [required: >=1.2, installed: 1.10.0]
...
...
maya==0.3.4
  - dateparser [required: Any, installed: ?]
  - humanize [required: Any, installed: ?]
  - pendulum [required: Any, installed: ?]
  - pytz [required: Any, installed: 2018.3]
  - ruamel.yaml [required: Any, installed: ?]
  - tzlocal [required: Any, installed: ?]
mock==2.0.0
  - pbr [required: >=0.11, installed: ?]
  - six [required: >=1.9, installed: 1.10.0]

notice maya for example has missing dependencies. But if I run pipenv install maya they are fixed:

maya==0.3.4
  - dateparser [required: Any, installed: 0.7.0]
    - python-dateutil [required: Any, installed: 2.6.1]
      - six [required: >=1.5, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.3]
    - regex [required: Any, installed: 2018.2.21]
    - tzlocal [required: Any, installed: 1.5.1]
      - pytz [required: Any, installed: 2018.3]
  - humanize [required: Any, installed: 0.5.1]
  - pendulum [required: Any, installed: 1.4.2]
    - python-dateutil [required: Any, installed: 2.6.1]
      - six [required: >=1.5, installed: 1.11.0]
    - pytzdata [required: Any, installed: 2018.3]
    - tzlocal [required: Any, installed: 1.5.1]
      - pytz [required: Any, installed: 2018.3]
  - pytz [required: Any, installed: 2018.3]
  - ruamel.yaml [required: Any, installed: 0.15.35]
  - tzlocal [required: Any, installed: 1.5.1]
    - pytz [required: Any, installed: 2018.3]

but if I remove and recreate the pipenv virtual environment, and run pipenv install, I get the same problem again.

If I run the good ole pip install -r requirements.txt within the virtual environment, all the missing dependencies are installed and the project runs without problem.

[abulka]

I updated to the latest version 11.5.2 (not sure why brew gave me an older version yesterday).

I've also become aware that people can get tricked if there is an existing Pipfile.lock with missing dependencies - then of course you are going to get missing dependencies when you pipenv install because pipenv install only looks at the lockfile.

So I have been careful to pipenv update which should fix up (add any needed) dependencies in the lockfile, yet I am still getting missing dependencies from the lockfile, and thus from the pipenv graph. I totally reinstalled pipenv using brew, pipenv --rm to remove the pipenv virtual env and rebooted, treid again, but am still getting the missing dependencies. So I ran update with the -v verbose option and saw:

...
...
  found candidate stripe==1.79.1 (constraint was <any>)
  found candidate treelib==1.5.1 (constraint was <any>)
  found candidate whitenoise==2.0.6 (constraint was ==2.0.6)

Finding secondary dependencies:
  sphinx-js==2.3.1          requires -
  sphinx==1.7.1             requires -
  requests-mock==1.4.0      requires -
  furl==1.0.1               requires -
  jsonpickle==0.9.6         requires -
  pytz==2018.3              requires -
  psycopg2==2.6.1           requires -
  dj-database-url==0.4.0    requires -
  attrs==17.4.0             requires -
  maya==0.3.4               requires -
  mock==2.0.0               requires -
  recommonmark==0.4.0       requires -
  sphinx-rtd-theme==0.1.9   requires -
  django-allauth==0.35.0    requires -
  treelib==1.5.1            requires -
  beautifulsoup4==4.6.0     requires -
  pyyaml==3.12              requires -
  django==1.11.8            requires -
  gunicorn==19.4.5          requires -
  django-extensions==2.0.5  requires -
  selenium==3.10.0          requires -
  django-sendgrid-v5==0.6.87 requires -
  whitenoise==2.0.6         requires -
  plantweb==1.1.0           requires -
  requests==2.18.4          requires -
  stripe==1.79.1            requires -
  astpretty==1.2.1          requires -
  sphinxcontrib-plantuml==0.6 requires -
------------------------------------------------------------
Result of round 1: stable, done
Warning: Error generating hash for mock
Warning: Error generating hash for recommonmark
Warning: Error generating hash for sphinx-rtd-theme
Warning: Error generating hash for sphinx-js
...
...

The secondary dependencies are coming up blank? On my other mac machine, the dependencies detect what's needed and go through various 'rounds' and it all works ok. The machine which has problems is a solid Mac machine with python 2 and python 3 installed using brew. No sure how to debug this or whether this deserves another open issue (apologies if I've hijacked the OP, but it seems related).

[codeb2cc]

d5b9b36713826746d69cde6d9256d33a528881f0

11.4.0:

• Stability.
• Don't install dependencies straight-away with pipenv–install — rely on the underlying pipenv sync architecture to pick up dependencies.
• Warn (abord) if requested update package is not in Pipfile.
• Don't configure the Pipfile for keep_outdated when update is used.

Temporary solution: use 11.3.3

[techalchemy]

@abulka if you’re having trouble with hash generation definitely open a new issue with the output of python -m pipenv.help and pipenv-resolver —debug —verbose —clear <whatever base package is causing the problem>

[kennethreitz42]

FYI I'm reverting that change.

[abulka]

@techalchemy Not having much luck with the commands you suggested - perhaps that is a clue?

$ python -m pipenv.help
/usr/local/opt/python@2/bin/python2.7: No module named pipenv

$ python3 -m pipenv.help
/usr/local/bin/python3: Error while finding module specification for 'pipenv.help' (ModuleNotFoundError: No module named 'pipenv')

$ pipenv-resolver —debug —verbose —clear requests
-bash: pipenv-resolver: command not found

I definately have pipenv installed ok using brew

$ which pipenv
/usr/local/bin/pipenv

$ pipenv --version
pipenv, version 11.5.2

$ brew list pipenv
/usr/local/Cellar/pipenv/11.5.2/bin/pipenv
/usr/local/Cellar/pipenv/11.5.2/libexec/bin/ (19 files)
/usr/local/Cellar/pipenv/11.5.2/libexec/lib/ (1520 files)
/usr/local/Cellar/pipenv/11.5.2/libexec/tools/ (3 files)
/usr/local/Cellar/pipenv/11.5.2/libexec/ (2 files)

and brew python2 and python3 installed

$ brew list python
/usr/local/Cellar/python/3.6.4_4/bin/2to3
/usr/local/Cellar/python/3.6.4_4/bin/2to3-3.6
/usr/local/Cellar/python/3.6.4_4/bin/idle3
/usr/local/Cellar/python/3.6.4_4/bin/idle3.6
/usr/local/Cellar/python/3.6.4_4/bin/pydoc3
/usr/local/Cellar/python/3.6.4_4/bin/pydoc3.6
/usr/local/Cellar/python/3.6.4_4/bin/python3
/usr/local/Cellar/python/3.6.4_4/bin/python3-config
/usr/local/Cellar/python/3.6.4_4/bin/python3.6
/usr/local/Cellar/python/3.6.4_4/bin/python3.6-config
/usr/local/Cellar/python/3.6.4_4/bin/python3.6m
/usr/local/Cellar/python/3.6.4_4/bin/python3.6m-config
/usr/local/Cellar/python/3.6.4_4/bin/pyvenv
/usr/local/Cellar/python/3.6.4_4/bin/pyvenv-3.6
/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/ (7050 files)
/usr/local/Cellar/python/3.6.4_4/IDLE 3.app/Contents/ (8 files)
/usr/local/Cellar/python/3.6.4_4/lib/pkgconfig/ (3 files)
/usr/local/Cellar/python/3.6.4_4/libexec/bin/ (4 files)
/usr/local/Cellar/python/3.6.4_4/libexec/pip/ (291 files)
/usr/local/Cellar/python/3.6.4_4/libexec/setuptools/ (172 files)
/usr/local/Cellar/python/3.6.4_4/libexec/wheel/ (33 files)
/usr/local/Cellar/python/3.6.4_4/Python Launcher 3.app/Contents/ (16 files)
/usr/local/Cellar/python/3.6.4_4/share/doc/ (1002 files)
/usr/local/Cellar/python/3.6.4_4/share/man/ (2 files)
Andys-MacBook-Air:proj_6 Andy$ brew list python2
/usr/local/Cellar/python@2/2.7.14_3/bin/2to3-2
/usr/local/Cellar/python@2/2.7.14_3/bin/2to3-2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/easy_install
/usr/local/Cellar/python@2/2.7.14_3/bin/easy_install-2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/idle
/usr/local/Cellar/python@2/2.7.14_3/bin/idle2
/usr/local/Cellar/python@2/2.7.14_3/bin/idle2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/pip
/usr/local/Cellar/python@2/2.7.14_3/bin/pip2
/usr/local/Cellar/python@2/2.7.14_3/bin/pip2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/pydoc
/usr/local/Cellar/python@2/2.7.14_3/bin/pydoc2
/usr/local/Cellar/python@2/2.7.14_3/bin/pydoc2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/python
/usr/local/Cellar/python@2/2.7.14_3/bin/python-config
/usr/local/Cellar/python@2/2.7.14_3/bin/python2
/usr/local/Cellar/python@2/2.7.14_3/bin/python2-config
/usr/local/Cellar/python@2/2.7.14_3/bin/python2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/python2.7-config
/usr/local/Cellar/python@2/2.7.14_3/bin/pythonw
/usr/local/Cellar/python@2/2.7.14_3/bin/pythonw2
/usr/local/Cellar/python@2/2.7.14_3/bin/pythonw2.7
/usr/local/Cellar/python@2/2.7.14_3/bin/smtpd.py
/usr/local/Cellar/python@2/2.7.14_3/bin/smtpd2.7.py
/usr/local/Cellar/python@2/2.7.14_3/bin/smtpd2.py
/usr/local/Cellar/python@2/2.7.14_3/bin/wheel
/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/ (5079 files)
/usr/local/Cellar/python@2/2.7.14_3/IDLE.app/Contents/ (8 files)
/usr/local/Cellar/python@2/2.7.14_3/lib/pkgconfig/ (3 files)
/usr/local/Cellar/python@2/2.7.14_3/libexec/pip/ (540 files)
/usr/local/Cellar/python@2/2.7.14_3/libexec/setuptools/ (306 files)
/usr/local/Cellar/python@2/2.7.14_3/libexec/wheel/ (52 files)
/usr/local/Cellar/python@2/2.7.14_3/Python Launcher.app/Contents/ (17 files)
/usr/local/Cellar/python@2/2.7.14_3/share/doc/ (1024 files)
/usr/local/Cellar/python@2/2.7.14_3/share/man/ (3 files)
/usr/local/Cellar/python@2/2.7.14_3/share/python@2/ (315 files)

and my /usr/local/bin is

$ ll /usr/local/bin/python*
lrwxr-xr-x  1 Andy  admin  38 11 Mar 21:49 /usr/local/bin/python -> ../Cellar/python@2/2.7.14_3/bin/python
lrwxr-xr-x  1 Andy  admin  45 11 Mar 21:49 /usr/local/bin/python-config -> ../Cellar/python@2/2.7.14_3/bin/python-config
lrwxr-xr-x  1 Andy  admin  39 11 Mar 21:49 /usr/local/bin/python2 -> ../Cellar/python@2/2.7.14_3/bin/python2
lrwxr-xr-x  1 Andy  admin  46 11 Mar 21:49 /usr/local/bin/python2-config -> ../Cellar/python@2/2.7.14_3/bin/python2-config
lrwxr-xr-x  1 Andy  admin  41 11 Mar 21:49 /usr/local/bin/python2.7 -> ../Cellar/python@2/2.7.14_3/bin/python2.7
lrwxr-xr-x  1 Andy  admin  48 11 Mar 21:49 /usr/local/bin/python2.7-config -> ../Cellar/python@2/2.7.14_3/bin/python2.7-config
lrwxr-xr-x  1 Andy  admin  36 11 Mar 22:16 /usr/local/bin/python3 -> ../Cellar/python/3.6.4_4/bin/python3
lrwxr-xr-x  1 Andy  admin  43 11 Mar 22:16 /usr/local/bin/python3-config -> ../Cellar/python/3.6.4_4/bin/python3-config
lrwxr-xr-x  1 Andy  admin  38 11 Mar 22:16 /usr/local/bin/python3.6 -> ../Cellar/python/3.6.4_4/bin/python3.6
lrwxr-xr-x  1 Andy  admin  45 11 Mar 22:16 /usr/local/bin/python3.6-config -> ../Cellar/python/3.6.4_4/bin/python3.6-config
lrwxr-xr-x  1 Andy  admin  39 11 Mar 22:16 /usr/local/bin/python3.6m -> ../Cellar/python/3.6.4_4/bin/python3.6m
lrwxr-xr-x  1 Andy  admin  46 11 Mar 22:16 /usr/local/bin/python3.6m-config -> ../Cellar/python/3.6.4_4/bin/python3.6m-config
lrwxr-xr-x  1 Andy  admin  39 11 Mar 21:49 /usr/local/bin/pythonw -> ../Cellar/python@2/2.7.14_3/bin/pythonw
lrwxr-xr-x  1 Andy  admin  40 11 Mar 21:49 /usr/local/bin/pythonw2 -> ../Cellar/python@2/2.7.14_3/bin/pythonw2
lrwxr-xr-x  1 Andy  admin  42 11 Mar 21:49 /usr/local/bin/pythonw2.7 -> ../Cellar/python@2/2.7.14_3/bin/pythonw2.7

and pipenv itself which I installed via brew, seems to have its Cellar...libexec/bin as follows, just in case that is relevant:

$ ll /usr/local/Cellar/pipenv/11.5.2/libexec/bin/
total 160
-rw-r--r--  1 Andy  admin   2098 12 Mar 16:22 activate
-rw-r--r--  1 Andy  admin   1040 12 Mar 16:22 activate.csh
-rw-r--r--  1 Andy  admin   2194 12 Mar 16:22 activate.fish
-rw-r--r--  1 Andy  admin   1137 12 Mar 05:31 activate_this.py
-rwxr-xr-x  1 Andy  admin    271 12 Mar 16:22 easy_install
-rwxr-xr-x  1 Andy  admin    271 12 Mar 16:22 easy_install-3.6
-rwxr-xr-x  1 Andy  admin    417 12 Mar 16:22 pewtwo
-rwxr-xr-x  1 Andy  admin    243 12 Mar 16:22 pip
-rwxr-xr-x  1 Andy  admin    243 12 Mar 16:22 pip3
-rwxr-xr-x  1 Andy  admin    243 12 Mar 16:22 pip3.6
-rwxr-xr-x  1 Andy  admin    417 12 Mar 16:22 pipenv
-rwxr-xr-x  1 Andy  admin    435 12 Mar 16:22 pipenv-resolver
lrwxr-xr-x  1 Andy  admin      9 12 Mar 05:31 python -> python3.6
-rwxr-xr-x  1 Andy  admin   2357 12 Mar 16:22 python-config
lrwxr-xr-x  1 Andy  admin      9 12 Mar 05:31 python3 -> python3.6
-rwxr-xr-x  1 Andy  admin  13068 12 Mar 05:31 python3.6
-rwxr-xr-x  1 Andy  admin    437 12 Mar 16:22 virtualenv
-rwxr-xr-x  1 Andy  admin    464 12 Mar 16:22 virtualenv-clone
-rwxr-xr-x  1 Andy  admin    250 12 Mar 16:22 wheel

I can reproduce the problem with various Pipfiles, for example

$ cat Pipfile
[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

requests = "*"
furl = "*"
maya = "*"

[dev-packages]

[requires]

python_version = "3.6"

after a pipenv install I get

$ pipenv graph
furl==1.0.1
  - orderedmultidict [required: >=0.7.8, installed: ?]
  - six [required: >=1.8.0, installed: 1.10.0]
maya==0.3.4
  - dateparser [required: Any, installed: ?]
  - humanize [required: Any, installed: ?]
  - pendulum [required: Any, installed: ?]
  - pytz [required: Any, installed: ?]
  - ruamel.yaml [required: Any, installed: ?]
  - tzlocal [required: Any, installed: ?]
requests==2.18.4
  - certifi [required: >=2017.4.17, installed: ?]
  - chardet [required: <3.1.0,>=3.0.2, installed: ?]
  - idna [required: >=2.5,<2.7, installed: ?]
  - urllib3 [required: <1.23,>=1

Notice the sub-dependencies with ?

[kennethreitz42]

homebrew does not have the latest version of pipenv.