Pipenv: Markers now override unconditional dependencies resulting in a broken environment

[jacebrowning]

This is a carryover from a discussion on https://github.com/pypa/pipenv/issues/1659 and the simplest way I could find to replicate this behavior.

With this Pipfile on macOS and pipenv 11.8.0:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]

python_version = "3.6"

[packages]

arrow = "*"
coverage-space = "*"
faker = "*"
fire = "*"
freezegun = "*"
inflection = "*"
jupyter = "*"
pillow = "~=4.3"
pycodestyle = "~=2.3.1"
pylint = "~=1.7.5"
pytest = "~= 3.3"
pytest-cov = "*"
pytest-describe = "*"
pytest-expecter = "~= 1.1"
pytest-html = "*"
pytest-json-report = "~= 0.6"
pytest-mock = "*"
pytest-ordering = "*"
pytest-random = "*"
splinter = "*"
tldextract = "*"
watchdog = "*"

$ pipenv install

Creating a virtualenv for this project…
Using /Users/Browning/.pyenv/versions/3.6.3/bin/python3.6m (3.6.3) to create virtualenv…
...
Installing dependencies from Pipfile.lock (1ba60a)…
Ignoring colorama: markers 'sys_platform == "win32"' don't match your environment
Ignoring funcsigs: markers 'python_version < "3.0"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 88/88 — 00:00:26
To activate this project's virtualenv, run the following:
 $ pipenv shell

which results in a broken environment:

$ pipenv run coverage.space
Traceback (most recent call last):
  File "/Users/Browning/.local/share/virtualenvs/colorama-demo-Q5bNGJGk/bin/coverage.space", line 7, in <module>
    from coveragespace.cli import main
  File "/Users/Browning/.local/share/virtualenvs/colorama-demo-Q5bNGJGk/lib/python3.6/site-packages/coveragespace/cli.py", line 24, in <module>
    import colorama
ModuleNotFoundError: No module named 'colorama'

despite coverage-space directly requiring it:

$ pipenv graph --reverse
...
colorama==0.3.7
  - coverage-space==1.0 [requires: colorama~=0.3]
...

So it seems that one of the other packages only requires colorama on Windows and their markers are overriding the fact that another package depends on colorama unconditionally.

[kennethreitz42]

what happens if you put colorama in your pipfile?

[jacebrowning]

After rm Pipfile.lock && pipenv --rm, adding colorama = "*" to my Pipfile still results in:

Ignoring colorama: markers 'sys_platform == "win32"' don't match your environment

[techalchemy]

@jacebrowning if you add it unpinned, the order will likely matter, put it on top and pipenv lock --clear --verbose & if it still doesn't work provide the output of that + pipenv-resolver colorama --debug -- I'm having trouble replicating this but it seems tricky

[kennethreitz42]

add:

colorama = {version = "*", sys_platform="!='fake'"}

to your pipfile

[jacebrowning]

BTW here's an even smaller test case:

pytest = "==3.4.2"
colorama = "==0.3.9"

$ pipenv install; pipenv run python -c "import colorama"

[kennethreitz42]

try adding

[kennethreitz42]

colorama = {version = "*", sys_platform="!='fake'"}

to your pipfile

[jacebrowning]

Bingo. That results in:

"colorama": {
            "hashes": [
                "sha256:463f8483208e921368c9f306094eb6f725c6ca42b0f97e313cb5d5512459feda",
                "sha256:48eb22f4f8461b1df5734a074b57042430fb06e1d61bd1e11b078c0fe6d7a1f1"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'fake'",
            "version": "==0.3.9"
        },

which allows colorama to be installed.

[kennethreitz42]

Enjoy :)

[jacebrowning]

So it seems like an "empty" markers needs to take precedence over markers.

[kennethreitz42]

send a PR :)

[jacebrowning]

It used to work like this. 😉 Can you point me to where this logic lives?

[kennethreitz42]

it's everwhere.

[kennethreitz42]

literally… everywhere

[kennethreitz42]

resolvers are hard.

[techalchemy]

fyi I did run your test case @jacebrowning and it works fine for me on ubuntu 17.10:

 /t/test  cat Pipfile
[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]
pytest = "==3.4.2"
colorama = "==0.3.9"

[dev-packages]

[requires]

python_version = "3.6"

 /t/test  pipenv install
Creating a virtualenv for this project…
Using /home/hawk/.pyenv/versions/3.6.3/bin/python3.6m (3.6.3) to create virtualenv…
⠋Running virtualenv with interpreter /home/hawk/.pyenv/versions/3.6.3/bin/python3.6m
Using base prefix '/home/hawk/.pyenv/versions/3.6.3'
New python executable in /home/hawk/.virtualenvs/test-hxkKlP5o/bin/python3.6m
Also creating executable in /home/hawk/.virtualenvs/test-hxkKlP5o/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/hawk/.virtualenvs/test-hxkKlP5o
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6b2b85)!
Installing dependencies from Pipfile.lock (6b2b85)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell

 /t/test  pipenv run python -c "import colorama"
 /t/test   

[kennethreitz42]

yeah, works fine for me on macos too

[jacebrowning]

Weird. Why is mine different? I have Homebrew-installed pipenv if that matters.

➜  demo2 pipenv --version
pipenv, version 11.8.0

➜  demo2 cat Pipfile
[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]

python_version = "3.6"

[packages]

pytest = "==3.4.2"
colorama = "==0.3.9"

➜  demo2 pipenv install
Creating a virtualenv for this project…
Using /Users/Browning/.pyenv/versions/3.6.3/bin/python3.6m (3.6.3) to create virtualenv…
⠋Running virtualenv with interpreter /Users/Browning/.pyenv/versions/3.6.3/bin/python3.6m
Using base prefix '/Users/Browning/.pyenv/versions/3.6.3'
New python executable in /Users/Browning/.local/share/virtualenvs/demo2-VO8n41Q8/bin/python3.6m
Also creating executable in /Users/Browning/.local/share/virtualenvs/demo2-VO8n41Q8/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/Browning/.local/share/virtualenvs/demo2-VO8n41Q8
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6b2b85)!
Installing dependencies from Pipfile.lock (6b2b85)…
Ignoring colorama: markers 'sys_platform == "win32"' don't match your environment
Ignoring funcsigs: markers 'python_version < "3.0"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 7/7 — 00:00:02
To activate this project's virtualenv, run the following:
 $ pipenv shell

➜  demo2 pipenv run python -c "import colorama"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'colorama'

[jacebrowning]

@techalchemy Shouldn't your example show this line at least?

Ignoring funcsigs: markers 'python_version < "3.0"' don't match your environment

[techalchemy]

@jacebrowning touchpoint -- are you still having this issue?

[jacebrowning]

@techalchemy Yes.

With pipenv 11.8.3, I still need this workaround in my Pipfile:

colorama = { version = "*", sys_platform = "!='fake'" }

in order to avoid a broken environment on macOS:

ModuleNotFoundError: No module named 'colorama'

[techalchemy]

@jacebrowning I just made a significant change to some of the markers parsing that was broken in 11.6.2 or thereabout, I'd be interested to know if installing from master resolves this

[techalchemy]

Also, thanks for your patience :+1:

[jacebrowning]

@techalchemy I am not able to reproduce this issue with 29a1a0b537f76586e543ea4d43b469e4ebf68174 on master.

[uranusjr]

I can confirm this is fixed. Adding a couple of tests for it.

[techalchemy]

Thanks @jacebrowning appreciate the help!