Pipenv: Problem with Pipfile and system specific packages

[Devligue]

Specifying certain packages to be installed only for some operating systems sometimes works, but sometimes not. For me it is difficult to overcome the problem of pyqt5 and pypiwin32 package entries in Pipfile when deploying on Linux.

Describe your environment

1. OS: Ubuntu 17.04
2. Python version: 3.5.3
3. Pipenv version: 11.0.2

Example

In my Pipfile the [packages] entry looks like this (as per the doc example):

[packages]
"pypiwin32" = {version = "*", os_name = "== 'windows'"}

or like this:

[packages]
"pyqt5" = {version = "*", os_name = "== 'windows'"}

Expected result

In the pypiwin32 example I simply expected this package not to be installed on other platforms than Windows. Self-explanatory.

In the pyqt5 example I would also expect this package to be omitted on other platforms than Windows. For whatever reason on Linux (not sure if every, but some) we can't install pyqt5 with pip anyways.

Or at least let the warning be thrown but I would like the rest of the packages listed in Pipfile to be installed at least.

Actual result

In pypiwin32 example:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches pywin32>=223
Tried: (no version found at all)

In pyqt5 example:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches pyqt5
Tried: (no version found at all)

As you can see it did not matter if I specified to not install those packages - the error still occurs. In fact it does not matter at all what I write after the "packagename" = bit because apparently the packages themselves just being there listed cause some problems. And non of the 'correct' packages that might be listed in Pipfile will be installed

And just for the record, the docs example of doing exactly the same thing (with pywinusb = {version = "*", os_name = "== 'windows'"}) worked seamlessly.

Steps to replicate

Pipfile

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

"pypiwin32" = {version = "*", os_name = "== 'windows'"}
"pyqt5" = {version = "*", os_name = "== 'windows'"}
requests = "*"


[dev-packages]


[requires]

• pipenv --python 3.5
• pipenv install or pipenv update

[uranusjr]

You’re looking for either os_name = "== 'nt'" or platform_version = "== 'Windows'". This has come up a couple of times (latest is #1523).

@kennethreitz We really need to correct the documentation. I don’t care if it’s taken directly from the PEP 508 docs, it is simply wrong, and is causing real problems.

[Devligue]

@uranusjr Well I have noticed that the os_name = "== 'windows'" was ignored on windows while platform_version = "== 'Windows'" worked, but since this issue is about linux and not windows I decided to use docs example.

But for what matters the issue described above persists (as mentioned) regardless of what is written after = so neither of os_name = "== 'nt'" or platform_version = "== 'Windows'" solves it.

[kennethreitz42]

this nipped me today: https://github.com/pypa/pipenv/blob/master/Pipfile#L13

what's the correct usage here?

[kennethreitz42]

got it

[kennethreitz42]

updated the docs

[kennethreitz42]

changed to

pywinusb = {version = "*", sys_platform = "== 'win32'"}

[Devligue]

Thank you @kennethreitz for fixing the docs, but the problem is, that you have closed this issue while it is still unresolved. My problem was not related to whether to use sys_platform, os_name or platform_version. My problem is that pypiwin32= {version = "*", sys_platform = "== 'win32'"} when used on Linux crashes all packages installation and throws error. Its installation should be omitted but pipenv still searches for this package.

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches pywin32>=223
Tried: (no version found at all)

[techalchemy]

@Devligue this is not an issue with pipenv, it's an issue with how pywin32 has organized their setup.py, we literally have no way to work around this and it is only a problem for this specific package.

[Devligue]

I see. I will try to report this issue to pywin32 then. Thank you for explanation @techalchemy

[Devligue]

On the second thought there might be a way...

You see if we do pip search pywin32 under linux we clearly see that the package is listed there. It is only if we do pip install pywin32 we get the error:

Could not find a version that satisfies the requirement pywin32 (from versions: )
No matching distribution found for pywin32

But since I it is specified to install pywin32 only under windows then why pipenv even looks at this package? My conclusion is that the specifiers are checked at wrong moment. They maybe should be checked before doing anything with pip and the package. If the specifier says clearly that for example system version is not matching then this package should not be taken into consideration at all. Am I wrong?

[altendky]

@Devligue, I'm still probably going to submit a PR for pywin32 but FYI it seems that this has at least changed as of pipenv 2018.6.25.

altendky@lt:~/pywin32.pipenv$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"pypiwin32" = {version = "*", os_name = "== 'windows'"}

[dev-packages]

[requires]
python_version = "3.6"

```python
altendky@lt:~/pywin32.pipenv$ /home/altendky/venvs/pipenv/bin/pip install --upgrade pipenv==2018.5.18
Requirement already up-to-date: pipenv==2018.5.18 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (2018.5.18)
Requirement already satisfied, skipping upgrade: virtualenv in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.5.18) (16.0.0)
Requirement already satisfied, skipping upgrade: setuptools>=36.2.1 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.5.18) (40.0.0)
Requirement already satisfied, skipping upgrade: certifi in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.5.18) (2018.4.16)
Requirement already satisfied, skipping upgrade: virtualenv-clone>=0.2.5 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.5.18) (0.3.0)
Requirement already satisfied, skipping upgrade: pip>=9.0.1 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.5.18) (18.0)
altendky@lt:~/pywin32.pipenv$ pipenv --version; pipenv --rm; rm -f Pipfile.lock; pipenv install
pipenv, version 2018.05.18
Removing virtualenv (/home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy)…
Creating a virtualenv for this project…
Using /home/altendky/.pyenv/versions/3.6.3/bin/python3.6m (3.6.3) to create virtualenv…
⠋Running virtualenv with interpreter /home/altendky/.pyenv/versions/3.6.3/bin/python3.6m
Using base prefix '/home/altendky/.pyenv/versions/3.6.3'
New python executable in /home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy/bin/python3.6m
Also creating executable in /home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (505b74)!
Installing dependencies from Pipfile.lock (505b74)…
Ignoring pypiwin32: markers 'os_name == "windows"' don't match your environment
An error occurred while installing pywin32==223! Will try again.
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00
Installing initially–failed dependencies…
Collecting pywin32==223 ▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

Could not find a version that satisfies the requirement pywin32==223 (from -r /tmp/pipenv-nu5q7_3u-requirements/pipenv-r1wj8fro-requirement.txt (line 1)) (from versions: )
No matching distribution found for pywin32==223 (from -r /tmp/pipenv-nu5q7_3u-requirements/pipenv-r1wj8fro-requirement.txt (line 1))

☤ ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00

```python
altendky@lt:~/pywin32.pipenv$ /home/altendky/venvs/pipenv/bin/pip install --upgrade pipenv==2018.6.25
Collecting pipenv==2018.6.25
  Using cached https://files.pythonhosted.org/packages/0d/e5/cbd43eaf3ca60c4b75ac8a4ff4f7ba102b629f519a2b325d67c4237b7fcc/pipenv-2018.6.25-py3-none-any.whl
Requirement already satisfied, skipping upgrade: setuptools>=36.2.1 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.6.25) (40.0.0)
Requirement already satisfied, skipping upgrade: virtualenv in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.6.25) (16.0.0)
Requirement already satisfied, skipping upgrade: certifi in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.6.25) (2018.4.16)
Requirement already satisfied, skipping upgrade: pip>=9.0.1 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.6.25) (18.0)
Requirement already satisfied, skipping upgrade: virtualenv-clone>=0.2.5 in /home/altendky/venvs/pipenv/lib/python3.6/site-packages (from pipenv==2018.6.25) (0.3.0)
Installing collected packages: pipenv
  Found existing installation: pipenv 2018.5.18
    Uninstalling pipenv-2018.5.18:
      Successfully uninstalled pipenv-2018.5.18
Successfully installed pipenv-2018.6.25
altendky@lt:~/pywin32.pipenv$ pipenv --version; pipenv --rm; rm -f Pipfile.lock; pipenv install
pipenv, version 2018.6.25
Removing virtualenv (/home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy)…
Creating a virtualenv for this project...
Pipfile: /home/altendky/pywin32.pipenv/Pipfile
Using /home/altendky/.pyenv/versions/3.6.3/bin/python3.6m (3.6.3) to create virtualenv...
⠋Running virtualenv with interpreter /home/altendky/.pyenv/versions/3.6.3/bin/python3.6m
Using base prefix '/home/altendky/.pyenv/versions/3.6.3'
New python executable in /home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy/bin/python3.6m
Also creating executable in /home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy/bin/python
Installing setuptools, pip, wheel...done.
Setting project for pywin32.pipenv-PibPzvEy to /home/altendky/pywin32.pipenv

Virtualenv location: /home/altendky/.local/share/virtualenvs/pywin32.pipenv-PibPzvEy
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (505b74)!
Installing dependencies from Pipfile.lock (505b74)...
Ignoring pypiwin32: markers 'os_name == "windows"' don't match your environment
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternativaly, run a command inside the virtualenv with pipenv run.
altendky@lt:~/pywin32.pipenv$

[E3V3A]

Just out of curiosity, what does sys_platform, os_name or platform_version show for Cygwin?
(I'm investigating a very similar problem, where stuff refuse to get pip installed when using Cygwin.)

Or better, since I have it, how can easily I check what it is?

[altendky]

@E3V3A I'd guess sys.platform, os.name, and platform.version. Hopefully they match. *fingers-crossed*

[E3V3A]

Just to report back on something I've forgotten about:

# Current OS variables as seen by Python3 (3.6.4)

os.getenv("TERM"):   xterm
os.name:             posix
os.sys.platform:     cygwin
sys.platform:        cygwin

platform:
        machine:       x86_64
        processor:
        system:        CYGWIN_NT-6.3
        release:       2.11.2(0.329/5/3)
        version:       2018-11-08 14:34
        platform:      CYGWIN_NT-6.3-2.11.2-0.329-5-3-x86_64-64bit-WindowsPE

[Overdrivr]

Are there any updates on this ?

In my case, the lock is performed successfully on a Windows machine, and the install fails with the lock file on a Unix machine, with the same error message as OP. Is it the same issue than described in this ticket ? I initially though so, but I'm starting to reconsider.

If that's the case, why isn't pipenv capable to ignore the package during install ?

[packages]
pyqt5 = "==5.12.1"
"pypiwin32" = {version = "*", sys_platform = "== 'win32'"}

[uranusjr]

There has been on-and-off discussions on this, but to summarise the problem:

1. Packages/packaging tools need to provide extra metadata to describe how a bunch of files can “provide” a package across the environment boundary.
2. Package managers need better resolution logic to interpret said metadata.

I cannot find the discussion off-hand, but you probably want to look on discuss.python.org and the Distutils-SIG mailing list.

[ErikBjare]

I managed to get the lock and install to work on a Linux machine by simply adding another line for pywin32:

[packages]
pypiwin32 = {version = "*", sys_platform = "== 'win32'"}
pywin32 = {version = "*", sys_platform = "== 'win32'"}

Output of pipenv lock -r:

-i https://pypi.org/simple
pypiwin32==223 ; sys_platform == 'win32'
pywin32==226 ; sys_platform == 'win32'

I haven't tried running it on Windows yet, but I expect it to work.