Pip-tools: add support for environment markers in requirements lines for pip-sync

[shimon]

Environment markers let you specify requirements that are conditional on the environment. They're particularly useful for OS-dependent modules. For instance, if I'm working on a django app from a Linux box, I'd like pyinotify installed. But my colleague who uses a mac won't be able to install it - the install will fail.

Pip handles these in its requirements specifiers as such:

SomeProject ==5.4 ; python_version < '2.7'
SomeProject; sys.platform == 'win32'

Unfortunately, this doesn't seem to be supported by pip-tools yet. If I put an environment marker into my requirements.in, that is stripped from the generated requirements.txt. I'm not sure if pip-sync respects an environment marker in an incoming requirement specifier, but it ought to also.

[ssbarnea]

This missing feature makes the entire pip-tools kinda useless for me. pynotify is clearly a good example of library that is needed on specific platforms.

[shimon]

FWIW, we've worked around this gap by having a separate dev-linux-requirements.in file, and handling different bundles of requirements with make. It would still be nice to have this case handled by pip-tools, though (simply passing through to the generated requirements.txt file).

[jbg]

FWIW, I’ve just observed that pip-sync ignores the environment markers in requirements.txt and tries to install packages that are marked for a different environment.

[tysonclugg]

Bonus points for allowing pip-compile --environment '{"python_version": "2.7", "sys.platform": "darwin"}' --output-file requirements-2.7-darwin.txt requirements.in as a kind of "cross-compilation" support. But seriously, don't let this request get in the way of getting something good in now - perfect comes _later_.

[merwok]

I checked today if pip-compile supported environment markers; the desired behaviour for me would be to get pinned versions with the environment markers copied in the output file. That way, I can still have just one requirement file, but cpython vs pypy could install the appropriate database driver (for CI tests).

That said, for now I have different files for local dev, testing, CI tools, and deployment, but I haven’t had a need yet for requirements specific to one of the deployment environments (e.g. debug tools on dev vs monitoring tools on staging and prod), so maybe later I’ll go down the road of splitting requirements more and handling the complexity in my pip-sync commands.

[barrywhart]

Related need: Support environment markers for indirect dependencies.

Example: I have an application that uses jsonschema. In Python 2.7, jsonschema requires functools32. In Python 3.3, it does not.

In my requirements.in file, I'd like a way to specify an environment marker for functools32 without implying that this is an application dependency.

Proposed syntax -- leading question mark for modifying indirect dependencies:

? functools32; python_version=='2.7'

[barrywhart]

Observation: pip-compile uses pip.req.parse_requirements to parse the requirements.in file, which does parse the environment markers, e.g.

> /home/vagrant/.pyenv/versions/2.7.10/envs/venv-2.7.10/lib/python2.7/site-packages/piptools/scripts/compile.py(183)cli()
(Pdb) constraints[6].markers
<Marker('python_version == "2.7"')>

This seems like good news because it's already doing part of what we want. I'll see if I can figure out a way to include these markers in the output requirements.txt file.

[adamchainz]

@barrywhart your feature is quite different to the original issue of outputting the markers. I think you should make a new issue for it and avoid spamming everyone here subscribed to the original issue.

[barrywhart]

Good point. BTW, I am creating a PR for the original issue (not including my addition).

[barrywhart]

The PR is up now. The existing and new tests are passing for me locally, but the Travis build is failing. For some reason, I am unable to view the Travis build log. Would appreciate any help getting unstuck here.

[adamchainz]

PR: #459

[barrywhart]

PR is passing Jenkins now.

[barrywhart]

Replaced PR https://github.com/jazzband/pip-tools/pull/459 with equivalent, squashed PR https://github.com/jazzband/pip-tools/pull/460

[davidovich]

closed by #460

[nirvana-msu]

I may be missing something obvious, but it's still not working for me. I have this following line in compiled requirements.txt:

hashpumpy==1.2; sys_platform=="linux"

This line comes from the same line in requirements.in (environment marker is simply copied).

If I try to run pip install -r requirements.txt on Windows, it correctly ignores the package. However if I try to run pip-sync requirements.txt, it tries to install the package (which fails because these binaries are not meant to be compiled under windows - only on linux).

[vphilippon]

You're right: pip-sync doesn't respect environment markers.

[barrywhart]

This ticket should be updated. It says that "pip-tools" does not support environment markers. Now that pip-compile supports them, the ticket only applies to pip-sync. Halfway there!

[vphilippon]

Fixed by #600

[lbolla]

Is this supposed to work if, instead of requirements.in I use setup.py to declare dependencies?
Because, it does not seem to work for me using setup.py.

[barrywhart]

Are you asking about the pip-compile tool in general, or environment markers specifically? If the first, then the answer is no. The tool takes a requirement.in file as input. You may find it useful to create a one-line requirements.in file, listing the name of your library, then run that through pip-compile.

In general, I would say pip-compile, requirements.in, and requirements.txt are intended for use by Python _applications_, and setup.py is intended for use by Python _libraries_. This is not a hard rule, but there are some occasional quirks if you use them in a way they were not intended.

[lbolla]

I meant in general. My use case is that my _application_ currently uses requirements.in to declare dependencies: all used to be good, but now we are trying to pull the app to do some integration tests and doing pip install my-app is not enough, because dependencies are not declared in setup.py.

I guess these kind of use-cases will become less frequent, with Docker and docker-compose kind of workflows.

[barrywhart]

Yeah, we've gone through something similar. I created setup.py for our apps, and now wish I hadn't. Could you leave setup.py as it is, then create a requirements.in/requirements.txt with your dependencies, then do pip install -r requirements.txt instead? I think that's the ideal workflow for most applications.

[merwok]

pip-compile does support transforming setup.py to requirements.txt, and setup.py supports environment markers:

install_requires = [
    "requests",
    "importlib; python_version == '2.6',
]

but I do not know if pip-compile setup.py will preserve the markers.

[lbolla]

@merwok it does not, from my experience.
@barrywhart the problem with that approach is that setup.py and requirements.in end up declaring the same dependencies, duplicating effort. I guess I would workaround by parsing requirements.in to figure out what dependencies to declare in setup.py, but it feels backwards to me.

[barrywhart]

If I have a requirements.in, I would try and avoid putting requirements in setup.py. It's not "wrong" or anything, but as you say, it seems backwards and may not be buying you much. AFAIK, requirements in setup.py don't buy you much unless it's a library where _applications_ that use it need to know what to install.

[AndydeCleyre]

This still does not work for pip-sync from 4.1.0, as far as I can tell. Here's an example dev-requirements.txt from the plumbum project:

pytest
pytest-cov
pytest-mock
idna<2.8 ; python_version < '2.7'
pycparser<2.18 ; python_version < '2.7'
paramiko<2.4 ; python_version < '2.7'
paramiko ; python_version >= '2.7'
setuptools
wheel ; python_version >= '2.7'
psutil

pip-sync dev-requirements.txt

Identical output whether in a Python 3.7.4 or 2.7.16 env:

Incompatible requirements found: paramiko (from -r dev-requirements.txt (line 7)) and paramiko<2.4 (from -r dev-requirements.txt (line 6))

No packages end up installed aside from pip-tools and its deps.