Pip-tools: Producing constraints.txt files

[nbren12]

What's the problem this feature will solve?

We have been using pip-compile to manage building the environments of several different "micro-projects" in a monorepo while ensuring that if two micro-projects share have the same dependency then the same version of that dependencey is used. To do this, we use the output of pip-compile as a constraints.txt file. The overall structure is like this

pip-compile app1 app2 app3 --output-file constraints.txt

# build environment for one app
pip install -c constraints.txt app1

This worked fine with the old pip, but with the new dependency solver, pip install -c does not support extra requirements. This is the error message:

$ pip install -c constraints.txt ipython
DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement.. You can find discussion regarding this at https://github.com/pypa/pip/issues/8210.
ERROR: Constraints cannot have extras

I'm not sure if this is a "bug" or an un-supported use-case. Until now it has worked quite well for our monorepo, and is part of why I chose pip-tools over less composable solutions like poetry.

[nbren12]

An more general approach would be to add some tools for performing set operations on lists of dependencies.

[tuukkamustonen]

I'm also doing a "monorepo" like setup and I've been using somewhat similar approach - after adding or upgrading a dependency, I've _copied_ the main level lockfile to the modules and then re-compiled them (which then removes all unnecessary content, leaving just the changes). This doesn't require using the main level lockfile as constraints file, but also feels even more hacky.

Some more "official" support for multi-module scenarios would be nice. I wonder if there's already some (more generic) ticket about it...

[ynouri]

We also have a Python monorepo setup based on Pants. We follow Pants recommendation to use a lockfile, and decided to use pip-compile to produce the constraints.txt lockfile.

Pants happily consumes the constraints.txt produced by pip-compile, which is great. Also, the formatting of constraints.txt is really great and easy to understand, the update workflow is super simple and straightforward thanks to the comments... we love pip-compile.

Unfortunately, as mentioned above, pip is not as accommodating as Pants about consuming that constraints.txt file. It complains about the extras. We hit this problem when trying to build a Docker image with the lockfile.

The poor man's workaround we found is to use sed to search for brackets and remove them:

sed -i 's/\[.*\]//g' /opt/constraints.txt
pip install -r /opt/requirements.txt -c /opt/constraints.txt

Hope it can help.

[nbren12]

We have nearly identical code in our makefile. It's cool that pip-tools works well with pants.

[jkowalleck]

wonder why pip-compile would add extras to the requirements-out - like coverage[toml]==5.5
the [toml] part should not be needed in the requirements-out, since all the extra-requirements caused by coverage[toml] should be individual requirements in the requirements-out already.

is there really a need for coverage[toml]==5.5, is it not enough to generate coverage==5.5 ?

[jkowalleck]

Hello @nvie, @atugushev,
could you maybe shed some light on the topic "pip-compile with extras"?

Is it necessary to write the used extras (like package-B[extra-C]) in the out-file?
If so, why? are there architectural reasons for this? are there requirements that it stays as it is?

from my current point of view, it would be enough to simply add the original package(package-B) to the out-file, without mentioning the extra(extra-C). The requirements that are caused by the extra should be part of the out-file anyways.

example/suggestion:

Given a package-A requires package-B[extra-C]
And a package-B has an extra extra-C
And package-B's extra extra-C requires package-C
When i compile an in-file with the following content

Then the out-file is this

package-A==1.0
    # via -r in-file
package-B==2.0
    # via package-A
package-C==3.0
    # via package-B[extra-C]

---

Currently i get an out-file i cannot use as a constraint file for pip install -c:

package-A==1.0
    # via -r in-file
package-B[extra-C]==2.0
    # via package-A
package-C==3.0
    # via package-B

[ssbarnea]

Did anyone found a workaround for this issue as I was using pip-tools inside various projects in order to produce constraint files, which due ot mentioned pip bug-fix are no longer valid.

AFAIK only fixing pytest-cov to avoid using dependencies with extras should buy us enough time to implement the missing feature in pip-tools.

It is quite sad as pip caused this failure and they could have instead ignore the extras and throw an warning but they decided to break what was working before.

[nbren12]

@ssbarnea This trick works: https://github.com/jazzband/pip-tools/issues/1300#issuecomment-818122483

[ssbarnea]

Using sed in a (bsd) compatible way is tricky and also makes the update non atomic. Yes, i am using it but I am far from happy,... probably a good reason/motivator to sacrifice half a day and fix this in pip-tools.

[jkowalleck]

thanks to #1404 there will be a switch --strip-extras.
this will cause the output to be constraint-compatible.