Pip-tools: pip-compile bumps setuptools within commented-out spec

[PeterJCLaw]

If you have a requirements.in which contains a package which depends on setuptools, for example:

$ cat requirements.in 
protobuf  # or google-api-core, ipython, pandas-gbq, pydata-google-auth

Then after running pip-compile, you get:

$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
protobuf==3.10.0
six==1.12.0               # via protobuf

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.4.0        # via protobuf

As you can see, pip-compile refuses to include setuptools directly in a generated requirements.txt, instead adding a section with the comment "The following packages are considered to be unsafe in a requirements file:".
This behaviour is fine in the general (and I don't disagree with the comment or behaviour).

However because the commented out setuptools line includes a version number, when setuptools releases an update then the requirements.txt file ends up looking out of date.

Say you had committed a requirements.txt which looked like:

# requirements...

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.4.0        # via protobuf

Then setuptools releases 41.5.0 and now when you run pip-compile you get:

# requirements...

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.5.0        # via protobuf

This is inconvenient if you happen to regenerate your requirements.txt at some point after an upstream releases, however if you're also using pip-compile to check that your requirements.txt is in step with your requirements.in (see #882) in CI then it means that your build now fails for what is essentially a non-change.

I'm not sure what the right solution is here, though perhaps if there was a way for pip-compile to understand its own commented out dependency that might help?

[atugushev]

Hello @PeterJCLaw,

Sorry, I didn't quite understand what you are trying to achieve and how you use pip-compile in that way. Could you add more details, attach the requirements.txt, provide commands you run?

[PeterJCLaw]

@atugushev sorry, yeah, on reflection I didn't include enough info. I've updated the description -- does that help?

[atugushev]

I see. The option --allow-unsafe might help. Have you tried it?

[PeterJCLaw]

Aha, thanks. I wasn't aware of that option -- I'll check that out. I think it would still be useful if there was a solution here which didn't involve doing something considered unsafe.

[atugushev]

Aha, thanks. I wasn't aware of that option -- I'll check that out.

This gave me an idea to add a mention about --allow-unsafe in the comment line, like we have when you try to compile with --generate-hashes unsafe packages, for example:

$ echo -e "six\nsetuptools" | pip-compile - -qo- --generate-hashes
The generated requirements file may be rejected by pip install. See # WARNING lines for details.
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes --output-file=- -
#
six==1.12.0 \
    --hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c \
    --hash=sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools==41.5.0

[atugushev]

I think it would still be useful if there was a solution here which didn't involve doing something considered unsafe.

The solution might be to unpin unsafe packages without --allow-unsafe, for example:

$ echo setuptools | pip-compile - -qo-
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file=- -
#

# The following packages are considered to be unsafe in a requirements file:
# setuptools

And pin unsafe packages with --allow-unsafe:

echo setuptools | pip-compile - -qo- --allow-unsafe
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --allow-unsafe --output-file=- -
#

# The following packages are considered to be unsafe in a requirements file:
setuptools==41.5.0

[atugushev]

@PeterJCLaw

I've submitted #975 which implements https://github.com/jazzband/pip-tools/issues/973#issuecomment-547010219. Please, review/test it if you have time.

[atugushev]

The solution might be to unpin unsafe packages without --allow-unsafe

FTR, it was already discussed before, see https://github.com/jazzband/pip-tools/pull/276#issuecomment-171567920, and fixed in #294, but looks like it has been broken later unintentionally.

[PeterJCLaw]

@atugushev thanks for fixing this, do you know when this will be available in a release? (setuptools has been bumped recently, so would be great if this could be out soon)

[atugushev]

@PeterJCLaw here is the tracking issue #1005. We'll release it very soon.

[atugushev]

pip-tools v4.3.0 is released.

[PeterJCLaw]

Awesome, thanks!