Pip-tools: Should --generate-hashes imply --allow-unsafe?

[jcushman]

Some packages, such as pytest and Markdown, include minimum versions of setuptools in their requirements. Running pip-compile --generate-hashes without --allow-unsafe for those packages creates a requirements.txt file that doesn't work with pip install. This creates a bug that is easy to miss until deployment. Should --allow-unsafe be the default behavior for --generate-hashes?

Environment Versions

1. OS Type: macOS 10.14.4
2. Python version: Python 3.5.3
3. pip version: pip 19.1
4. pip-tools version: pip-compile, version 3.6.2.dev19+g8a1a8ef

Steps to replicate

1. New virtualenv:

$ pyenv virtualenv 3.5.3 piptest
$ pyenv local piptest
$ pip show setuptools
Name: setuptools
Version: 28.8.0

1. Require Markdown, which requires a newer version of setuptools:

$ echo "Markdown" > requirements.in
$ $ pip-compile --generate-hashes
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes
#
markdown==3.1 \
    --hash=sha256:fc4a6f69a656b8d858d7503bda633f4dd63c2d70cf80abdc6eafa64c4ae8c250 \
    --hash=sha256:fe463ff51e679377e3624984c829022e2cfb3be5518726b06f608a07a3aad680

1. Attempt pip install:

$ pip install -r requirements.txt
Collecting markdown==3.1 (from -r requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/f5/e4/d8c18f2555add57ff21bf25af36d827145896a07607486cc79a2aea641af/Markdown-3.1-py2.py3-none-any.whl
Collecting setuptools>=36 (from markdown==3.1->-r requirements.txt (line 7))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=36 from https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl#sha256=c7769ce668c7a333d84e17fe8b524b1c45e7ee9f7908ad0a73e1eda7e6a5aebf (from markdown==3.1->-r requirements.txt (line 7))

Expected result

Successful install of Markdown.

Actual result

Install fails because setuptools==41.0.1 is not pinned.

Discussion

Seems like it would be good to do one of the following:

• Print a warning to use --allow-unsafe when --generate-hashes is provided and there are unsafe package requirements.
• Treat --generate-hashes as implying --allow-unsafe, maybe with a different comment in the generated requirements.txt.

I don't know why pip-compile chooses not to include setuptools or why pip chooses to require it, so I'm not sure what the correct thing to do is, but it seems like pip-compile has all the info it needs to recommend whatever that thing is.

(FWIW this is easy to miss and create latent bugs -- I did pip install Markdown first, which upgraded setuptools and masked the error, and then things broke a few days later when another dev tried to recreate their virtualenv.)

[atugushev]

Hello @jcushman,

Thank you for the very detailed report and good catch!

I don't know why pip-compile chooses not to include setuptools

See the comment in the examples/ptrotection.in file: https://github.com/jazzband/pip-tools/blob/8a1a8efaf0d4b3d17e57ea69d41b86373264a8f4/examples/protection.in#L1-L2

or why pip chooses to require it

The pip uses setuptools to install packages in editable mode (i.e. setuptools "develop mode"), see:

$ pip uninstall setuptools -y
Uninstalling setuptools-41.0.1:
  Successfully uninstalled setuptools-41.0.1

$ pip install -e [email protected]:jazzband/pip-tools.git#egg=pip-tools
Obtaining pip-tools from [email protected]:jazzband/pip-tools.git#egg=pip-tools
  Cloning [email protected]:jazzband/pip-tools.git to ./.venv/src/pip-tools
  Running command git clone -q [email protected]:jazzband/pip-tools.git /apps/pip-tools/.venv/src/pip-tools
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /apps/pip-tools/.venv/src/pip-tools/

Seems like it would be good to do one of the following:

• Print a warning to use --allow-unsafe when --generate-hashes is provided and there are unsafe package requirements.

Sounds good to me. I'd rather to print a warning, instead of implying --allow-unsafe with --generate-hashes, since it may cause conflicts with the pip. Let a user to decide.

[jcushman]

The pip uses setuptools to install packages in editable mode

But that's because the package you're installing is importing setuptools -- not because pip is importing setuptools. pip vendors setuptools and doesn't care what version you have installed locally. You can't break pip itself by pinning setuptools -- you should only have a problem if a target requirement relies on an incompatible version. But that's no different from any other incompatible sub-dependency, and can be resolved the same way.

[atugushev]

pip vendors setuptools and doesn't care what version you have installed locally.

It's not vendored, look at here. Pip only calls subprocess setuptools, see this block. But you are right - pip doesn't care the version.

You can't break pip itself by pinning setuptools -- you should only have a problem if a target requirement relies on an incompatible version. But that's no different from any other incompatible sub-dependency, and can be resolved the same way.

It might be some legacy reason to consider the setuptools unsafe. Insights from creators @nvie @vphilippon @alekzvik would be very helpful.

PS
@jcushman it so nice that you've raised an issue in the pip. I'll Keep an eye on it 👀

[jcushman]

It's not vendored, look at here.

It's partially vendored. :) From your link, "setuptools is completely stripped to only keep pkg_resources", which is the part of setuptools that pip imports rather than calling as a subprocess.

This discussion leads me to think that --allow-unsafe might actually be safer to use than not use. If you have pytest in your requirements.in then all your deployments will end up with different versions of setuptools based on the time of installation, because pytest will upgrade setuptools for you when it's installed. If you use --allow-unsafe then you end up with identical deployments starting from an identical base image, which sounds safer.

[atugushev]

It's partially vendored. :) From your link, "setuptools is completely stripped to only keep pkg_resources", which is the part of setuptools that pip imports rather than calling as a subprocess.

Yes, only pkg_resources.

FTR unsafe packages are also used in pip-sync to ignore uninstall those packages.

[atugushev]

I'll close this in favour of #814. Feel free to reopen if there's something to say.

[jcushman]

Yep, thanks for closing this! I'm starting to think it might make sense to deprecate --allow-unsafe and make it the default behavior, but that would be as a separate issue/PR for discussion.

[funtikas25]

hello pleas help what the hash is hier ???????
[18:21:27] [WARNING] there has been a problem while writing to the session file ('OperationalError: attempt to write a readonly database')
[18:21:30] [INFO] retrieved: 07b1e0439ba077bc7bdfbbf2341b1d1f:dc
[18:21:30] [DEBUG] performed 249 queries in 380.19 seconds
[18:21:30] [INFO] retrieving the length of query output
[18:21:30] [INFO] retrieved: 21
[18:23:18] [DEBUG] performed 10 queries in 107.48 seconds
[18:23:18] [DEBUG] starting 8 threads
[18:24:14] [INFO] retrieved: o_i________________ 2/21 (9%)

[atugushev]

@jcushman

Yep, thanks for closing this! I'm starting to think it might make sense to deprecate --allow-unsafe and make it the default behavior, but that would be as a separate issue/PR for discussion.

FTR, this topic on StackOverflow reminds me that we should file an issue and discuss it.