Pip-tools: pip-compile 3.7.0 no longer honors --no-index

[jbergknoff-rival]

pip-compile no longer honors the --no-index argument, so the index URL (containing repository credentials) is present in the header of the generated file. This is a regression between pip-tools 3.6.1 and 3.7.0. My guess is it's related to this change: https://github.com/jazzband/pip-tools/pull/800.

Environment Versions

1. OS Type: Linux
2. Python version: $ python -V: 3.6.8
3. pip version: $ pip --version: 19.0.3
4. pip-tools version: $ pip-compile --version: 3.7.0

Steps to replicate

1. pip-compile --no-index --output-file requirements/runtime.txt requirements/runtime.in
2. Look at the header in requirements/runtime.txt

Expected result

The header of requirements/runtime.txt shouldn't have an index URL containing repository credentials

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --no-index --output-file requirements/runtime.txt requirements/runtime.in
#

Actual result

The index URL is present.

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --index-url=... --no-index --output-file=requirements/runtime.txt requirements/runtime.in
#

[atugushev]

Hello @jbergknoff-rival,

Thanks for the report! Where did you get the index-url? Is it configured in pip.conf? Can't reproduce locally though.

[jbergknoff-rival]

Hi, it's set as an environment variable PIP_INDEX_URL.

3.7.0

$ docker run -it --rm python:3.6.8-alpine3.9 sh
/ # pip install pip-tools==3.7.0
Collecting pip-tools==3.7.0
  Downloading https://files.pythonhosted.org/packages/96/43/34412d316bdbf1cd9c9a0e487138b40db4e1e11212cee2d46440b6b49b08/pip_tools-3.7.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools==3.7.0)
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
    100% |████████████████████████████████| 81kB 3.2MB/s 
Collecting six (from pip-tools==3.7.0)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: click, six, pip-tools
Successfully installed click-7.0 pip-tools-3.7.0 six-1.12.0
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/ # echo requests > requirements.in
/ # PIP_INDEX_URL=https://pypi.org/simple pip-compile --no-index --output-file requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --index-url=https://pypi.org/simple --no-index --output-file=requirements.txt requirements.in
#
certifi==2019.3.9         # via requests
chardet==3.0.4            # via requests
idna==2.8                 # via requests
requests==2.21.0
urllib3==1.24.3           # via requests

3.6.1

$ docker run -it --rm python:3.6.8-alpine3.9 sh
/ # pip install pip-tools==3.6.1
Collecting pip-tools==3.6.1
  Downloading https://files.pythonhosted.org/packages/06/96/89872db07ae70770fba97205b0737c17ef013d0d1c790899c16bb8bac419/pip_tools-3.6.1-py2.py3-none-any.whl
Collecting six (from pip-tools==3.6.1)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools==3.6.1)
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
    100% |████████████████████████████████| 81kB 3.9MB/s 
Installing collected packages: six, click, pip-tools
Successfully installed click-7.0 pip-tools-3.6.1 six-1.12.0
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/ # echo requests > requirements.in
/ # PIP_INDEX_URL=https://pypi.org/simple pip-compile --no-index --output-file requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --no-index --output-file requirements.txt requirements.in
#
certifi==2019.3.9         # via requests
chardet==3.0.4            # via requests
idna==2.8                 # via requests
requests==2.21.0
urllib3==1.24.3           # via requests

[atugushev]

Oh, i see now. Changing values using environments is not tested :( As a workaround you can set index-url in pip.conf. I think pip-compile shouldn't render values from environment in command header. Will be fixed soon.

[atugushev]

@jbergknoff-rival

I've completely forgotten about this issue and finally had a chance to look into it.

Technically that feature is still working as expected (but has a side-effect described below the line). The option --index/--no-index means "Add index URL to generated file". For example:

will add index URL

$ echo "" | pip-compile - --index-url=https://foo -qo-
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --index-url=https://foo --output-file=- -
#
--index-url https://foo

will not add index URL:

$ echo "" | pip-compile - --index-url=https://foo -qo- --no-index
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --index-url=https://foo --no-index --output-file=- -

Everything is okay, isn't it? The only thing is that --no-index conflicts with pip's option. See pip install --help:

  --no-index                  Ignore package index (only looking at --find-
                              links URLs instead).

on the pip-tools side I would rather rename it to --emit-index-url/--no-emit-index-url to avoid ambiguity (see also #373).

---

Now, speaking of the issue. Since #800 we have side-effect: if you use env var (such as PIP_INDEX_URL/PIP_FIND_LINKS/etc) the values from that envs will be rendered in "To update, run:" header. See:

$ echo "" | PIP_INDEX_URL=https://foo pip-compile - -qo-
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --index-url=https://foo --output-file=- -
#
--index-url https://foo

same as

$ echo "" | pip-compile - --index-url=https://foo -qo-
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --index-url=https://foo --output-file=- -
#
--index-url https://foo

The benefit of it that now you have deterministic command in the header. If you re-run the given command from the header without env vars you'll get the same result. That might be useful.

If you have sensitive data in the input values you can always suppress the header by --no-header option.

[atugushev]

I'll close this based on the above, but please let us know if it doesn't resolve your issue. Thanks!

[rooterkyberian]

The benefit of it that now you have deterministic command in the header. If you re-run the given command from the header without env vars you'll get the same result. That might be useful.

If you have sensitive data in the input values you can always suppress the header by --no-header option.

IMO these two statements are opposing each other.
I need to suppress sensitive token embedded in extra index URL from header, but I also want to have a command for other developers to regenerate the file when they please. I could try going after this with some sed magic, but then I wouldn't have a "deterministic command".

Is it just me, or case when somebody(/team of people) sometimes like to set extra index URL in ENV and sometimes using command line option sounds like an edge case as opposed to having sensitive information passed through ENV to prevent having it written to any files, especially one contained in source code repository?

[atugushev]

Hello @rooterkyberian,

Thanks for the feedback! I really appreciate it.

I need to suppress sensitive token embedded in extra index URL from header, but I also want to have a command for other developers to regenerate the file when they please. I could try going after this with some sed magic, but then I wouldn't have a "deterministic command".

This sounds reasonable and I revisited my opinion. Especially with #1028.

[orneta19]

I am also experiencing this issue, from reading all the threads I couldn't see a fix, only a discussion on one. Am I missing it, or is there no fix yet?
Thanks,
Neta

[jbergknoff-rival]

I think the upshot is that the interpretation of --no-index, from the issue statement, was incorrect. At least, as far as I know, the behavior of <3.7.0 was never restored. My team started using --no-header to avoid leaking secrets in the pip-compile output.

[atugushev]

Here is the fix #1146. Please test and review it, any feedback is welcomed!