Pip: PEP 518 build dependency installs don't respect --cert command line option

[skylerbunny]

Environment

• pip version: 10.0.1
• Python version: Python 2.7.13
• OS: Debian Stretch (latest)

Description
pip ignores provided --cert options on the command line when it makes an installation attempt on a library that has a pyproject.toml file, meaning PEP 518 build dependencies kick in.

(Note: Whether or not attrs has a valid toml file because there's no [build-system] section present is technically irrelevant, because the failure I'm reporting is related to _how pip tries to pull what it derives_, not _what pip derives to pull_.)

Expected behavior
When pip attempts to install PEP 518 build dependencies, it should use any provided --cert or --client-cert option. I tested that it doesn't use --cert. I would guess it probably doesn't use the related --client-cert either.

How to Reproduce

1. Create a virtualenv to isolate what you're going to be doing.
2. Create/use a devpi repository with a self-signed certificate in front of it, so the problem can be seen. (Alternatively, I suppose, you could probably purposely move your verify bundle to a nonstandard location, then explicitly specify 'that file' with the --cert command line option even if you used something like https://pypi.python.org/.)
3. Run pip download --no-binary :all: --index-url https://my-devpi-server/myindex/+simple/ --cert /etc/ssl/certs/ca-certificates.crt attrs
4. The source file will successfully download. Then it will attempt to install build dependencies, which fails on an CERTIFICATE_VERIFY_FAILED error.

Output

(Note that my-index has root/pypi as an index parent, so that redirection would be expected.)

(temp) wheel-builder@build1:~/temp$ pip download --no-binary :all: --index-url https://my-devpi-server/my-index/+simple/ --cert /etc/ssl/certs/ca-certificates.crt attrs
Looking in indexes: https://my-devpi-server/my-index/+simple/
Collecting attrs
  Downloading https://my-devpi-server/root/pypi/+f/e0d/0eb91441a3b53/attrs-18.1.0.tar.gz (106kB)
  Saved ./attrs-18.1.0.tar.gz
  Installing build dependencies ... error
  Complete output from command /var/lib/wheel-builder/temp/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-QsBWnb https://my-devpi-server/root/pypi/+f/8fc/a9275c89964f1/setuptools-39.2.0-py2.py3-none-any.whl#sha256=8fca9275c89964f13da985c3656cb00ba029d7f3916b37990927ffdf264e7926 https://my-devpi-server/root/pypi/+f/800/44e51ec5bbf6c/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f:
  Collecting setuptools==39.2.0 from https://my-devpi-server/root/pypi/+f/8fc/a9275c89964f1/setuptools-39.2.0-py2.py3-none-any.whl#sha256=8fca9275c89964f13da985c3656cb00ba029d7f3916b37990927ffdf264e7926
    Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)'),)': /root/pypi/+f/8fc/a9275c89964f1/setuptools-39.2.0-py2.py3-none-any.whl
(((Repeats four more times)))
  Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='my-devpi-server', port=443): Max retries exceeded with url: /root/pypi/+f/8fc/a9275c89964f1/setuptools-39.2.0-py2.py3-none-any.whl (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)'),))


  ----------------------------------------
Command "/var/lib/wheel-builder/temp/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-QsBWnb https://my-devpi-server/root/pypi/+f/8fc/a9275c89964f1/setuptools-39.2.0-py2.py3-none-any.whl#sha256=8fca9275c89964f13da985c3656cb00ba029d7f3916b37990927ffdf264e7926 https://my-devpi-server/root/pypi/+f/800/44e51ec5bbf6c/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f" failed with error code 1 in None

[skylerbunny]

I'm guessing this problem is here:

https://github.com/pypa/pip/blob/77f6ac676a012ca48d82f848b28b3a706b8cca4e/src/pip/_internal/build_env.py#L79

(Which is to say that --cert nor --client-cert are checked for nor appended in this use case.)

[benoit-pierre]

Indeed, session specific options are not being forwarded.

[adamgreg]

This is still the case in 19.2.2

[onlineque]

Still the case with 20.0.2. Is there a chance that this bug will be fixed ? It effectively prevents me using local PYPI repository with HTTPS due to that and I do not want to workaround this bug with ugly --trusted-host

[uranusjr]

IIUC the fix to this issue should not be too technically involved—locate where the PEP 517 hook installs build dependencies, and pass the flags to them (likely via environment variables for better compatibility). The biggest problem would be to find someone interested enough in this to do the necessary code-tracing and patching, which is still significant but boring work. Feel free to give it a shot if you’re inclined to.

[chrahunt]

Anyone that needs this behavior now should be able to work around it by specifying their cert path in the PIP_CERT environment variable or specifying cert in one of the pip configuration files.