Pip: Trusted Hosts Aren't Cached

[natedogith1]

Environment

• pip version: 20.0.2
• Python version: 3.7.3
• OS: CentOS Linux release 7.4.1708
• Virtualenv version: 16.6.1 (ran as "virtualenv --python=3.7 build/python_env")

Description
If a host is added as a trusted-host, it won't use the cache.

Expected behavior
trusted-hosts use the cache.

How to Reproduce

1. pip config --site set global.index-url https://example.com/api/pypi/pypi/simple/
2. pip config --site set global.trusted-host example.com
3. pip install -r pip_requirements.txt

Output
Line from broken output:

Collecting requests==2.23.0
  Downloading https://artifactory.rnclab.us.alcatel-lucent.com/artifactory/api/pypi/pypi/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 3.0 MB/s

Line from non-broken output (not configuring trusted-host):

Collecting requests==2.23.0
  Using cached https://example.com/artifactory/api/pypi/pypi/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58 kB)

[NoahGorny]

As you can see in commit 5bab65d1, a change was introduced that did not cache responses for any "unsafe" origin, such as http, or invalid https, even with the "trusted-host" option.

I also encounter this behavior as in my workplace I must use "trusted-host" as well, and it is pretty annoying that it does not cache any responses at all, so I have 2 options to offer:
1) Change this behavior to allow caching from any "trusted-host" source. Makes sense as you specifically said the host should be trusted. Maybe refrase the "trusted-host" docs to specify that. This option however changes default behavior and more drastic.
2) New command line option called cache-trusted-host that will do just that, then you can specify this option in pip config etc...

Out of these two option I prefer the first, but I think we should have a discussion about it.
In any way this is not a lot of diff, I can open a PR if needed :)

[NoahGorny]

hey @pradyunsg. can you please take a look at this and the PR I opened to solve it? 😄

[d3dave]

The change discussed for not caching packages from insecure origins was introduced alongside the --trusted-host flag in pip version 6. Then, installing from insecure origins was deprecated but not disabled (pip reported a warning, but still fetched the packages). As a result, the cache could still be poisoned by untrusted origins.

Since version 7, pip does not allow fetching packages from insecure origins unless they are explicitly marked trusted. As a result, the original change for not caching packages loses merit, because the cache cannot be poisoned anymore by untrusted origins (pip refuses to fetch the packages).

In light of the above, I believe the correct solution here is option 1.

[sbidoul]

Could it be a third option to use --cert with a custom certificate bundle, as an alternative way to trust that host?

[NoahGorny]

In my workplace some certs are expired, some use weak algorithms that are no longer supported, etc...
It would take years for me to convince the IT department to do anything about it, and I do not think that I (or my workplace) are unique in this regard.
I am fine with caching "insecure" sites when I have no other option to do otherwise