Pip: Ability to use multiple pypi mirrors with fallback support

[ssbarnea]

At this moment pip can use only two index URLS and because most people have a dual setup with one public repository and one private one, it prevents them from setting up failover setups.

I will describe this common setup: main index url points to a pypi mirror (usually a local proxy) which accelerates public package lookup and retrieval. The second url points to a local repository which is containes private or not-yet released packages (like ones you test in CI).

If the main mirror/proxy goes down or starts to misbehave, your entire build system is message as you have no way to configure pip to use a fallback.

Real life example http://logs.openstack.org/95/619595/10/check/tripleo-ci-centos-7-containers-multinode-pike/3169d86/job-output.txt.gz#_2018-11-28_09_53_10_469581 -- which is caused by some broken IPv6 routing on rax hosting.

As there is no way to assure 100% reliability on a single mirror we need a way to tell pip to use fallbacks in case primary one timesout.

[pfmoore]

At this moment pip can use only two index URLS

Why do you say that? I'm pretty sure you can have multiple --extra-index-url options...

https://github.com/pypa/pip/blob/master/src/pip/_internal/cli/cmdoptions.py#L305

[ssbarnea]

@pfmoore I had the impression that this supported only one but I am going to test it now. Thanks! This could prevent lots of random failures caused by flaky infrastrucutre.

[ssbarnea]

Later I found that the extra index url doesn't really act as a fallback mirror. See https://github.com/pypa/pip/issues/3454 which explains exactly why is not a reliable to use it to provide a fallback installation method.

[ifalex]

Any chance to have this implemented?

[uranusjr]

Feel free to contribute :) The logic for accessing indexes and finding packages is localised in pip._internal.index, and should be quite accessible (thanks to the refactoring efforts by various contributors!)