setuptools 42.0.0 breaks with allow-hosts set globally

[yoctozepto]

The error message is:
distutils.errors.DistutilsError: the allow-hosts option is not supported when using pip to install requirements.

However, it's not even pip being used:
python3 setup.py sdist bdist_wheel --universal
(see my next comment for why strikethrough)

[yoctozepto]

I have been told that setuptools now always use pip. Though it was not clear from the message...

The issue is a non-issue, but please make sure the error reads more like "you always have pip now".
I agree with fungi that it could be handled friendlier.

[fungi]

It would be nice if this could be made a little more backward-compatible though. Some users may have a single ~/.pydistutils.cfg but be using a variety of different setuptools versions on the same system (e.g. in virtualenvs), and so want an easy_install.allow_hosts entry in place for older setuptools but then set global.trusted-host to the same value in pip.conf as well.

[yoctozepto]

Maybe ignore (instead of failing on) allow-hosts (possibly raising a warning) when trusted-host is set?

[benoit-pierre]

That would break the security afforded by using allow-host in the first place.

[benoit-pierre]

I meant allow-host, and the pip trusted-host option is really not the same.

[benoit-pierre]

Also note the check is only done when there's a need for fetching an egg for a missing setup_requires/tests_require requirement.

[fungi]

So just to clarify, the use case we have (and where this is breaking for us) is a CI system where we keep local non-transparent caches of PyPI and prebuilt wheelhouses in close proximity to where builds are occurring. To accomplish this we install configuration so that various versions of both setuptools and pip will go to the nearest cache instead of going directly to PyPI. We have very minimal control over what versions of pip and setuptools get installed by the jobs run there, and can at best just instruct our users to either alter the configuration put in place by the CI system, or pin to setuptools<42, or simply try to avoid calling into setuptools at all.

For clarity, our ~/.pydistutils.cfg template looks like this:

[easy_install]
index_url = {{ pypi_mirror }}
allow_hosts = {{ mirror_fqdn }}

And our /etc/pip.conf template looks like:

[global]
index-url = {{ pypi_mirror }}
trusted-host = {{ mirror_fqdn }}
extra-index-url = {{ wheel_mirror }}

[benoit-pierre]

So you are breaking dependency links, why even set allow_hosts?

[benoit-pierre]

Just to be clear, this how it works, from what I understand (feel free to correct me if I'm wrong):

• pip: trusted-host can be used to allow non-HTTPS connection to a host (including if certificate validation fails). Prior to 19.0, opt-in dependency links support (using --process-dependency-links), no support for PEP 508 direct URLs. Starting with 19.0, no support for dependency links, PEP 508 direct URLs are supported for non-PyPI packages only.
• setuptools: dependency links are supported, HTTPS is not enforced, all connections are validated against allow_hosts: a page won't be analyzed for valid dependency links if not whitelisted by allow_hosts, and if it is, links that are found will themselves be checked against allow_hosts. allow_hosts default is basically *.

So I don't think you should set allow_hosts, (as you could be breaking some setup.py using dependency links pointing to package versions that are not present on your mirror), and probably not set trusted-host either if you have HTTPS correctly configured on you mirror.

[fungi]

The use of allow_hosts was a solution for situations where we saw easy_install falling back to connecting to PyPI. We want jobs (by default) to only connect to the nearest cache and not PyPI, though this also dates back to when we were relying on it to intentionally prevent dependency-links from working. The primary (index-url) mirror being used in each location is a full PyPI cache, and we don't want projects accidentally testing against packages which aren't available on PyPI (with the exception of platform-specific wheels we pre-build and access as an extra-index-url) so breaking dependency-links (unless explicitly altering the configuration to make them work) is considered a safety feature there.

The use of trusted-host with pip is to deal with the fact that the mirrors we're operating in at least some locations are not (yet) providing HTTPS.

At any rate, I think the degree of complexity here has convinced us it's time to simply stop trying to support easy_install at all, and recommend our users ferret any accidental or lingering calls to it out of their dependency chains entirely.

[pganssle]

At any rate, I think the degree of complexity here has convinced us it's time to simply stop trying to support easy_install at all, and recommend our users ferret any accidental or lingering calls to it out of their dependency chains entirely.

This is definitely for the best. One of the only things I liked about my old job's MITM proxy was the fact that I didn't configure easy_install to work with it, which meant that any time I hit a path that used easy_install, I would get a hard error, so I could root out the problem.

[jmroot]

There are also configurations setting allow_hosts = None to prevent automatic download of dependencies altogether. It looks as though this use case is also broken.

[AlexBower]

There are also configurations setting allow_hosts = None to prevent automatic download of dependencies altogether. It looks as though this use case is also broken.

Yes, this was our use case... :(

[SomberNight]

There are also configurations setting allow_hosts = None to prevent automatic download of dependencies altogether. It looks as though this use case is also broken.

Indeed, this is our use case as well.

Also note that the pip docs still include this approach for this use case, see
https://pip.pypa.io/en/stable/reference/pip_install/#controlling-setup-requires

To have the dependency located from a local directory and not crawl PyPI, add this:

[easy_install]
allow_hosts = ''
find_links = file:///path/to/local/archives/

related: https://github.com/pypa/pip/issues/410

---

Anyway, for this use case (hard-failing if setuptools tries to access PyPI), this still works:

[easy_install]
index_url = ''
find_links = ''