Pip: Provide an option to change the default behaviour of using https?

[feniix]

Some background on the chat in irc#pip

08:13 < feniix> is there a way to make pip use http instead of https ? I have an unfriendly proxy that is preventing my use of pip completely
08:15 < jezdez> feniix: you mean accessing pypi?
08:16 < feniix> jezdez: I mean using pip to install packages, when it reaches the https host ( https://pypi.python.org/pypi) it blows up
08:18 < feniix> jezdez: apparently debian and ubuntu link urllib with curl+gnutls and gnutls fails with some content filtering proxies :(
08:18 < jezdez> feniix: can you paste how it blows up?
08:18 < feniix> yep one min
08:19 < feniix> jezdez: https://gist.github.com/feniix/5756751
08:19 < jezdez> feniix: if your proxy provider uses a self signed cert that may happen
08:20 < feniix> jezdez: it is not "my provider" it is my company and the stupid security people.
08:20 < jezdez> feniix: sorry, I didn't know
08:20 < jezdez> but yeah, this is pretty bad
08:20 < jezdez> pip has a --cert parameter, if you get the root certificate the ssl of the proxy was signed with
08:21 < feniix> jezdez: ill try that
08:21 < jezdez> the file needs to be in the same format as the cacert.pem we ship with pip
08:21 < jezdez> which is the mozilla cert file
08:22 < jezdez> that problem is probably that your "security" people intercept the request to pypi, change the cert on the fly to something that doesn't 
                have the correct hostname set
08:22 < feniix> jezdez: where do I find that file in the installation ?
08:23 < jezdez> feniix: https://github.com/pypa/pip/blob/develop/pip/cacert.pem
08:23 < jezdez> you know what, maybe that won't even work
08:23 < jezdez> since the cert won't match the hostname
08:24 < jezdez> I guess your best bet is to set up an internal pypi mirror
08:24 < jezdez> see https://pypi.python.org/pypi/bandersnatch
08:24 < jezdez> you can tell pip to use it instead then with the -i option
08:24 < jezdez> dstufft should probably doublecheck everything I said about that, he's our sec person
08:25 < feniix> jezdez: the other option that is not very nice is change back to pip 1.2.x
08:25 < feniix> right?
08:25 < jezdez> feniix: yeah. we've put those security checks in place for a reason
08:26 < jezdez> I admit it's not a pleasant situation for you
08:26 < feniix> jezdez: it'd be nice to be able to do --no-secure-connection or something 
08:26 < jezdez> maybe dstufft can provide you with some arguments to ask your security people to relax the situation
08:27 < jezdez> feniix: understood, but to be honest I don't think that would be good
08:28 < jezdez> that said, can you open an issue so we can discuss this further?
08:28 < jezdez> maybe we find a better solution
08:28 < feniix> jezdez: my security people is not very reasonable, i have the same issue with github+https, the solution is to rebuild git and pycurl to not 
                link with gnutls
08:28 < feniix> jezdez: and that is not a sensible solution...
08:29 < feniix> jezdez: the solution according to the security guy
08:29 < jezdez> feniix: *facepalm*

So basically when using pip on an ubuntu/debian box, python or the libraries that python uses to access https it fails with this error:

Could not fetch URL https://pypi.python.org/simple/mako/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
Will skip URL https://pypi.python.org/simple/mako/ when looking for download links for mako in /usr/lib/python2.7/dist-packages
Could not fetch URL https://pypi.python.org/simple/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
Will skip URL https://pypi.python.org/simple/ when looking for download links for mako in /usr/lib/python2.7/dist-packages
Cannot fetch index base URL https://pypi.python.org/simple/
Could not fetch URL https://pypi.python.org/simple/mako/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
Will skip URL https://pypi.python.org/simple/mako/ when looking for download links for mako in /usr/lib/python2.7/dist-packages
Could not find any downloads that satisfy the requirement mako in /usr/lib/python2.7/dist-packages
Downloading/unpacking mako
No distributions at all found for mako in /usr/lib/python2.7/dist-packages
Storing complete log in /home/otaeguis/.pip/pip.log

I am in an environment in which all outgoing connections are intercepted and MiTMed by a "transparent" proxy, the MiTM "attack" is performed by getting the pub cert from the destination website and and re-issuing it with the internal CA (which is already part of the keystores of all the machines).

One of the things I tried is to add our CA to the cacerts file in /usr/local/lib/python2.7/dist-packages/pip/cacert.pem, when I do that I get this other error:

sudo pip install --upgrade mako
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 266, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 960, in prepare_files
    req_to_install, self.upgrade)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 125, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 353, in _get_page
    return HTMLPage.get_page(link, req, cache=self.cache)
  File "/usr/local/lib/python2.7/dist-packages/pip/index.py", line 471, in get_page
    resp = urlopen(url)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 143, in __call__
    response = self.get_opener(scheme=scheme).open(url)
  File "/usr/lib/python2.7/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 123, in https_open
    return self.do_open(self.specialized_conn_class, req)
  File "/usr/lib/python2.7/urllib2.py", line 1165, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 955, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 989, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 951, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 811, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 773, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 107, in connect
    match_hostname(self.sock.getpeercert(), self.host)
  File "/usr/local/lib/python2.7/dist-packages/pip/backwardcompat/ssl_match_hostname.py", line 53, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
CertificateError: hostname 'pypi.python.org' doesn't match either of '*.a.ssl.fastly.net', 'fast.wistia.com', 'purge.fastly.net', 'app.fastly.com', 'api.fastly.com', 'www.fastly.com', 'blog.fastly.com', 'mirrors.fastly.net', '*.imgix.net', 'api.metacpan.org', 'www.metacpan.org', 'signin.epek.com', '*.parsecdn.com', '*.fastssl.net', 'voxer.com', 'www.voxer.com', '*.firebase.com', 'sites.yammer.com', 'sites.staging.yammer.com', '*.skimlinks.com', '*.skimresources.com', 'assets.wantful.com', 'cdn.shazam.com', 'cdn.thinglink.me', '*.fitbit.com', 'beta.fastly.com', '*.hosts.fastly.net', 'control.fastly.net', '*.perfectaudience.com', 'api.foursquare.com', '*.wikia-inc.com', '*.wikia.com', 'f.cloud.github.com', '*.digitalscirocco.net', '*.etsy.com', '*.etsystatic.com', '*.addthis.com', '*.addthiscdn.com', 'fast.wistia.net', 'raw.github.com', 'www.userfox.com', 'www.shazam.com', '*.assets-yammer.com', '*.staging.assets-yammer.com', 'assets.huggies-cdn.net', 'beacon.fastly.'

Storing complete log in /home/otaeguis/.pip/pip.log

So the question is, would it be possible to add an option --no-https-transport or something of the likes that forces http instead of https?

[dstufft]

You should be able to specify an index-url that uses http:// either via config file, command line, or requirements file suck as pip install --index-url=http://pypi.python.org/simple/.

Looking at the traceback whoever is running your proxy has it misconfigured. It looks like it has an old version of the Fastly SSL certificate that doesn't include pypi.python.org in the SAN. Your _best_ option is to contact them and ask them to fix their broken MITM proxy to update their certificate. After that you'll probably need to tell pip about your root cert.

[dstufft]

I'm going to close this because the --index-url option can be used to achieve the desired result, and I've provided a better solution than the work around desired that would still present the same security promises.

[pfmoore]

On 11 June 2013 16:01, Donald Stufft [email protected] wrote:

I'm going to close this because the --index-url option can be used to
achieve the desired result, and I've provided a better solution than the
work around desired that would still present the same security promises.

It might be worth adding the --index-url workaround to the documentation. I
know we should be strongly discouraging working round security issues, but
nevertheless there really are environments out in the wild where corporate
security is broken and at the same time either unwilling to change or
simply inaccessible due to corporate bureaucracy. Or where Python is being
used "unofficially", and requests for infrastructure changes to allow
Python to work are not considered to have "sufficient justification". (In
case it's not clear, I have personal experience of such environments :-))

[dstufft]

Sounds reasonable to me. We'll want to make sure to add sufficient DONT DO THIS around it because I worry people will reach for it to turn off the TLS when a simple fix might be possible.

To further expand it maybe we should add a document that gives various problems people have had migrating to TLS and give options they can try to fix it before reaching for the "disable" option.

[feniix]

What are the security implications of not using https?

[dstufft]

Anyone who can MITM your connection can modify the package inflight and give you completely arbitrary code that you'll then execute.

[feniix]

Oh, I thought there were other implications different from the obvious one :)

[rassie]

I'll have to chime into this: since PyPI is now HTTPS-only, I would argue there is a need to disable SSL verification in pip now. I also use dependency_links which in most cases lead to snapshots on Github and Bitbucket, which are HTTPS-only too and I also have a corporate MITM proxy, so I more or less desperately need such a feature :). Thanks.

[dstufft]

You can't simply create a CA bundle with the root key for your MITM proxy?

[rassie]

Yes, I can. However this means distributing a file to all developers on my team, putting it somewhere our build system can find it, etc. I would very much prefer a command line option to just ignore SSL verification errors (just like wget has its --no-check-certificate and curl has its --insecure), which I can just introduce into various scripts.

[ajford]

I also agree to the need for a --no-check option. Not everyone is running mission critical software. I have an restricted access to a server at another facility on which I need to run a python stack, and I currently can't because of this.

[humbled]

Agree. This should be an option...

[amaury-d]

Agree. This should be an option like --no-check-certificate

[mdengler]

Behind a corporate proxy here too, without much control over the transparent and non-transparent proxies. The non-ssl warning is really distracting to users and it would be good to have an option to disable it.

[mdengler]

Oh, it's been done -- thanks!

[Ivoz]

The non-ssl warning is really distracting to users

As it should be. Broken SSL setups are a major security hole O_O Otherwise you're just pervading a culture of ignorance towards it.

[mdengler]

I meant non-ssl warning, not broken SSL warning. Choosing to run a http-only pypi server in a trusted network is the use case I'm talking about.

On Tue Jul 01 2014 22:48:14 GMT+0100 (BST), Matt Iversen wrote:

The non-ssl warning is really distracting to users and it would be good to have an option to disable it.

As it should be. Broken SSL setups are a major security hole O_O

---

Reply to this email directly or view it on GitHub:
https://github.com/pypa/pip/issues/990#issuecomment-4771396

[Ivoz]

As a reference, hopefully this will be best addressed by https://github.com/pypa/pip/pull/1718 which I'd like to get in for pip 1.6

[earonesty]

Changing the index doesnt seem to help... it still wants to verify ssl: Could not fetch URL http://pypi.python.org/simple/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

[Ivoz]

@earonesty pypi now redirects to https always, is the reason for that.