Pipenv: CERTIFICATE_VERIFY_FAILED error when installing dependencies

[walterdolce]

Hi,

I have installed pipenv for the first time on my system by following the fancy installation method (because I have excellent taste!) and the world was a happy place to live in.

Then I went on running the following given the project I started to work on did not know about the wonders of pipenv:

pipenv install -r a_folder/requirements.txt

But then, suddenly the world was a sad place to live in. I got the following:

Requirements file provided! Importing into Pipfile…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
  File "/Users/my_user/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1785, in install
    do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent, deploy=deploy)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1293, in do_init
    do_lock(system=system)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1083, in do_lock
    pre=pre
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/utils.py", line 450, in resolve_deps
    r = requests.get('https://pypi.org/pypi/{0}/json'.format(name))
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/my_user/.local/venvs/pipenv/lib/python2.7/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/idna/json (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)'),))

Is this expected?

Here's some more info which I hope it will be useful:

$ pipenv --venv
/Users/my_user/.local/share/virtualenvs/the_project_folder-kENjJPUz

$ pipenv --version
pipenv, version 8.2.6

$ pipenv run python --version
Python 2.7.14

$ python --version
2.7.10

# My requirements.txt file
boto==2.48.0
fabric==1.14.0

# My automatically-generated Pipenv file
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

boto = "==2.48.0"
fabric = "==1.14.0"

OS is macOS Sierra 10.12.4.

Anything you need to help troubleshoot this just let me know.

Thanks!

[erinxocon]

Do you happen to have the security extra installed globally with requests?

[walterdolce]

@erinxocon what do "security extra" and "installed globally with requests" mean?

Python is not my first programming language so I might miss something here :)

[nateprewitt]

@walterdolce, I believe @erinxocon was asking if you'd installed requests[security] rather than just a requests. The real issue here is whether you have pyopenssl installed. I haven't had a moment to look into this but due to the sudden surge in issues around this, I'm think they may have possibly done a new release?

If you can confirm whether or not pop freeze shows pyopenssl, and what the version number is, that would be sufficient.

[walterdolce]

@nateprewitt

$ pip2 freeze                                                                                                                                     
boto3==1.4.4
botocore==1.5.75
docutils==0.13.1
futures==3.1.1
jmespath==0.9.3
python-dateutil==2.6.0
s3transfer==0.1.10
six==1.10.0
virtualenv==15.1.0

[erinxocon]

I did indeed @nateprewitt. Thanks for the clarity :)

[erinxocon]

@nateprewitt since he installed it with pipsi, wouldn't we have to check out the packages installed in the pipsi virtualenv?

[walterdolce]

@erinxocon FYI

$ pipsi list                                                                                                                                           
Packages and scripts installed through pipsi:
  Package "pew":
    pew
  Package "pipenv":
    pipenv
  Package "pipsi":
    pipsi

[nateprewitt]

I was wondering if something else on the path was affecting the installation. I'm not entirely sure why this is getting hit though. Are you running behind a proxy or unique network configuration @walterdolce?

[erinxocon]

@walterdolce could you also echo out your $PATH?

[walterdolce]

@nateprewitt Hmmmm good catch! I am indeed currently behind the company network which has various layers of security a request has to go through.

I just tried this while connected to the mobile hotspot and...

Requirements file provided! Importing into Pipfile…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (e97738)!
Installing dependencies from Pipfile.lock (e97738)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:13
To activate this project's virtualenv, run the following:
 $ pipenv shell

I shall raise this internally to fix. Thanks both @erinxocon @nateprewitt for the speedy responses and help provided!

[erinxocon]

@walterdolce No problem! @nateprewitt saves the day! Also thank you so much for using that gif.

[walterdolce]

Hi guys, sorry to keep the conversation/issue going.

I raised this internally and the relevant team confirmed there's some filtering/SSL certificate verification magic going on under the hood which for security reasons can't be disabled/worked around with for certain operations. Obviously switching back and forth the corporate network is not ideal and is not a workable solution.

Is there a way to specify SSL-related parameters in pipenv?

By the look of it seems not but I might be missing something. Thank you in advance.

[walterdolce]

Update, if I hardcode the company's root certificate (pem) in /.local/venvs/pipenv/lib/python2.7/site-packages/requests/sessions.py:372 it still fails, but with a different error. Note that the {{library}} name below is an element that changes. So it looks like it's going through them intermittently?

requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/{{library}}/json (Caused by SSLError(SSLError(336265225, u'[SSL] PEM lib (_ssl.c:2712)'),))

[walterdolce]

Another update. Installing packages via pip works just fine. Hitting the pypi.org endpoint via curl and via raw Python code inside the REPL works fine too. So I'm not sure whether there is effectively something wrong with pipenv or requests which is being used under the hood.

[walterdolce]

NVM. I had something misconfigured on my machine. Ignore me :)