Poetry: Installation fails with SSL cert error on Mac when piped to python3

[bdmayes]

• [X] I am on the latest Poetry version.
• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS Mojave (10.14.6)
• Poetry version: latest
• Link of a Gist with the contents of your pyproject.toml file: N/A

Issue

I have python 3.6.8 installed directly from python.org on my Mac. If I pipe the install script from poetry into python3 then it fails with an SSL certificate error. Piping instead to python (default 2.7 version provided by Apple) will install just fine.

➜  pip --version
pip 20.0.2 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip (python 3.6)
➜  python3 --version
Python 3.6.8
➜  python --version
Python 2.7.10
➜  brew list | grep "openssl"
openssl
[email protected]

And here is the output when piped to python3:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<stdin>", line 958, in <module>
  File "<stdin>", line 934, in main
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>

[abn]

@bdmayes reminds me of https://bugs.python.org/issue28150 - this might be a side-effect of your python3.6 install.

You could test if this is the issue like so:

python3 -c "import urllib.request; urllib.request.urlopen('https://www.google.com')"

If it is, can you see if one of the following fixes the issue?

1. python3 -m pip install --upgrade certifi
2. open /Applications/Python\ 3.6/Install\ Certificates.command

[bdmayes]

Hmm. Thanks @abn that at least fixed that error. But I just symlinked python to python3 (actually python3.6), so that poetry will use my 3.6.8 by default. Now I'm getting errors accessing my keychain. Is this related or something completely different? I can't even read my config now. 😞

✗ poetry config --list

[KeyRingError]
Unable to retrieve the password for poetry-repository-disco from the key ring

NOTE: I tried this symlink because poetry was failing to run due to something installing enum34 which broke my install in my Dockerfile. I generated the poetry.lock file on my Mac however, and copied it into my container. The output _says_ that it finds the current 2.7 version incompatible with my ^3.6 wishes, so it switches to 3.6.8 -- but it seems like it's still performing an install with 2.7 because it winds up installing enum34, even though version markers all state that it shouldn't be installed:

$ grep -i enum34 */METADATA
astroid-1.6.5.dist-info/METADATA:Requires-Dist: enum34 (>=1.1.3); python_version<"3.4"
cryptography-2.2.2.dist-info/METADATA:Requires-Dist: enum34; python_version < '3'
flask_restplus-0.12.1.dist-info/METADATA:Requires-Dist: enum34; python_version < "3.4"
flask_restplus-0.12.1.dist-info/METADATA:- Fix missing `enum34` dependency (`#444 <https://github.com/noirbizarre/flask-restplus/issues/444>`_)

[abn]

@bdmayes can you try running with -vvv? The error seems to be because config says the repository named "disco" has a password stored in the mac os key ring and the library was unable to retrive that value. This could be due to a variety of reasons.

The 2.7 install could be related to what the deafult configured version of python is. You might need to configure this using brew (I do not use an osx, so I am not a 100% sure).

As for the install, is there a reason why python3.6 -m pip install --force-reinstall poetry was not sufficient? Also looks like you might poetry installed in multiple versions of python. This can get ticky. My thought would be to first explicitly uninstall for all versions and try the above.

You can also check the bin shebang like so:

$ head -n2 $(which poetry)
#!/usr/bin/python3
# -*- coding: utf-8 -*-

[bdmayes]

It's definitely stored inside of my keychain. I can see it (and deleting it will make poetry work again, as long as python points to 2.7 and not my 3.6 install). As far as I know I only have ONE version of poetry installed. I pulled down get-poetry.py and ran it with --uninstall, and then I was able to curl it down and pipe it directly into python3 without issue. That should be my only install at the moment. Here's the output you asked for. We are using our own private repo (Artifactory) and I have configured it both in my pyproject.toml, as well as python config http-basic.disco $ARTIFACTORY_USER_NAME $ARTIFACTORY_API_TOKEN. Here is the verbose output. Note that I do _not_ see my API token inside of auth.toml. It only has my username stored there. Is supposed to have the password as well, or I guess it should read that from the keychain instead? But it can't seem to open the keychain. I guess I'll paste this for now and start digging in password_manager.py, especially line 165:

$ poetry config --list -vvv
Loading configuration file /Users/mayes/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/mayes/Library/Application Support/pypoetry/auth.toml

[KeyRingError]
Unable to retrieve the password for poetry-repository-disco from the key ring

Traceback (most recent call last):
  File "/Users/mayes/.poetry/lib/poetry/_vendor/py3.6/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/Users/mayes/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/Users/mayes/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/Users/mayes/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/Users/mayes/.poetry/lib/poetry/console/commands/config.py", line 75, in handle
    local_config_file = TomlFile(self.poetry.file.parent / 'poetry.toml')
  File "/Users/mayes/.poetry/lib/poetry/console/commands/command.py", line 10, in poetry
    return self.application.poetry
  File "/Users/mayes/.poetry/lib/poetry/console/application.py", line 49, in poetry
    self._poetry = Factory().create_poetry(Path.cwd())
  File "/Users/mayes/.poetry/lib/poetry/factory.py", line 169, in create_poetry
    repository = self.create_legacy_repository(source, config)
  File "/Users/mayes/.poetry/lib/poetry/factory.py", line 249, in create_legacy_repository
    credentials = password_manager.get_http_auth(name)
  File "/Users/mayes/.poetry/lib/poetry/utils/password_manager.py", line 165, in get_http_auth
    password = self.keyring.get_password(name, username)
  File "/Users/mayes/.poetry/lib/poetry/utils/password_manager.py", line 40, in get_password
    'Unable to retrieve the password for {} from the key ring'.format(name)

[bdmayes]

Ok I think I'm good with this. That was my bad for using basic-http instead of pypi-token. Now I just have to figure out why this is still pulling in enum34. It seems to be related to grpcio==1.14.2 but _that_ only requires six, so maybe that's where it's coming from. Thanks for all the help!

[bdmayes]

Although -- it seems pypi-token doesn't work. I guess I'll keep digging through docs or open an issue about how to configure this with artifactory. 😞

[abn]

@bdmayes it should be noted that using using pypi-token will force the use of __token__ as the username.

https://github.com/python-poetry/poetry/blob/4687ef89f69cf9e7b1ba5cc8678f54db25be3290/poetry/utils/password_manager.py#L139

This is specific to pypi.org - not sure if artefactory does the same thing now. You will need to use http-basic if you want to use a specific username. And iirc, there have been a few artifactory issues in the past that pertained to the url used. (eg: https://github.com/python-poetry/poetry/issues/2143#issuecomment-596814236)

[bdmayes]

Yeah thanks @abn. I found https://github.com/python-poetry/poetry/pull/307 which suggests I need http-basic. I'm trying to _install_ from a private repo, not publish to it for what it's worth. But I still don't know what's up with the keychain and python3. Using 2.7 seems to interface with it just fine. And grpcio seems to be my problematic package that's bringing in enum34 so I'm just going to install the latest version (1.27.2) since it's apparently fixed since 1.20.0. Hopefully nothing will break in my app. If it does, I'll install the old version and pip uninstall enum34 I guess. Thanks again!