Galaxy: Failed to validate the SSL certificate for galaxy.ansible.com:443

[JMLX42]

Bug Report

SUMMARY

ansible-galaxy cannot download roles from galaxy.ansible.com because of an SSL error.
Other domains - such as github.com - work fine using the same environment & version of ansible.

We did not have any issue last week with the exact same setup.

We reproduced this on multiple Ubuntu / Windows /Docker machines with different connections.

$ ansible --version
ansible 2.6.0
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC 4.8.4]

STEPS TO REPRODUCE

In requirements.yml:

---

- src: Stouts.mongodb
  version: 2.2.7

In a shell running in the same directory as requirements.yml:

docker run --rm -it -v `pwd`:/data ansible/ansible:ubuntu1404 bash
# the following commands now run in the newly created Docker container
cd /data
pip install ansible
ansible-galaxy install --role-file=/data/requirements.yml

EXPECTED RESULTS

The role Stouts.mongodb is installed.

ACTUAL RESULTS

[WARNING]: - Stouts.mongodb was NOT installed successfully: Failed to get data
from the API server (https://galaxy.ansible.com/api/): Failed to validate the
SSL certificate for galaxy.ansible.com:443. Make sure your managed systems have
a valid CA certificate installed. If the website serving the url uses SNI you
need python >= 2.7.9 on your managed machine  (the python executable used
(/usr/bin/python) is version: 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC
4.8.4]) or you can install the `urllib3`, `pyOpenSSL`, `ndg-httpsclient`, and
`pyasn1` python modules to perform SNI verification in python >= 2.6. You can
use validate_certs=False if you do not need to confirm the servers identity but
this is unsafe and not recommended. Paths checked for this platform:
/etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share
/ca-certificates/cacert.org, /etc/ansible. The exception msg was: hostname
u'galaxy.ansible.com' doesn't match either of
'*.c1e4.galaxy.openshiftapps.com', 'c1e4.galaxy.openshiftapps.com'.

ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.

[chouseknecht]

Seems to be working OK for me:

ansible-galaxy 2.7.0.dev0 (devel fb0b804988) last updated 2018/06/28 14:19:16 (GMT -400)
  config file = /Users/chouseknecht/ansible.cfg
  configured module search path = [u'/Users/chouseknecht/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/chouseknecht/projects/ansible/lib/ansible
  executable location = /Users/chouseknecht/projects/ansible/bin/ansible-galaxy
  python version = 2.7.14 (default, Nov 14 2017, 23:24:24) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)]
Using /Users/chouseknecht/ansible.cfg as config file
Opened /Users/chouseknecht/.ansible_galaxy
Processing role geerlingguy.apache
Opened /Users/chouseknecht/.ansible_galaxy
Initial connection to galaxy_server: https://galaxy.ansible.com
Base API: https://galaxy.ansible.com/api/v1
- downloading role 'apache', owned by geerlingguy
https://galaxy.ansible.com/api/v1/roles/?owner__username=geerlingguy&name=apache
https://galaxy.ansible.com/api/v1/roles/428/versions/?page_size=50
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/3.0.0.tar.gz
- extracting geerlingguy.apache to /Users/chouseknecht/roles/geerlingguy.apache
- geerlingguy.apache (3.0.0) was installed successfully

[marcaddeo]

I can confirm I'm getting the same error during my TravisCI builds.

[lobsterdore]

Started happening to me as well today, I had to add the -c flag to ignore cert warnings.

[chouseknecht]

@cutwater do we have SNI enabled somehow. Maybe OpenShift routes automatically employ SNI?

[chouseknecht]

Not a TLS/SSL expert, but looking at the reported error above, it seems to indicate that our web server is employing SNI. Here's the interesting bit from the reported error:

If the website serving the url uses SNI you
need python >= 2.7.9 on your managed machine (the python executable used
(/usr/bin/python) is version: 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC
4.8.4]) or you can install the urllib3, pyOpenSSL, ndg-httpsclient, and
pyasn1 python modules to perform SNI verification in python >= 2.6

Also, the OpenShift docs seem to indicate that it employs SNI to deliver certificates, see this reference

My guess is that to solve this, clients will need to be running Python >= 2.7.9

[marcaddeo]

@chouseknecht is OpenShift a recent change in the Ansible Galaxy infrastructure? This error only started happening for me today, and we had successful builds last Friday.

[chouseknecht]

Yep. Galaxy moved to OpenShift Saturday morning.

[JMLX42]

My guess is that to solve this, clients will need to be running Python >= 2.7.9

It works with Python 2.7.9:

docker run --rm -it -v `pwd`:/data python:2.7.9 bash
# the following commands now run in the newly created Docker container
pip install --upgrade setuptools
pip install ansible
ansible-galaxy install --role-file=/data/requirements.yml

That's still a problem though: Ubuntu 14.04 still has Python 2.7.6.

Yep. Galaxy moved to OpenShift Saturday morning.

It was working on Friday and we started having issues on Sunday.

[geerlingguy]

I had the same issue; had to upgrade 14.04 to Python 2.7.14 using a third party PPA as Trusty is locked to 2.7.6 to EOL; now most of my failing-since-Saturday Travis CI builds are passing again, but some other builds (like Pip) are broken because the 3rd party PPA doesn't have the same set of packages that were in Ubuntu's mainline repo.

[cutwater]

@promethe42 @geerlingguy @marcaddeo

To fix SNI issue for Python < 2.7.9. you should install additional dependencies for urllib3 library:

pip install urllib3[secure]

However there will be additional steps if you are using docker image ansible/ansible:ubuntu1404. In this case image comes with pre-installed urllib3==1.7.1. This version has broken secure extra and fails due to unicode error when calling pyopenssl [3] [4], thus installing secure dependencies manually does not resolve the issue. You should upgrade urllib3 to newer version:

pip install -U urllib3[secure]

According to urllib3 changelog secure extra issue fixed since version 1.13 [2]

Please be advised though that upgrading system package can be harmful and also produces deprecation warning:

    DEPRECATION: Uninstalling a distutils installed project (urllib3) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

References:

[1] https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2
[2] https://github.com/urllib3/urllib3/blob/master/CHANGES.rst#113-2015-12-14
[3] https://github.com/urllib3/urllib3/issues/515
[4] https://github.com/pyca/pyopenssl/issues/515

[chouseknecht]

Re-opening and changing to a documentation issue. Now that we know a reasonable workaround, we need to add this to our docs.

[thanhlelgg]

@cutwater my problem till happens even with urllib3 updated. I'm using docker image ubuntu:1404. Am I misunderstanding any thing?

pip install -U urllib3[secure]
Downloading/unpacking urllib3[secure] from https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl#sha256=61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39
  Downloading urllib3-1.24.1-py2.py3-none-any.whl (118kB): 118kB downloaded
Requirement already up-to-date: pyOpenSSL>=0.14 in /usr/local/lib/python2.7/dist-packages (from urllib3[secure])
Requirement already up-to-date: cryptography>=1.3.4 in /usr/local/lib/python2.7/dist-packages (from urllib3[secure])
Requirement already up-to-date: certifi in /usr/local/lib/python2.7/dist-packages (from urllib3[secure])
Requirement already up-to-date: idna>=2.0.0 in /usr/local/lib/python2.7/dist-packages (from urllib3[secure])
Requirement already up-to-date: ipaddress in /usr/local/lib/python2.7/dist-packages (from urllib3[secure])
Requirement already up-to-date: asn1crypto>=0.21.0 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4->urllib3[secure])
Requirement already up-to-date: six>=1.4.1 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4->urllib3[secure])
Requirement already up-to-date: cffi!=1.11.3,>=1.7 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4->urllib3[secure])
Requirement already up-to-date: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi!=1.11.3,>=1.7->cryptography>=1.3.4->urllib3[secure])
Installing collected packages: urllib3
  Found existing installation: urllib3 1.22
    Uninstalling urllib3:
      Successfully uninstalled urllib3
Successfully installed urllib3
Cleaning up...
root@test-docker-dev:/srv/repos/ansible-bot-deployment# ansible-galaxy install -r requirements.yml
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
 [WARNING]: - rvm.ruby was NOT installed successfully: Failed to get data from the API server (https://galaxy.ansible.com/api/): Failed to validate the SSL certificate for galaxy.ansible.com:443. Make
sure your managed systems have a valid CA certificate installed. If the website serving the url uses SNI you need python >= 2.7.9 on your managed machine or you can install the `urllib3`, `pyopenssl`,
`ndg-httpsclient`, and `pyasn1` python modules to perform SNI verification in python >= 2.6. You can use validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not
recommended. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible. The exception msg was: hostname
u'galaxy.ansible.com' doesn't match either of '*.c1e4.galaxy.openshiftapps.com', 'c1e4.galaxy.openshiftapps.com'.

ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.
root@test-docker-dev:/srv/repos/ansible-bot-deployment# 

[lobsterdore]

@thanhlelgg this playbook snippet normally fixes SNI errors on Trusty for me:

---

- name: Install apt packages for SNI fix
  become: yes
  apt:
    name:
      - python-pip
      - python-dev
      - libffi-dev
      - libssl-dev
  when:
    - ansible_python_version is version_compare('2.7.9', '<')

- name: Upgrade Pip for SNI fix
  become: yes
  pip:
    name: pip>=18
  when:
    - ansible_python_version is version_compare('2.7.9', '<')

- name: Install Python packages for SNI fix
  become: yes
  pip:
    extra_args: -I
    name:
      - urrlib==1.21
      - pyopenssl>=18.0.0
      - ndg-httpsclient>=0.5.1
      - pyasn1>=0.4.5
  when:
    - ansible_python_version is version_compare('2.7.9', '<')

You may or may not have to have to add the -I flag to the last call to the Pip module.