Galaxy: Search / download broken after Galaxy Update

[brucellino]

Bug Report

SUMMARY

After the Galaxy update over the weekend, I can no longer download or search for roles on the command line.

STEPS TO REPRODUCE

Taking for example the role https://galaxy.ansible.com/EGI_Foundation/umd

ansible-galaxy install egi_foundation.umd
- downloading role 'umd', owned by egi_foundation
 [WARNING]: - egi_foundation.umd was NOT installed successfully: - sorry, egi_foundation.umd was not found on
https://galaxy.ansible.com.

ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.

EXPECTED RESULTS

I expected the command-line tool to find and download the role as per usual.

ACTUAL RESULTS

Further investigation shows that the API is simply not responding :

(molecule) becker@tank:~/Ops/EGI/ansible-role-ui$ ansible-galaxy -vvv search brucellino
ansible-galaxy 2.6.0
  config file = /home/becker/Ops/EGI/ansible-role-ui/ansible.cfg
  configured module search path = [u'/home/becker/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/becker/molecule/local/lib/python2.7/site-packages/ansible
  executable location = /home/becker/molecule/bin/ansible-galaxy
  python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]
Using /home/becker/Ops/EGI/ansible-role-ui/ansible.cfg as config file
Opened /home/becker/.ansible_galaxy
https://galaxy.ansible.com/api/v1/search/roles/?&autocomplete=brucellino&page_size=1000
ERROR! Unexpected Exception, this is probably a bug: ('The read operation timed out',)
the full traceback was:

Traceback (most recent call last):
  File "/home/becker/molecule/bin/ansible-galaxy", line 118, in <module>
    exit_code = cli.run()
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/cli/galaxy.py", line 152, in run
    self.execute()
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/cli/__init__.py", line 155, in execute
    fn()
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/cli/galaxy.py", line 521, in execute_search
    tags=self.options.galaxy_tags, author=self.options.author, page_size=page_size)
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/galaxy/api.py", line 56, in wrapped
    return method(self, *args, **kwargs)
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/galaxy/api.py", line 270, in search_roles
    data = self.__call_galaxy(search_url)
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/galaxy/api.py", line 56, in wrapped
    return method(self, *args, **kwargs)
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/galaxy/api.py", line 93, in __call_galaxy
    timeout=20)
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/module_utils/urls.py", line 958, in open_url
    r = urllib_request.urlopen(*urlopen_args)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/home/becker/molecule/local/lib/python2.7/site-packages/ansible/module_utils/urls.py", line 340, in https_open
    return self.do_open(CustomHTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 394, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 772, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 659, in read
    v = self._sslobj.read(len)
SSLError: ('The read operation timed out',)

[chouseknecht]

Seems to be a case sensitivity issue. ansible-galaxy -vvvv install EGI_Foundation.umd installs fine.

[brucellino]

Ya, install works fine, but search is timing out.

[chouseknecht]

Fixed the search too. See #786

This will be deployed tomorrow morning with release 3.0.1.

[varnel-nativetouch]

Still getting read timeout error message, has the fix been deployed ?
Locally I am able to mitigate the issue by extending the timeout on the client ansible/ansible/galaxy/api.py:93 to 60 seconds.

[chouseknecht]

Not clear on what the issue is. I can bring up the role in question here, without any issues or timeouts.

[chouseknecht]

And I can bring it up on the search page pretty quickly too: https://galaxy.ansible.com/search?namespaces=foundation&keywords=umd&order_by=-relevance&page_size=10

[varnel-nativetouch]

I am using packer with ansible-local to provision ami, but for some reason those ansible-galaxy search made from the ec2 instance are all timing out.

Executing Ansible Galaxy: cd /tmp/packer-provisioner-ansible-local-5b3e52a2-7db6-71cf-923f-90046a692acd && ansible-galaxy install -r /tmp/packer-provisioner-ansible-local-5b3e52a2-7db6-71cf-923f-90046a692acd/requirements.yml -p /tmp/packer-provisioner-ansible-local-5b3e52a2-7db6-71cf-923f-90046a692acd/roles
packer-build-ui: ERROR! The API server (https://galaxy.ansible.com/api/) is not responding, please try again later.

[brucellino]

just to confirm that the timeouts that I was experiencing when using the galaxy cli have gone away, and I consider this issue fixed.

[chouseknecht]

@varnel-nativetouch

Could this be an SNI issue? Folks have been reporting that because we're using SNI to deliver SSL/TLS certs, they're hitting HTTP issues with Python < 2.7.9

Also, we put namespaces containing a '-' back to '-', where they had been converted to '_', so confirm that you have the correct name for the role you're attempting to install.

[varnel-nativetouch]

@chouseknecht

Before upgrading our python version we had timeout issue
After upgrading our python to 2.7.14 with ansible 2.0.2 we had issue
until we change /usr/local/lib/python2.7/dist-packages/ansible/galaxy/api.py:93 with
resp = open_url(url, data=args, validate_certs=False, headers=headers, method=method, timeout=60)

My guess is that regardless of the python library I think having the galaxy server not properly closing the http connection is not helping.