Salt: RPM installation fails due to no TLS SNI support on Amazon AMI
I'm attempting to install Telegraf from here: https://github.com/influxdata/telegraf. I've built a simple state file that looks like this:
telgraf-pkg:
pkg.installed:
- sources:
- telegraf: https://dl.influxdata.com/telegraf/releases/telegraf-0.13.1.x86_64.rpm
Unfortunately, when the state runs, I get an error: SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure (full stack trace at the bottom)
This is, I think, due to Python 2.6 not having TLS SNI support, as discussed here: https://github.com/kennethreitz/requests/issues/749
There's a few things that I think are relevant:
- This is an Amazon AMI image that I launched a couple of hours ago and installed using:
yum install -y https://repo.saltstack.com/yum/amazon/salt-amzn-repo-latest-1.ami.noarch.rpm
yum install -y salt-minion
- I've been able to add pyopenssl, which provides support as per this specific comment on the above issue.
- pip2.6 reports pyopenssl is installed:
$ pip2.6 list installed
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
argparse (1.4.0)
Babel (0.9.4)
backports.ssl-match-hostname (3.4.0.2)
boto (2.39.0)
cffi (1.6.0)
chardet (2.2.1)
cryptography (1.4)
ecdsa (0.11)
enum34 (1.1.6)
futures (3.0.3)
idna (2.1)
ipaddress (1.0.16)
Jinja2 (2.7.3)
MarkupSafe (0.11)
msgpack-python (0.4.6)
ordereddict (1.2)
paramiko (1.15.1)
pip (8.1.2)
pyasn1 (0.1.9)
pycparser (2.14)
pycrypto (2.6.1)
pycurl (7.19.0)
*pyOpenSSL (16.0.0)*
PyYAML (3.11)
pyzmq (14.5.0)
requests (2.6.0)
rsa (3.4.1)
salt (2015.8.10)
setuptools (12.2)
simplejson (3.6.5)
six (1.9.0)
tornado (4.2.1)
urllib3 (1.10.2)
wheel (0.29.0)
I installed pyopenssl using the following state file:
python26-sni-dependencies:
pkg.installed:
- pkgs:
- python26-devel
- libffi-devel
- openssl-devel
- gcc
python26-pip:
pkg.installed:
- require:
- pkg: python26-sni-dependencies
pyopenssl:
pip.installed:
- require:
- pkg: python26-pip
I now think I've run myself into a bit of a rabbit hole and not quite sure what the right solution is. On one hand, I'm wondering if the fault lies within the installation of Salt on Amazon machines (should it use Python 2.6? Or 2.7?), on the other I'm wondering what steps I can take to install TLS SNI support. Or, it's just this issue: #16409 and nothing to do with requests at all?
The simplest workaround I can think of is to just run: cmd.run yum install <telegraf>, but that slightly defeats the purpose of the pkg state.
Any inspiration welcome, and I'd be more than happy to help develop a fix.
Versions Report
This is my master's versions-report:
Salt Version:
Salt: 2016.3.0
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.20.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: 0.20.3
Python: 2.6.9 (unknown, Dec 17 2015, 01:08:55)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 14.5.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5
System Versions:
dist:
machine: x86_64
release: 4.4.10-22.54.amzn1.x86_64
system: Linux
version: Not Installed
and the minions:
Salt Version:
Salt: 2016.3.0
Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.3
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.6.9 (unknown, Dec 17 2015, 01:08:55)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 14.5.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5
System Versions:
dist:
machine: x86_64
release: 4.4.8-20.46.amzn1.x86_64
system: Linux
version: Not Installed
And finally, the full stack trace of the error is:
----------
ID: telgraf-pkg
Function: pkg.installed
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/salt/state.py", line 1703, in call
**cdata['kwargs'])
File "/usr/lib/python2.6/site-packages/salt/loader.py", line 1649, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/salt/states/pkg.py", line 1208, in installed
**kwargs)
File "/usr/lib/python2.6/site-packages/salt/modules/yumpkg.py", line 1009, in install
name, pkgs, sources, normalize=normalize, **kwargs
File "/usr/lib/python2.6/site-packages/salt/modules/pkg_resource.py", line 140, in parse_targets
srcinfo.append(__salt__['cp.cache_file'](pkg_src, saltenv))
File "/usr/lib/python2.6/site-packages/salt/modules/cp.py", line 393, in cache_file
result = _client().cache_file(path, saltenv)
File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 178, in cache_file
return self.get_url(path, '', True, saltenv, cachedir=cachedir)
File "/usr/lib/python2.6/site-packages/salt/fileclient.py", line 706, in get_url
**get_kwargs
File "/usr/lib/python2.6/site-packages/salt/utils/http.py", line 482, in query
**req_kwargs
File "/usr/lib64/python2.6/site-packages/tornado/httpclient.py", line 102, in fetch
self._async_client.fetch, request, **kwargs))
File "/usr/lib64/python2.6/site-packages/tornado/ioloop.py", line 444, in run_sync
return future_cell[0].result()
File "/usr/lib64/python2.6/site-packages/tornado/concurrent.py", line 214, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
SSLError: [Errno 1] _ssl.c:493: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
Started: 11:49:54.540076
Duration: 58.455 ms
Changes:
edhgoose
All 20 comments
The same problem also occurs with this state too, making it kinda tricky to download the files required:
{% set telegraf_version = "telegraf-0.13.1.x86_64.rpm" %}
telegraf-pkg:
file.managed:
- name: /tmp/{{ telegraf_version }}
- source: https://dl.influxdata.com/telegraf/releases/{{ telegraf_version }}
- source_hash: md5=15b543dad1bd3187761c41cc53477137
- unless: test -f /tmp/{{ telegraf_version }}
Outcome is:
----------
ID: telegraf-pkg
Function: file.managed
Name: /tmp/telegraf-0.13.1.x86_64.rpm
Result: False
Comment: Unable to manage file: [Errno 1] _ssl.c:493: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
Started: 12:14:54.150573
Duration: 727.408 ms
Hi @edhgoose, that issue you linked on requests mentions the contrib library in urllib3, and several packages needed to get it to provide SNI support (not just pyopenssl, but also ndg-httpsclient and pyasn1). Getting them all installed correctly also requires a compiler and a few devel packages. I'm not sure this will clear your issue, but a while back I was getting annoyed at the urllib3 warnings and came up with the following to clear them (usually I include this in userdata for my instances):
# Fix python urllib3 warnings
yum -y install gcc python-devel libffi-devel openssl-devel
pip install pyopenssl ndg-httpsclient pyasn1
# Remove gcc now that it is no longer needed
yum -y remove gcc --setopt=clean_requirements_on_remove=1
Apologies for the noise if this doesn't help... :disappointed:
lorengordon
on 5 Jun 2016
I have the same problem with same URL (InfluxDB download links).
I tried to install all dev packages (for Ubuntu/Debian: python-dev libffi-dev libssl-dev). And almost all tricks here, but nothing works.
Some info:
System: Ubuntu 14.04.4 LTS
Python: 2.7.6
SaltStack: 2015.8.8.2 (Beryllium)
aabouzaid
on 7 Jun 2016
@AAbouZaid can you try upgrading to python 2.7.10 by chance and see if you are still seeing the error?
@edhgoose I am able to replicate this issue on amazon linux 2015.09 but not centos7 with python27 and my guess is this might be related to the conflicts between python26 and python27 with amazon linux. I know amazon 2015.03 is based on rhel6 which typically uses python26 but they are using python27. The salt package is using python26.
I have tried to search a workaround and have not been able to find anything yet.
ping @cachedout do you have an idea as to why this is occurring?
Ch3LL
on 7 Jun 2016
This seems to be because ssl object in Python 2.6 doesn't have SNI. And everything is using that. And even if you do install the pyOpenSSL stuff - all the libraries won't use it by default - they need telling to in the code.
Ansible looks to have fixed this. Not sure if it's something that might provide some insight into what's needed within SaltStack?
Hit this problem today and it's really frustrating as seems there's no way to implement any of the workarounds without modifying SaltStack somehow.
https://github.com/ansible/ansible-modules-core/issues/3355
https://github.com/ansible/ansible/pull/15289
driskell
on 21 Jul 2016
I just ran into this issue on a RHEL 6 server while trying to use file.managed with a CloudFront backed URL.
I was able to workaround the problem by creating a python2.7 virtualenv (with --no-site-packages) and making a symlink from site-packages/salt to my git checkout of salt.
I then copied all of the salt scripts from /usr/bin/ into my virtualenv's bin directory. Rewrote the #! lines to reference python2.7 in the virtualenv and then pip installed any dependencies until salt-call ran cleanly.
I am now able to successfully run the following state, where I was getting an SSLv3 handshake error previously.
upsource_download:
file.managed:
- name : /tmp/upsource-3.0.4389.zip
- source : https://download-cf.jetbrains.com/upsource/upsource-3.0.4389.zip
- source_hash : https://download.jetbrains.com/upsource/upsource-3.0.4389.zip.sha256
dwiest
on 25 Jul 2016
I started digging into this a bit more, and as far as I can tell, only requests and urllib3 offer SNI support for python 2.6 (and only when the necessary dependencies are installed), see the ansible PR linked above. Salt's utils/http.py module has backends for requests, urllib2, and tornado. The salt module defaults to tornado, but tornado utilizes the builtin ssl module exclusively, which does not and seemingly will not ever implement SNI support for python 2.6.
So, potentially, salt could fix this for the requests backend and a user could maybe somehow be able to specify the requests backend to get SNI support. However, I cannot find an existing way to set the backend in a way that it will be passed to salt's utils/http.py module to test if it might already work. Perhaps a salt dev can chime in and point out if I'm missing something obvious. ;)
lorengordon
on 26 Jul 2016
Hitting this problem again with a file.managed on a site that is now using CloudFront. :(
driskell
on 6 Oct 2016
Theoretically this should all work out of the box for requests, simply by adding requests_lib: True to the /etc/salt/minion configuration file and ensuring the python2-ndg_httpsclient package is installed from EPEL. (This is for CentOS 6 / RedHat 6 using the EPEL salt packages, which aren't the new ones with the backend configuration.)
It should just work because requests already performs the necessary pyopenssl patching to bring in SNI support. Unfortunately, when Salt tries to use it, it seems this patching is not happening, or isn't taking effect.
I've now tracked the root cause of this problem for CentOS 6 and RedHat 6. The issue lies with the distribution maintainers. If you read the comments in the header of requests's vendor package importer you'll begin to understand there's some politics and the distributors are modifying requests when they package it: https://github.com/kennethreitz/requests/blob/master/requests/packages/__init__.py
It seems that on RedHat 6 (and therefore CentOS 6) the distributed requests library is modified to pull in the system installed urllib3 package rather than using a bundled version inside requests. Unfortunately, due to how the mapping works you actually end up with a partially broken requests library. If you modify the system package urllib3's urllib3.connection module to print when it is imported, you'll see it gets imported TWICE. The first import is as a result of the SNI patching, so is modified by requests to add support for SNI. The second import isn't. It's the second import that actually gets used by SaltStack. It seems after the patching, the sys.modules isn't updated correctly and therefore every subsequent request to import urllib3.connection pulls in a brand new imported version and doesn't use the patched one. This breaks SNI.
The workaround for now on those distributions is to modify salt/util/http.py to modify the import requests part of the code to the following code, which will patch the SECOND imported urllib3.connection. This then fixes the issue completely (assuming you added requests_lib: True to your minion configuration.)
# Import 3rd party libs
try:
import requests
try:
from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
except ImportError:
pass
HAS_REQUESTS = True
except ImportError:
HAS_REQUESTS = False
If I can find some time I will try raise BugZilla for RedHat 6 to get this fixed as their stub __init__.py in their requests package is clearly busting SNI support.
In the meantime I do wonder if SaltStack would accept a PR to add the above code snippet to salt/util/http.py anyway? It'll probably help a lot of people out for now.
Thanks for the great product!
driskell
on 7 Oct 2016
TL;DR:
To fix on Red Hat 6 and CentOS 6 with EPEL SaltStack packages:
Install the python2-ndg_httpsclient package from EPEL.
Add the following to /etc/salt/minion:
requests_lib: True
Modify the following lines in /usr/lib/python2.6/site-packages/salt/utils/http.py:
# Import 3rd party libs
try:
import requests
try:
from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
except ImportError:
pass
HAS_REQUESTS = True
except ImportError:
HAS_REQUESTS = False
To:
# Import 3rd party libs
try:
import requests
HAS_REQUESTS = True
except ImportError:
HAS_REQUESTS = False
To test, run:
salt-call http.query https://downloads.atlassian.com/
For CentOS 7 and RedHat 7, which use Python 2.7, SNI support is available to requests as it was backported, so just adding requests_lib: True into /etc/salt/minion I think will probably work (haven't tried except for running http.query)
driskell
on 7 Oct 2016
I forgot to add, CentOS 6 / RedHat 6 you need the python2-ndg_httpsclient package installed from EPEL for the SNI patching to work. I've edited my comment above to add this in.
driskell
on 7 Oct 2016
Raised at RedHat here to see what they think if anyone is interested: https://bugzilla.redhat.com/show_bug.cgi?id=1382682
driskell
on 7 Oct 2016
ping @terminalmage can I get your eyes on this issue? what do you think of @driskell 's potential PR/change above?
Ch3LL
on 8 Oct 2016
I haven't had much time to test this all on Amazon Linux. So far I can say that SNI with Python 2.6 might not work on Amazon Linux at all without serious changes. On my 2013.03 machine, the requests version they ship is 1.2.3. Even though this version should have the SNI bits inside it, they have stripped it out (although they seem to have handled the urllib3 vendoring really well). On a 2016.09 machine it seems the exact same thing for the installed Python 2.6 (though the default now seems to be Python 2.7.)
Looking into it, I can see why. They ship pyOpenSSL 0.10. This doesn't work with SNI. It needs pyOpenSSL 0.13. RedHat / CentOS 6.8 at least have that. So even the workaround above won't work on Amazon Linux!
At least since Amazon Linux 2013.03, Python 2.7.10 is available and installed. And since before 2016.09 it is now the default Python. However, Salt Stack is still using Python 2.6 regardless of the default. I think the only way to get SNI to work for Salt Stack on Amazon Linux at the moment is for it to switch to Python 2.7. The available version is 2.7.10 which has SNI support builtin, back-ported from Python 3. No additional dependencies required.
Having said all that - if people remove pyOpenSSL and/or get the latest version installed via pip then the workaround above will likely start working. I'm just not a fan of such invasive modifications and don't yet have test machine to test it with 鈽癸笍
driskell
on 8 Oct 2016
I tested and got it to work on Amazon Linux. Using the above-mentioned workaround for RedHat 6 / CentOS 6 (requests_lib setting and modifying utils/http.py and installing python2-ndg_httpsclient) and violently butchering the entire Python 2.6 installation with pip install --upgrade pyopenssl results in working SNI. This is likely to break if Amazon update certain python packages though as the update will overwrite the upgrade
Really Salt needs to use Python 2.7 鈽癸笍
driskell
on 8 Oct 2016
Just a heads up, a better workaround than modifying utils/http.py is to create a _modules/sni.py in your file_roots with the following content. It'll only fix high states though and not direct module runs like salt-call file.managed as it only loads in the state run I think.
'''
Fix SNI support in SaltStack
'''
# Attempt to enable urllib3's SNI support, if possible
try:
from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
except ImportError:
pass
Then once you've installed python2-ndg_httpsclient and enabled requests_lib: True in /etc/salt/minion everything works for me (tested on RedHat 6.8). For Amazon Linux you'll need to butcher pyopenssl still unfortunately. I can't think of any other solution 鈽癸笍
This works through upgrades as a result.
driskell
on 11 Oct 2016
@driskell, nice research, thanks!
lorengordon
on 11 Oct 2016
We discussed this internally today and decided the better long-term solution would be to build Amazon Linux RPMs based on Python 2.7. This should help alleviate this and a number of other issues which up until now we have just made band-aid solutions to work around the weirdness that results from a Python 2.6 app running on a box which defaults to Python 2.7 as its default system Python.
terminalmage
on 11 Oct 2016
Based on @driskell research for CentOS this worked for me on Ubuntu 14.04:
apt-get install -y python-pip python-dev libffi-dev python-openssl libssl-dev
pip install --upgrade pyopenssl ndg-httpsclient pyasn1 requests[security] urllib3[secure] certifi idna pip chardet cryptography
And adding the settings he posted for minion config and _modules/sni.py.
amontalban
on 27 Sep 2017
Related issues
allyunion
路
3Comments
erwindon
路
3Comments
qiushics
路
3Comments
twangboy
路
3Comments
Arguros
路
3Comments