salt-api with rest_cherrpy doesn't work with Ubuntu 16.04 Xenial

[edgan]

Description of Issue/Question

I just upgraded my salt master from Ubuntu 14.04(trusty) to Ubuntu 16.04(xenial). I am using the salt-api to retrieve salt keys for AWS instances. This worked with trusty, but is failing with xenial. I found mention of the documentation that 3.2.3 is the ideal version of python-cherrypy. The version in trusty is 3.2.2, and xenial is 3.5.0. I made my own version of python-cherrypy3 package, 6.0.1.

The error comes from the client. The salt-api doesn't log anything even in debug. It seems like the connection times out.

Steps to Reproduce Issue

Start salt_api
Run openssl s_client -connect salt-master:8000

Error:
139707275663224:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:362:

Versions Report

Works:
3.2.2-4ubuntu5(trusty)
6.0.1(custom for xenial)

Fails:
3.5.0-2build1(xenial)

[Ch3LL]

@edgan would you mind sharing what version of salt this is on. If you could paste the output of salt-api --versions-report that would be great. Thanks!

[edgan]

salt-api --versions-report
Salt Version:
Salt: 2016.3.1

Dependency Versions:
cffi: 1.5.2
cherrypy: 6.0.1
dateutil: 2.4.2
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pygit2: 0.24.0
Python: 2.7.11+ (default, Apr 17 2016, 14:00:29)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-22-generic
system: Linux
version: Ubuntu 16.04 xenial

[whiteinge]

Shot in the dark: this may have something to do with the version of SSL that Python is built against on each distro. Is the Python and OpenSSL packages that are installed the most up-to-date for each distro version?

[edgan]

My understanding is it is just an incompatibility between certain versions of cherrypy and salt. It is an unfortunate situation of Canonical picking a less than ideal version of a piece of software for a LTS release.

[ctrlrsf]

Also ran into this when we upgraded our salt master to xenial. Below is a quick gist to build the python-cherry3 package from trusty (3.2.2). This got our salt-api working again.

https://gist.github.com/ctrlrsf/3272ae4c0cfc58ec0060d685818e9a11

[ctrlrsf]

@Ch3LL looks like @edgan already provided output needed. Can the Info Needed label be removed so we get more visibility into this issue? If anything else is needed from us, please let us know.

[Ch3LL]

Thanks for the bump. Yeah I'll change the labels. So just to clarify it seems the issue here is ubuntu's doesn't provide the package that we are compatible with. But maybe the bigger issue is to get it working with 3.5.0 cherrypy. So i'll lable it as a bug for this reason, unless I'm incorrect in this logic please correct me.

[Deshke]

just FYI, this still occurs with a fresh ubuntu 16.04.02

[Reiner030]

Happy birthday to this bug... hope you have a nice party today ;)

Any news about fix status/howto workaround this problem?
It also happens on "good old Debian Jessie" with latest saltstack 2016.11.5

[edgan]

I made my own cherrypy package, and threw in my private apt repo. I don't see Ubuntu fixing this anytime soon.

[Ch3LL]

I'm not sure if ubuntu16 provides the cherrypy 10 package but you can always use pip. We just updated our docs here: https://github.com/saltstack/salt/pull/41866/files . Seems version 10 is working. Can anyone test that out and see if its working?

[ctrlrsf]

Hi @Ch3LL, Ubuntu 16.04 doesn't provide a cherrypy 10 package. It currently installs 3.5.0-2build1, which doesn't work with salt-api. We prefer not to upgrade it with pip at this time, so we're holding the package version at 3.2.x as a workaround.

[Reiner030]

Its interesting how such old package is needed / seems to be used for testing...
For Debian luckily there is till the old old stable = wheezy package available with the right version:
https://packages.debian.org/search?keywords=python-cherrypy3&searchon=names&suite=all&section=all
=> https://packages.debian.org/wheezy/python-cherrypy3

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.