Salt: Issues starting salt-master or salt-minion with openssl-1.1.1-1

[ephreal]

Description of Issue/Question

Some incompatibility exists between openssl-1.1.1-1 and salt 2018.3.2-2. I am using Archlinux fully updated (pacman -Syu). Downgrading to openssl-1.1.0.i-1 solves the issue, but makes me worry about updating my system and breaking things again.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)
Any install should provide the same result. No special setups were used. The issue happens on fresh installs with the config files as default.

Steps to Reproduce Issue

Fully update your system.
Check to see if openssl-1.1.1-1 has installed. If not, you may need to manually install it.
Stop any running instances of salt-master or salt-minion on the system.
Attempt to run systemctl start salt-minion or systemctl start salt-master.

journalctl -xe     # Post update and attempting to run systemctl start salt-master  
--SNIP--
Sep 14 10:35:19 archlinux sudo[15494]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 14 10:35:19 archlinux systemd[1]: Starting The Salt Master Server...

-- Subject: Unit salt-master.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Unit salt-master.service has begun starting up.

Sep 14 10:35:19 archlinux salt-master[15498]: Usage: salt-master [options]
Sep 14 10:35:19 archlinux salt-master[15498]: salt-master: error: Failed to load configuration: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)
Sep 14 10:35:19 archlinux systemd[1]: salt-master.service: Main process exited, code=exited, status=64/USAGE
Sep 14 10:35:19 archlinux systemd[1]: salt-master.service: Failed with result 'exit-code'.
Sep 14 10:35:19 archlinux systemd[1]: Failed to start The Salt Master Server.

-- Subject: Unit salt-master.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Unit salt-master.service has failed.

--SNIP--

Attempting to run from the command line rather than a service also fails.

salt-master -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: archlinux
Usage: salt-master [options]

salt-master: error: Failed to load configuration: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)

Here is the newest version of openssl that I have confirmed working with salt

$ pacman -Qi openssl
Name            : openssl
Version         : 1.1.0.i-1
Description     : The Open Source toolkit for Secure Sockets Layer and Transport Layer Security
Architecture    : x86_64
URL             : https://www.openssl.org
Licenses        : custom:BSD

--SNIP--

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

All O/S are fully updated and on the same version of salt. Attempting to get a version report while openssl-1.1.1-1 is installed gives the following error.

Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 480, in salt_main
    import salt.cli.salt
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 10, in <module>
    import salt.utils.job
  File "/usr/lib/python2.7/site-packages/salt/utils/job.py", line 11, in <module>
    import salt.minion
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 75, in <module>
    import salt.client
  File "/usr/lib/python2.7/site-packages/salt/client/__init__.py", line 30, in <module>
    import salt.config
  File "/usr/lib/python2.7/site-packages/salt/config/__init__.py", line 98, in <module>
    _DFLT_IPC_WBUFFER = _gather_buffer_space() * .5
  File "/usr/lib/python2.7/site-packages/salt/config/__init__.py", line 88, in _gather_buffer_space
    import salt.grains.core
  File "/usr/lib/python2.7/site-packages/salt/grains/core.py", line 44, in <module>
    import salt.utils.dns
  File "/usr/lib/python2.7/site-packages/salt/utils/dns.py", line 32, in <module>
    import salt.modules.cmdmod
  File "/usr/lib/python2.7/site-packages/salt/modules/cmdmod.py", line 34, in <module>
    import salt.utils.templates
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 32, in <module>
    import salt.utils.http
  File "/usr/lib/python2.7/site-packages/salt/utils/http.py", line 41, in <module>
    import salt.loader
  File "/usr/lib/python2.7/site-packages/salt/loader.py", line 28, in <module>
    import salt.utils.event
  File "/usr/lib/python2.7/site-packages/salt/utils/event.py", line 74, in <module>
    import salt.payload
  File "/usr/lib/python2.7/site-packages/salt/payload.py", line 17, in <module>
    import salt.crypt
  File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 68, in <module>
    import salt.utils.rsax931
  File "/usr/lib/python2.7/site-packages/salt/utils/rsax931.py", line 86, in <module>
    libcrypto = _init_libcrypto()
  File "/usr/lib/python2.7/site-packages/salt/utils/rsax931.py", line 77, in _init_libcrypto
    raise OSError("Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)")
OSError: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)

After downgrading to openssl-1.1.0.i-1, the report is as follows.

Salt Version:
           Salt: 2018.3.2
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.30.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 3.6.6
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15 (default, Jun 27 2018, 13:05:28)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 5.1
            ZMQ: 4.2.5

System Versions:
           dist:   
         locale: UTF-8
        machine: x86_64
        release: 4.17.8-1-ARCH
         system: Linux
        version: Not Installed

[ephreal]

While I thought I was thorough in my bug searching, I had someone bring another bug report to my attention. My issue appears to be the same as referenced here: https://github.com/saltstack/salt/issues/46884 .
If this is the case, it appears that a patch has been merged and will be fixed in salt 2018.3.3, sorry for the noise! If not, then hopefully this will point someone in the right direction.
EDIT: I tested using the current rsax931.py module with my setup, and salt-master/salt-minion do indeed start with the new changes. I believe this has been fixed already.

[garethgreenaway]

@ephreal Thanks for the update. I'll go ahead and close this one since the issue is fixed.

[kiemlicz]

I've asked some time ago about this but no reply was given

Why such important fixes like this one are not immediately back-ported and released?

It simply disables salt on recent ubuntu distros (e.g. all recent azure ubuntu cloud images)

You have undefined release cycle and based on the history of releases the 2018.3.3 can be released in 2020 as well

[sim0nx]

Any update on this plz?

[rmader]

I'd also kindly request a backport of openssl 1.1.1 compatibility to 2017.7. We have lots of issues with 2018.3 and while it's of course better to fix those for good, it's really unfortunate that salt blocks us from using tls 1.3.

It's important to add that while the fix from https://github.com/saltstack/salt/pull/48580 was backported to 2017.7.8 already, salt-master still errors (ubuntu 18.04 with openssl from https://launchpad.net/~ondrej/+archive/ubuntu/nginx):

Dec 11 14:59:51 tls13 systemd[1]: Starting The Salt Master Server...
Dec 11 14:59:52 tls13 salt-master[5472]: Usage: salt-master [options]
Dec 11 14:59:52 tls13 salt-master[5472]: salt-master: error: Failed to load configuration: unknown error (_ssl.c:2214)
Dec 11 14:59:52 tls13 systemd[1]: salt-master.service: Main process exited, code=exited, status=64/n/a
Dec 11 14:59:52 tls13 systemd[1]: salt-master.service: Failed with result 'exit-code'.
Dec 11 14:59:52 tls13 systemd[1]: Failed to start The Salt Master Server.

[Ch3LL]

@treba123 thanks for bringing this to our attention. We will have someone take a look for the 2017.7.9 release.

@kiemlicz to answer your question we have to make a decision based on how many people this would affect and the severity of the issue and if there is a current workaround. Hopefully its working for you now on 2018.3.3

[dubb-b]

@Ch3LL That bootstrap option did not fix the issue. just an FYI

[garethgreenaway]

@treba123 @kiemlicz If you're able to test the change in that PR, it should allow 2017.7 to start up as expected and will be available in the 2017.7.9 release.

[rmader]

@garethgreenaway ouch xD
can confirm it now works perfectly fine, thanks alot!

[garethgreenaway]

Yup! That was a nasty one and hard to find, kudos to @Ch3LL for tracking it down.
@treba123 Thanks for testing it out.

[garethgreenaway]

Just an FYI, it appears that salt-key has a similar issue. Currently investigating the cause.

[kiemlicz]

@Ch3LL yes it is working fine, and to be honest I would relax your condition to

severity of the issue and if there is a current workaround

if the severity is high enough (declared functionality is not working and no workaround exists) why don't you patch the already released version?

[Ch3LL]

In cases that the severity is high enough and there is no workaround and it impacts most salt users, yes in the past we have done a new point release as quickly as possible in the past. If we were to patch a release we could only patch the packages and not the pip packages or tagged version, so if there is a bug that would require a quick re-release, we have done a new point release quickly after the previous release in the past.

In this particular case for the 2018.3 branch there was a patch provided which could be applied using the sync_utils here: https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.sync_utils or downgrade openssl.

[xnox]

Hi,
currently in Ubuntu we ship salt 2017.7.4 in 18.04 (bionic) and 18.10 (cosmic). In cosmic we ship OpenSSL 1.1.1 and salt is currently broken there. In bionic we are about to ugprade openssl from 1.1.0 to 1.1.1 which does break salt.

In the next development series 19.04 (disco) we have upgraded salt to 2018.3.4 and that appears to be working fine with OpenSSL 1.1.1

I found https://github.com/saltstack/salt/commit/736b382e913e46bce69940350e32d530f7560dac already, which I should cherrypick into bionic/cosmic. But are there any further backports available on top of 2017.7.4 to get it going with OpenSSL 1.1.1?

At the moment it crashes like so for us:

(cosmic-amd64)root@ottawa:~# salt --versions-report
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 476, in salt_main
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/salt.py", line 33, in run
    import salt.client
  File "/usr/lib/python3/dist-packages/salt/client/__init__.py", line 31, in <module>
    import salt.cache
  File "/usr/lib/python3/dist-packages/salt/cache/__init__.py", line 18, in <module>
    import salt.loader
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 26, in <module>
    import salt.utils.event
  File "/usr/lib/python3/dist-packages/salt/utils/event.py", line 70, in <module>
    import tornado.iostream
  File "/usr/lib/python3/dist-packages/tornado/iostream.py", line 40, in <module>
    from tornado.netutil import ssl_wrap_socket, _client_ssl_defaults, _server_ssl_defaults
  File "/usr/lib/python3/dist-packages/tornado/netutil.py", line 45, in <module>
    ssl.Purpose.SERVER_AUTH)
  File "/usr/lib/python3.6/ssl.py", line 502, in create_default_context
    context = SSLContext(PROTOCOL_TLS)
  File "/usr/lib/python3.6/ssl.py", line 391, in __new__
    self = _SSLContext.__new__(cls, protocol)
ssl.SSLError: unknown error (_ssl.c:2788)

[xnox]

Ah, the pull request for this open issue seems to fix everything on cosmic! Will upload that into Ubuntu now.

https://github.com/saltstack/salt/pull/51655/files

[Ch3LL]

thanks for verifying the fix @xnox i'll go ahead and close here now that its been verified by multiple people.

[rmader]

Just for information: Ubuntu 18.04 just rolled out Openssl 1.1.1. So until Salt 2017.7.9 comes out with the fix for this issue, people who use Salt 2017.7 from https://repo.saltstack.com will need to manually patch their installation (see https://github.com/saltstack/salt/pull/51655/files)

[xnox]

Salt as shipped in Ubuntu 18.04 is patched for this issue

https://launchpad.net/ubuntu/+source/salt/2017.7.4+dfsg1-1ubuntu18.04.1

salt (2017.7.4+dfsg1-1ubuntu18.04.1) bionic; urgency=medium

  * Cherrypick two upstream patches to fix compat with OpenSSL 1.1.1,
    without these salt fails to start when OpenSSL is upgraded from 1.1.0
    to 1.1.1. LP: #1823332
  * Fix up install call in debian/rules to resolve FTBFS.

 -- Dimitri John Ledkov <[email protected]>  Fri, 05 Apr 2019 15:41:52 +0100

[xnox]

However, I also now wonder if this is a regression in OpenSSL 1.1.1 which was fixed in 1.1.1b https://github.com/openssl/openssl/issues/7350

Ie. that no patches to salt are required with OpenSSL 1.1.1b or not.....

[xnox]

I am getting requests to backport this issue to the v2016 salt. Has anybody tried running v2016 salt with OpenSSL 1.1.1?

[stiobhan]

@xnox I've tested this on Ubuntu 18.04. If I install libssl1.1 version 1.1.1b-1ubuntu2.1 (deb pkg from disco) the salt minion (version 2017.7.4+ds-1 in this case) starts again without problems. So yes looks like with OpenSSL 1.1.1b patches are not needed.

[xnox]

We are in progress backporting the patch that allows once again multi-init into bionic.

https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.3

[nikolas]

@xnox thanks, it looks like this issue has been fixed after upgrading to the patched openssl on ubuntu 18.04. I was having the SSL issue with Salt 2016.x.