Salt: _raw_desmodule.so: cannot open shared object file: No such file or directory

[DeviantEng]

Description of Issue/Question

salt-master won't start:

Jun 13 22:03:49 salt salt-master[627]: salt-master: error: Failed to load configuration: Cannot load native module 'Cryptodome.Cipher._raw_des': Trying '_raw_des.so': cannot load library /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_des.so: /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_des.so: undefined symbol: des_ecb_decrypt, Trying '_raw_desmodule.so': cannot load library /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_desmodule.so: /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_desmodule.so: cannot open shared object file: No such file or directory

Setup

Salt-Master on CentOS 7, salt-2018.3.1. Been running fine for months, but seems something has recently changed and broked salt-master (guessing an update?).

Versions Report

Unfortunately, won't run.

[root@salt ~]# salt --versions-report
Traceback (most recent call last):
  File "/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 480, in salt_main
    import salt.cli.salt
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 10, in <module>
    import salt.utils.job
  File "/usr/lib/python2.7/site-packages/salt/utils/job.py", line 11, in <module>
    import salt.minion
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 75, in <module>
    import salt.client
  File "/usr/lib/python2.7/site-packages/salt/client/__init__.py", line 31, in <module>
    import salt.cache
  File "/usr/lib/python2.7/site-packages/salt/cache/__init__.py", line 16, in <module>
    from salt.payload import Serial
  File "/usr/lib/python2.7/site-packages/salt/payload.py", line 17, in <module>
    import salt.crypt
  File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 40, in <module>
    from Cryptodome.PublicKey import RSA
  File "/usr/lib64/python2.7/site-packages/Cryptodome/PublicKey/RSA.py", line 38, in <module>
    from Cryptodome.IO import PKCS8, PEM
  File "/usr/lib64/python2.7/site-packages/Cryptodome/IO/PKCS8.py", line 44, in <module>
    from Cryptodome.IO._PBES import PBES1, PBES2, PbesError
  File "/usr/lib64/python2.7/site-packages/Cryptodome/IO/_PBES.py", line 44, in <module>
    from Cryptodome.Cipher import DES, ARC2, DES3, AES
  File "/usr/lib64/python2.7/site-packages/Cryptodome/Cipher/DES.py", line 57, in <module>
    """)
  File "/usr/lib64/python2.7/site-packages/Cryptodome/Util/_raw_api.py", line 191, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Cryptodome.Cipher._raw_des': Trying '_raw_des.so': cannot load library /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_des.so: /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_des.so: undefined symbol: des_ecb_decrypt, Trying '_raw_desmodule.so': cannot load library /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_desmodule.so: /usr/lib64/python2.7/site-packages/Cryptodome/Util/../Cipher/_raw_desmodule.so: cannot open shared object file: No such file or directory

[stpfeil]

salt-minion on CentOS 7 is affected as well.

[martinrm77]

I'm using 2017.7 and also getting these failures.
It fails after these packages were updated from salt pkg repo:
salt-2017.7.6-1.el7.noarch
salt-minion-2017.7.6-1.el7.noarch
python2-pycryptodomex-3.4.11-2.el7.x86_64
zeromq-4.1.4-7.el7.x86_64

But the salt repo dont have old versions of packages, so I cant downgrade...

[stpfeil]

But the salt repo dont have old versions of packages, so I cant downgrade...

I also tried to downgrade.
I removed all salt-packages and python2-msgpack (which was marked as duplicate [again]). Then I reinstalled salt 2018.3.1 and then downgraded. This installed salt 2015-5.10 which was completely unusable in our case.

Then I changed the repo from latest to 2017.7 [1] and upgraded to 2017.7.6. I encountered the same problem. Then I changed the repo again from 2017.7 to latest and upgraded to salt 2018.3.1 again.

Now magically salt-master and salt-minion are working again - at least on the master server, I don't want to have to do this procedure on all our minions (which all but one I haven't upgraded yet)...

Edit: _On the broken minion salt-minion worked again after downgrade to 2015.5.10 and upgrading again to 2017.7.6 (and onwards after the update to 2018.3.1). It seems the downgrade to 2015-5.10 is fixing the problem._

[1] While on salt version 2015-5.10 I edited the repo-file, in our case /etc/yum.repos.de/salt-latest.repo
comment the line
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/latest
and replace it with
baseurl=https://repo.saltstack.com/yum/redhat/$releasever/$basearch/2017.7
yum clean all to clear the cached data.
To reenable the latest repo remove the added line with 2017.7 and uncomment the latest line.

[DeviantEng]

This is what got my salt-master back up this morning:

[root@salt ~]# rpm -qa | grep python2-pycryptodomex
python2-pycryptodomex-3.4.11-2.el7.x86_64
[root@salt ~]# rpm -e python2-pycryptodomex
[root@salt ~]# yum install https://repo.saltstack.com/yum/redhat/7/x86_64/2016.11/python2-pycryptodomex-3.4.3-2.el7.x86_64.rpm

I was then able to restart the salt-master service with systemctl restart salt-master and it started successfully. Definitely seems that pycryptodomex is to blame, but not sure specifically why. Going to continue testing to see if I'm fixed (at least for now).

[sithmein]

Same problem here. A faster way to revert back to the old python2-pycryptodomex version:

yum downgrade https://repo.saltstack.com/yum/redhat/7/x86_64/2016.11/python2-pycryptodomex-3.4.3-2.el7.x86_64.rpm

[garethgreenaway]

Thanks everyone. Appears to be an issue with the pycryptodomex we're providing on the SaltStack repo. We'll get that fixed up ASAP.

[dmurphy18]

@martinrm77 FYI the older versions are always available, you just need to use the instructions for
'Pin to Minor Release' and set the version to whichever one you want,for example 2017.7.6 -> 2017.7.5 for the previous release.

Looking into the pycryptodomex issue

[dmurphy18]

Python 3 version works fine, examining failures in Python 2 version

[dmurphy18]

Fixes associated with Salt 2018.3.1 for Python 2 and Python 3 on Redhat 7 & 6 have been sent to QA

[OrlandoArcapix]

Very quick workaround for those needing it:

yum downgrade http://repo.saltstack.com/yum/redhat/7/x86_64/archive/2017.7.5/python2-pycryptodomex-3.4.3-2.el7.x86_64.rpm

[dmurphy18]

Fixes associated with Salt 2017.7.6 for Python 2 and Python 3 on Redhat 7 & 6 have been sent to QA

Given time of day in Mountain States USA, if QA passes the fixes, then the respective repos should be updated beginning next week.

[dmurphy18]

@DeviantEng A new version of pycryptodomex has been posted for Salt 2017.7.6 and 2018.3.1 for Redhat 7 & 6. Can you test that this resolves issues encountered, and close this issue, if you consider it resolved.

Note seeing some issues with having to restart salt-api, as detailed on repo.saltstack.com pages for Redhat 7 & 6, this is being examined, see: https://github.com/saltstack/salt-pack/issues/561

[rhoths]

I can confirm that this seems to fix the problem for us on 2017.7.6

[DeviantEng]

@dmurphy18
Appears to be working for me as well, on 2018.3.1.

Closing.

[sithmein]

Thanks for fixing the issue. However, this is about the third time in the last two years (since we started using Salt) that an update was completely broken and rendered everything unusable. This makes we wonder about how and whether you are doing QA? RHEL/CentOS seems to be a widely used OS on servers where salt makes sense therefore I would expect this combination to be thoroughly tested. Which doesn't seem to be the case, though.

[DeviantEng]

@sithmein I can relate, unfortunately. I had another issue just a few months ago, #46868, which was also a dependency issue on EL7. I likewise share your concern that they may not be able to do QA across all distro/environments, but that's definitely a big task to undertake.