Salt: [BUG] Missing salt-ssh dependencies on Ubuntu 16.04

[RobHooper]

Description
Following the upgrade of my salt client from 3000.3+ds-1 to 3001+ds-1 I started getting the following error.

$ salt-ssh '*' test.ping
Traceback (most recent call last):
  File "/usr/bin/salt-ssh", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pycryptodomex>=3.4.7' distribution was not found and is required by salt

Setup
Start Ubuntu 16 desktop/server (haven't tested any other os)
Add the salt repo http://repo.saltstack.com/py3/ubuntu/16.04/amd64/latest
sudo apt-get install salt-ssh

Steps to Reproduce the behavior
salt-ssh '*' test.ping

You can fix the issue by running:
sudo apt-get install python3-pycryptodome

Expected behavior
The salt-ssh or salt-common package should include python3-pycryptodome as a dependency so it is installed by default.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

$ salt-ssh --versions-report
Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: Not Installed
         Python: 3.5.2 (default, Apr 16 2020, 17:47:17)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 17.1.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: ubuntu 16.04 Xenial Xerus
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-184-generic
         system: Linux
        version: Ubuntu 16.04 Xenial Xerus

[OrangeDog]

python3-pycryptodome isn't available for 16.04.
https://packages.ubuntu.com/xenial/python3-pycryptodome

[RobHooper]

python3-pycryptodome is available for 16.04 from the salt stack repositories.

http://repo.saltstack.com/py3/ubuntu/16.04/amd64/latest/pool/main/p/pycryptodome/

[bryceml]

It appears that along with all of the changes going into moving to pycryptodome, I marked pycryptodome as a dependency of salt-master and salt-minion since that's where the dependencies on python3-crypto were.

--- 3000_3/ubuntu1604/spec/debian/control   2020-02-01 14:59:13.000000000 -0700
+++ 3001/ubuntu1604/spec/debian/control 2020-06-10 18:54:41.000000000 -0600
@@ -21,12 +21,13 @@


 Package: salt-common
 Architecture: all
 Depends: python3-apt,
          python3-dateutil,
+         python3-distro,
          python3-jinja2,
          python3-msgpack (>= 0.5.6),
          python3-pkg-resources,
          python3-requests,
          python3-yaml,
          python3-systemd,
@@ -57,14 +58,14 @@
  This particular package provides shared libraries that
  salt-master, salt-minion, and salt-syndic require to function.


 Package: salt-master
 Architecture: all
-Depends: python3-crypto (>= 2.6.1),
-         python3-zmq (>= 14.4.0),
+Depends: python3-pycryptodome,
+         python3-zmq (>= 17.0.0),
          salt-common (= ${source:Version}),
          ${misc:Depends},
          ${python3:Depends}
 Recommends: python3-git
 Description: remote manager to administer servers via salt
  salt is a powerful remote execution manager that can be used to
@@ -87,14 +88,14 @@


 Package: salt-minion
 Architecture: all
 Depends: bsdmainutils,
          dctrl-tools,
-         python3-crypto (>= 2.6.1),
-         python3-zmq (>= 14.4.0),
+         python3-pycryptodome,
+         python3-zmq (>= 17.0.0),
          salt-common (= ${source:Version}),
          ${misc:Depends},
          ${python3:Depends}
 Recommends: debconf-utils, dmidecode
 Suggests: python3-augeas
 Description: client package for salt, the distributed remote execution system

Looks like we need to move those dependencies to the salt-common package. I'm not sure why this wasn't breaking with python3-crypto. Probably something we changed in the python requirements.

Currently this should only affect people who aren't already installing the salt-master or salt-minion packages.

[bryceml]

@dwoz says we're going to do a re-package release before 3001.1, which should show up in repo.saltstack.com as

repo.saltstack.com/..../archive/3001-2
repo.saltstack.com/..../3001 -> archive/3001-2 # this is a symlink.

using the bootstrap script you will get the update.

[bryceml]

This also affects raspbian 9

[sagetherage]

Yes, that is a confusing, we will do more packages on repo.saltstack.com which will be agnostic of a salt release. I am removing the release label.

[sagetherage]

I am adding the version - since I can. Not to confuse, we are working this now and will publish as soon as we are able.

[bryceml]

@dwoz says we're going to do a re-package release before 3001.1, which should show up in repo.saltstack.com as

repo.saltstack.com/..../archive/3001-2
repo.saltstack.com/..../3001 -> archive/3001-2 # this is a symlink.

using the bootstrap script you will get the update.

Actually, I think it would be better to re-name the existing folder in archive to 3001-1 and have the new release replace 3001
so

repo.saltstack.com/..../archive/3001-1 # this is the old release
repo.saltstack.com/..../archive/3001 # this is the new release
repo.saltstack.com/..../3001 -> archive/3001 # this is a symlink.

Is that ok @dmurphy18 ?

[dmurphy18]

@bryceml it is similar to what I did for Raspbian 9
[root@qa-master /srv/rsync/repo_contents_staging/py3/debian/9/armhf]# l total 12K drwxr-xr-x. 4 root root 4.0K Apr 21 18:35 .. lrwxrwxrwx. 1 root root 16 May 9 02:58 2019.2 -> archive/2019.2.5 lrwxrwxrwx. 1 root root 14 May 9 15:59 3000 -> archive/3000.3 lrwxrwxrwx. 1 root root 12 Jun 11 17:12 3001 -> archive/3001 lrwxrwxrwx. 1 root root 12 Jun 11 17:13 latest -> archive/3001 drwxr-xr-x. 3 root root 4.0K Jun 11 17:13 . drwxr-xr-x. 8 root root 4.0K Jun 19 22:13 archive You have new mail in /var/spool/mail/root [root@qa-master /srv/rsync/repo_contents_staging/py3/debian/9/armhf]# l archive/ total 32K drwxr-xr-x. 6 root root 4.0K Apr 24 00:04 2019.2.4 drwxr-xr-x. 6 root root 4.0K Apr 24 00:44 3000.2 drwxr-xr-x. 6 root root 4.0K May 9 02:58 2019.2.5 drwxr-xr-x. 6 root root 4.0K May 9 15:59 3000.3 drwxr-xr-x. 3 root root 4.0K Jun 11 17:13 .. drwxr-xr-x. 6 root root 4.0K Jun 11 17:13 3001-1 drwxr-xr-x. 6 root root 4.0K Jun 19 22:13 3001-2 lrwxrwxrwx. 1 root root 6 Jun 19 22:13 3001 -> 3001-2 drwxr-xr-x. 8 root root 4.0K Jun 19 22:13 . [root@qa-master /srv/rsync/repo_contents_staging/py3/debian/9/armhf]#

[bryceml]

This is fixed and has been pushed live on repo.saltstack.com.

[JimACarter]

Just to confirm this fix has worked for me.

My test process was salt-ssh 'target' test.ping

My set-up process is the same as @RobHooper I'm on 3001+ds-2 and python3-pycryptodome was installed when I updated to it.