salt ssh error with debian 7 on target

[jerob]

salt-ssh and 2 targets ( debian 7 and debian 8 )

salt-ssh '*' -r 'ifconfig' = works well with both targets

salt-ssh '*' cmd.run 'ifconfig'

This command works well with debian 8. But with debian 7 (wheezy) :

web1:
    ----------
    retcode:
        1
    stderr:
        [CRITICAL] Unable to import msgpack or msgpack_pure python modules
        [ERROR   ] An un-handled exception was caught by salt's global exception handler:
        ImportError: No module named backports.ssl_match_hostname
        Traceback (most recent call last):
          File "/tmp/.root_a3a1dc_salt/salt-call", line 4, in <module>
            salt_call()
          File "/tmp/.root_a3a1dc_salt/salt/scripts.py", line 325, in salt_call
            import salt.cli.call
          File "/tmp/.root_a3a1dc_salt/salt/cli/call.py", line 8, in <module>
            import salt.cli.caller
          File "/tmp/.root_a3a1dc_salt/salt/cli/caller.py", line 20, in <module>
            import salt.minion
          File "/tmp/.root_a3a1dc_salt/salt/minion.py", line 83, in <module>
            import salt.pillar
          File "/tmp/.root_a3a1dc_salt/salt/pillar/__init__.py", line 15, in <module>
            import salt.fileclient
          File "/tmp/.root_a3a1dc_salt/salt/fileclient.py", line 29, in <module>
            import salt.utils.http
          File "/tmp/.root_a3a1dc_salt/salt/utils/http.py", line 58, in <module>
            import tornado.simple_httpclient
          File "/tmp/.root_a3a1dc_salt/tornado/simple_httpclient.py", line 8, in <module>
            from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
          File "/tmp/.root_a3a1dc_salt/tornado/http1connection.py", line 30, in <module>
            from tornado import iostream
          File "/tmp/.root_a3a1dc_salt/tornado/iostream.py", line 40, in <module>
            from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
          File "/tmp/.root_a3a1dc_salt/tornado/netutil.py", line 49, in <module>
            import backports.ssl_match_hostname
        ImportError: No module named backports.ssl_match_hostname
        Traceback (most recent call last):
          File "/tmp/.root_a3a1dc_salt/salt-call", line 4, in <module>
            salt_call()
          File "/tmp/.root_a3a1dc_salt/salt/scripts.py", line 325, in salt_call
            import salt.cli.call
          File "/tmp/.root_a3a1dc_salt/salt/cli/call.py", line 8, in <module>
            import salt.cli.caller
          File "/tmp/.root_a3a1dc_salt/salt/cli/caller.py", line 20, in <module>
            import salt.minion
          File "/tmp/.root_a3a1dc_salt/salt/minion.py", line 83, in <module>
            import salt.pillar
          File "/tmp/.root_a3a1dc_salt/salt/pillar/__init__.py", line 15, in <module>
            import salt.fileclient
          File "/tmp/.root_a3a1dc_salt/salt/fileclient.py", line 29, in <module>
            import salt.utils.http
          File "/tmp/.root_a3a1dc_salt/salt/utils/http.py", line 58, in <module>
            import tornado.simple_httpclient
          File "/tmp/.root_a3a1dc_salt/tornado/simple_httpclient.py", line 8, in <module>
            from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
          File "/tmp/.root_a3a1dc_salt/tornado/http1connection.py", line 30, in <module>
            from tornado import iostream
          File "/tmp/.root_a3a1dc_salt/tornado/iostream.py", line 40, in <module>
            from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
          File "/tmp/.root_a3a1dc_salt/tornado/netutil.py", line 49, in <module>
            import backports.ssl_match_hostname
        ImportError: No module named backports.ssl_match_hostname
    stdout:

Host :

salt-ssh --versions-report
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: Not Installed
           Mako: 1.0.0
         PyYAML: 3.11
          PyZMQ: Not Installed
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: Not Installed
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: Debian 8 
        machine: x86_64
        release: 3.16.0-4-amd64
         system: Debian 8

[jfindlay]

@jerob, thanks for the report.

[duk3luk3]

This appears to still be present in 2015.8.1

Version info:

erlacher@vmrbg51:/srv/saltstack$ salt-ssh --version
salt-ssh 2015.8.1 (Beryllium)
erlacher@vmrbg51:/srv/saltstack$ sudo aptitude show salt-common
Package: salt-common
State: installed
Automatically installed: yes
Version: 2015.8.1+ds-1

[nixuser]

Same issue was reproduced in following configuration:
Target: CentOS release 6.6 (Final)
Host: salt-ssh --versions-report

Salt Version:
           Salt: 2015.8.3

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: 1.0.0
         PyYAML: 3.11
          PyZMQ: Not Installed
         Python: 2.7.9 (default, Mar  1 2015, 12:57:24)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed
           cffi: 1.2.1
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: 1.2.3
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed

I guess that module backports should be installed on Target system. I tied to find this file on tartget directory with command: find /tmp/.root_734552_salt | grep -i back and only empty output. Any other suggestion where to find backports module ?

I tried to install python-backports.ssl-match-hostname on Host: system but this did not improve situation.

Full error message looks like this:

# salt-ssh '*' test.ping
host0:
    ----------
    retcode:
        1
    stderr:
        [CRITICAL] Unable to import msgpack or msgpack_pure python modules
        [ERROR   ] An un-handled exception was caught by salt's global exception handler:
        ImportError: No module named backports.ssl_match_hostname
        Traceback (most recent call last):
          File "/tmp/.root_734552_salt/salt-call", line 4, in <module>
            salt_call()
          File "/tmp/.root_734552_salt/salt/scripts.py", line 325, in salt_call
            import salt.cli.call
          File "/tmp/.root_734552_salt/salt/cli/call.py", line 9, in <module>
            import salt.cli.caller
          File "/tmp/.root_734552_salt/salt/cli/caller.py", line 20, in <module>
            import salt.minion
          File "/tmp/.root_734552_salt/salt/minion.py", line 84, in <module>
            import salt.pillar
          File "/tmp/.root_734552_salt/salt/pillar/__init__.py", line 15, in <module>
            import salt.fileclient
          File "/tmp/.root_734552_salt/salt/fileclient.py", line 30, in <module>
            import salt.utils.http
          File "/tmp/.root_734552_salt/salt/utils/http.py", line 58, in <module>
            import tornado.simple_httpclient
          File "/tmp/.root_734552_salt/tornado/simple_httpclient.py", line 8, in <module>
            from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
          File "/tmp/.root_734552_salt/tornado/http1connection.py", line 30, in <module>
            from tornado import iostream
          File "/tmp/.root_734552_salt/tornado/iostream.py", line 40, in <module>
            from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
          File "/tmp/.root_734552_salt/tornado/netutil.py", line 49, in <module>
            import backports.ssl_match_hostname
        ImportError: No module named backports.ssl_match_hostname
        Traceback (most recent call last):
          File "/tmp/.root_734552_salt/salt-call", line 4, in <module>
            salt_call()
          File "/tmp/.root_734552_salt/salt/scripts.py", line 325, in salt_call
            import salt.cli.call
          File "/tmp/.root_734552_salt/salt/cli/call.py", line 9, in <module>
            import salt.cli.caller
          File "/tmp/.root_734552_salt/salt/cli/caller.py", line 20, in <module>
            import salt.minion
          File "/tmp/.root_734552_salt/salt/minion.py", line 84, in <module>
            import salt.pillar
          File "/tmp/.root_734552_salt/salt/pillar/__init__.py", line 15, in <module>
            import salt.fileclient
          File "/tmp/.root_734552_salt/salt/fileclient.py", line 30, in <module>
            import salt.utils.http
          File "/tmp/.root_734552_salt/salt/utils/http.py", line 58, in <module>
            import tornado.simple_httpclient
          File "/tmp/.root_734552_salt/tornado/simple_httpclient.py", line 8, in <module>
            from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
          File "/tmp/.root_734552_salt/tornado/http1connection.py", line 30, in <module>
            from tornado import iostream
          File "/tmp/.root_734552_salt/tornado/iostream.py", line 40, in <module>
            from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
          File "/tmp/.root_734552_salt/tornado/netutil.py", line 49, in <module>
            import backports.ssl_match_hostname
        ImportError: No module named backports.ssl_match_hostname
    stdout:
>

[nixuser]

Moving one step forward. My investigation on Host: showed that running manually code
dist-packages/salt/utils
In [1]: import thin
In [2]: thin.gen_thin('/tmp/tst')
generates file
thin.tgz that contains
backports/init.py
backports/ssl_match_hostname/init.py
Unfortunately it is not placed to directory on Target:
Probably it missed during unpack procedure, or I am running incorrect command and thin is not related to salt-ssh command.

[nixuser]

Correct steps to reproduce issue.

1. Install Debian 8 from Salt repo
2. Fill roaster file and run command
   salt-ssh '*' test.ping
3. You get error ImportError: No module named backports.ssl_match_hostname
4. Check content of /var/cache/salt/master/thin/thin.tgz
   /usr/lib/python2.7/dist-packages/salt/utils# tar -tzf ~/thin.tgz | grep back

Expected result:
backports module in /var/cache/salt/master/thin/thin.tgz

As workaround you can install package:
aptitude install python-backports.ssl-match-hostname
and remove already generated /var/cache/salt/master/thin/thin.tgz
After that rerun command:
salt-ssh '*' test.ping and it completed successfully.
gs-host0:
True

Also check thin.tgz content:
tar -tzf /var/cache/salt/master/thin/thin.tgz | grep back
backports/init.py
backports/ssl_match_hostname/init.py

[babilen5]

Unfortunately python-backports.ssl-match-hostname is not available on either wheezy or squeeze, so this workaround doesn't work there.

[babilen5]

This bug really renders salt-ssh unusable in larger heterogeneous environments. Is there a way to fix this in salt so that we do not have to rely on the local state of the minion when using it?

[nixuser]

Are you trying to use wheeze and squeeze on host system where salt-ssh is running? Because I installed wheezy and tried to run simple test command and did't see this issue:

# salt-ssh '*' -r 'lsb_release -a'
deb:
    ----------
    retcode:
        0
    stderr:
        No LSB modules are available.
    stdout:
        [email protected]'s password:
        Distributor ID: Debian
        Description:    Debian GNU/Linux 7.9 (wheezy)
        Release:        7.9
        Codename:       wheezy
# salt-ssh '*' test.ping
deb:
    True

python-backports.ssl-match-hostname - copied to target host as content of .tgz file. I guess you can use any distribution for target system.

[babilen5]

The bug is about squeeze or wheezy target systems.

[nixuser]

When wheezy was installed on target system salt-ssh test.ping command completed successfully. On host system I have Debian GNU/Linux 8.3 (jessie). Could you please provide steps to reproduce error and some details about salt version used in your environment?

[babilen5]

For example:

$ salt-ssh --version
salt-ssh 2015.8.7 (Beryllium)
$ salt-ssh -c ~/salt/ceph/ '*node1*' test.ping
foo-node1.example.com:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/tmp/.babilen_1a964f_salt/salt-call", line 4, in <module>
            salt_call()
          File "/tmp/.babilen_1a964f_salt/salt/scripts.py", line 329, in salt_call
            import salt.cli.call
          File "/tmp/.babilen_1a964f_salt/salt/cli/call.py", line 9, in <module>
            import salt.cli.caller
          File "/tmp/.babilen_1a964f_salt/salt/cli/caller.py", line 20, in <module>
            import salt.minion
          File "/tmp/.babilen_1a964f_salt/salt/minion.py", line 84, in <module>
            import salt.pillar
          File "/tmp/.babilen_1a964f_salt/salt/pillar/__init__.py", line 15, in <module>
            import salt.fileclient
          File "/tmp/.babilen_1a964f_salt/salt/fileclient.py", line 30, in <module>
            import salt.utils.http
          File "/tmp/.babilen_1a964f_salt/salt/utils/http.py", line 58, in <module>
            import tornado.simple_httpclient
          File "/tmp/.babilen_1a964f_salt/tornado/simple_httpclient.py", line 8, in <module>
            from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters
          File "/tmp/.babilen_1a964f_salt/tornado/http1connection.py", line 30, in <module>
            from tornado import iostream
          File "/tmp/.babilen_1a964f_salt/tornado/iostream.py", line 40, in <module>
            from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults
          File "/tmp/.babilen_1a964f_salt/tornado/netutil.py", line 49, in <module>
            import backports.ssl_match_hostname
        ImportError: No module named backports.ssl_match_hostname
    stdout:
foo-node1:~$ cat /etc/debian_version 
7.9

[nixuser]

Could you find in cache directory file thin.tgz and check that it contains ssl_match_hostname.py ?
in my case it looks like

$ tar -tzf mysalt/cache/thin/thin.tgz  | grep ssl_match_hostname
salt/ext/ssl_match_hostname.py
backports/ssl_match_hostname/__init__.py

[babilen5]

I get the following:

$ tar -tzf thin.tgz  | grep ssl_match_hostname
salt/ext/ssl_match_hostname.py
salt/ext/ssl_match_hostname.py

So backports.ssl_match_hostname is obviously missing.

$ salt-ssh --versions-report
Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: Not Installed
           Mako: 1.0.3
         PyYAML: 3.11
          PyZMQ: Not Installed
         Python: 2.7.11+ (default, Feb 22 2016, 16:38:42)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: debian stretch/sid 
        machine: x86_64
        release: 4.4.0-1-amd64
         system: debian stretch/sid

[nixuser]

It is strange that backports/ssl_match_hostname/init.py file missing. However this is other issue not related to target machines. Something goes wrong during thin.tgz preparation, but I can't reproduce it on my environment. Do you have python-backports.ssl-match-host installed from pip or deb package? Try to remove thin.tgz and re-run command one more time.

[babilen5]

Deleting thin.tgz did indeed cause salt to re-generate it and it included backports/ssl_match_hostname/__init__.py this time around. I am not sure what caused it to be missing in the first place though.

Unfortunately I had to login to the minions manually and delete /tmp/.$USER_$RANDOM_salt before I could use salt-ssh successfully. I am not entirely sure why salt didn't upload the, now updated, thin.tgz, but reused what appeared to be an outdated version.

[babilen5]

I guess this issue can be closed now, can't it?

[jfindlay]

Closing according to @babilen and @nixuser's suggestions. If anyone can still reproduce this, we will reopen the issue.

[nate-byrnes]

I had some centos 6 boxes where this was occurring.

I brute-forced a work around with:

salt-ssh '*' -r "yum install -y python-setuptools"
salt-ssh '*' -r 'easy_install backports.ssl_match_hostname'

not particularly graceful.... the boxes are 6.5 and 6.6 and my salt-ssh is 2016.3.1

[mirko]

I'm having the same issue on an Ubuntu 12.04 machine using latest salt-ssh release on Debian Jessie.
It's the only target it fails on - even after deleting /tmp/.$USER_$RANDOM_salt - so it might be a good machine to debug this issue on.
I can also say, that it's not the only Ubuntu 12.04 target I have and it works fine on those other ones.