salt-ssh uses sudo to create cache dir, later fails to access it

Created on 28 Dec 2016 · 14Comments · Source: saltstack/salt

Description of Issue/Question

Any states using the file cache fail in salt-ssh 2016.11.1 due to permission problems. Apparently the file is created as root but later accessed with user rights.

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ salt-ssh --refresh --user $(whoami) -L 'localhost' state.sls test test=True -l debug
[INFO    ] Loading Saltfile from '/srv/saltstack/Saltfile'
[DEBUG   ] Reading configuration from /srv/saltstack/Saltfile
[DEBUG   ] Reading configuration from /srv/saltstack/etc/master
[DEBUG   ] Configuration file path: /srv/saltstack/etc/master
....
[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var/cache/salt/minion/salt-ssh'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 967, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 95, in sls
    st_kwargs['id_'])
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/state.py", line 204, in prep_trans_tar
    path = file_client.cache_file(name, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 178, in cache_file
    return self.get_url(path, '', True, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 486, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1025, in get_file
    rel_path, saltenv, cachedir=cachedir) as cache_dest:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 150, in _cache_loc
    os.makedirs(destdir)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var/cache/salt/minion/salt-ssh'
[DEBUG   ] LazyLoaded nested.output
localhost:
    An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var/cache/salt/minion/salt-ssh'

$ ls -al /var/tmp/
total 28
drwxrwxrwt  7 root     root    4096 Dec 28 11:26 .
drwxr-xr-x 12 root     root    4096 Nov 23  2015 ..
drwxrwx---  4 root     tumuser 4096 Dec 28 11:33 .erlacher_f62e6b_salt

Setup

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat states/test/
init.sls  test.foo  

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat states/test/init.sls 
'/tmp/foo':
  file.managed:
    - source: salt://test/test.foo

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat states/test/test.foo 
testfoo

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ grep -A 5 'localhost' etc/roster 
localhost:
  host: vmrbg51
  sudo: True
ubuntumirror:
  host: vmrbg59
  sudo: True

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat Saltfile 
salt-ssh:
  config_dir: etc
  ssh_max_procs: 30
  wipe_ssh: true

erlacher at vmrbg51 in /srv/saltstack on default <!> 
$ cat etc/master 
pki_dir: /srv/saltstack/pki/
pillar_roots:
  base:
  - /srv/saltstack/pillar
file_roots:
  base:
  - /srv/saltstack/states
  - /srv/saltstack/secret
ssh_minion_opts:
  log_level: debug
  log_level_logfile: debug
  log_file: /var/log/salt/minion
ssh_list_nodegroups:
  lsvpn: vmrbg44, localhost

Steps to Reproduce Issue

See above

Versions Report

$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-93-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Happens also with repository version of salt -

$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.0-562-g5cba84c

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-93-generic
         system: Linux
        version: Ubuntu 14.04 trusty

EDIT: Does not happen with v2016.9 tag (f76dc0f9c06dd0690447a31544b7bd1fe7f5765a) from repo

Bug Core P2 Salt-SSH ZRELEASED - 2016.11.4 fixed-pending-your-verification severity-critical severity-high

Source

duk3luk3

Most helpful comment

Thanks for reporting this, I am able to reproduce the issue, we will get it fixed.

Thanks,
Daniel

gtmanfred on 3 Jan 2017

👍2 ❤1

All 14 comments

Having the same issue after an upgrade (from a much older version though).

$ salt-ssh 'X' 'state.sls' 'backup'
[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.Y_7c7bc9_salt/running_data'

mirko on 2 Jan 2017

Thanks for reporting this, I am able to reproduce the issue, we will get it fixed.

Thanks,
Daniel

gtmanfred on 3 Jan 2017

👍2 ❤1

Is there a schedule for this, or a workaround? I think that this breaks salt-ssh for anyone using sudo.

duk3luk3 on 9 Feb 2017

Let me ask and see if we have dev time to fix this in 2016.11.3

gtmanfred on 9 Feb 2017

It looks like we are going to be getting in all the 2016.11.3 blockers in this week, so we aren't going to be able to get this one in. It will be a blocker for 2016.11.4, so it will be in that next release.

Thanks,
Daniel

gtmanfred on 9 Feb 2017

I think I ran across this bug, but there also seems to be a twist with trying to run without sudo on the master (where you run salt-ssh from). It seems to be trying to create file cache directories on the salt master but using the thin_dir from the roster, which is out of sync with the default behavior of everything else.

So with this roster:

minion-name:
  host: 1.2.3.4
  user: ubuntu
  priv: /path/to/key-file
  sudo: True
  thin_dir: /.salt-ssh-thin-dir

I get:

% salt-ssh -l debug [...]state.apply <state-using-file://something>
[...]
[DEBUG   ] In saltenv 'base', looking at rel_path 'file-settings/foobar' to resolve 'salt://file-settings/foobar'
[ERROR   ] An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/.salt-ssh-thin-dir'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 970, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 288, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 95, in sls
    st_kwargs['id_'])
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/state.py", line 204, in prep_trans_tar
    path = file_client.cache_file(name, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 179, in cache_file
    return self.get_url(path, '', True, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 487, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1029, in get_file
    rel_path, saltenv, cachedir=cachedir) as cache_dest:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 151, in _cache_loc
    os.makedirs(destdir)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/.salt-ssh-thin-dir'

I added some debug to see that it's trying to operate on this as destdir:

/.salt-ssh-thin-dir/running_data/var/cache/salt/minion/salt-ssh/minion-name/files/base/file-settings

Since none of this exists on the master, it fails at the first path component.

On the minion itself, /.salt-ssh-thin-dir/running_data/var/cache/salt/minion/ does exist, but is also owned by root, so it probably wouldn't work because of a lack of remote sudo, just like the original bug reporter mentioned.

shallot on 16 Mar 2017

👍1

For the benefit of random users who might be reading this dazed and confused at why salt:// file URIs with file source don't work with salt-ssh, it should probably be noted that this issue - while most annoying - isn't necessarily critical, because many use cases of source can be replaced with the use of contents, esp. with external pillars, which does seem to work fine with salt-ssh.

shallot on 17 Mar 2017

This appears to have been introduced here, https://github.com/saltstack/salt/commit/71e0bd023f480ab635ea1186e5dae3a8e0ed26ff

gtmanfred on 30 Mar 2017

@shallot @duk3luk3 does this happen on anything that is not localhost?

I am testing against other machines, and not getting the same error.

gtmanfred on 30 Mar 2017

Yes this definitely causes the same failure when executing the same state on localhost or a remote minion.

erlacher at vmrbg51 in /srv/saltstack on default < >
$ salt-ssh --refresh --user $(whoami) -L 'localhost' state.sls hostdb test=True
[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'
localhost:
    An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'

erlacher at vmrbg51 in /srv/saltstack on default <!>
$ salt-ssh --refresh --user $(whoami) -L 'vmrbg6' state.sls hostdb test=True

[ERROR   ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'
vmrbg6:
    An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.erlacher_f62e6b_salt/running_data/var'

erlacher at vmrbg51 in /srv/saltstack on default <!>
$ salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-93-generic
         system: Linux
        version: Ubuntu 14.04 trusty

EDIT: I just tested it with the test state I set out in the initial report too, same results.

EDIT: Here is a self-contained reproduction: https://github.com/duk3luk3/salt-ssh-minimal/tree/salt-38458. It reproduces both from my arch linux workstation using salt-ssh 2016.11.3 as well as from my normal salt master machine using 2016.11.2.

EDIT: Your PR seems to fix it though! 👍

duk3luk3 on 31 Mar 2017

It actually will work if you remove /var/tmp/*_salt and run it against a remote minion, it will work.

The problem was the commit I linked, accidentally moved the file cache on the master for salt:// links from /var/cache/salt/master/ to /var/tmp/<user>_<uuid>_salt/running_data, which would get overwritten when the thin tarball gets copied to the salt-ssh minion, and since you were using sudo, the running_data directory is owned by root:root on the minion, so this causes it to fail when the salt-ssh command tries to copy over more files for the salt_states.tgz

gtmanfred on 31 Mar 2017

The above commit has been merged, I am closing this issue

Thanks!
Daniel

gtmanfred on 31 Mar 2017

Confirming that https://github.com/gtmanfred/salt/commit/2f0e2edd3b4f9734d5e0d8617e9b84eefe16b755 on top of 2016.11.3 fixes the problem, thanks!
When will this be in a released version?

shallot on 4 Apr 2017

It will be in 2016.11.4 .

https://github.com/saltstack/salt/issues?q=is%3Aopen+is%3Aissue+label%3A2016.11.4+label%3ABlocker

gtmanfred on 4 Apr 2017

Was this page helpful?

0 / 5 - 0 ratings