salt-ssh sync_grains not working

[guettli]

Description of Issue/Question

I can't see my custom grains. I use salt-ssh

Steps to Reproduce Issue

salt:/srv # cat salt/_grains/mygrain.py
def mygrain():
    return dict(mygrain='grain ooooooooooooooooooo')

salt:/srv # salt-ssh w123 saltutil.sync_grains
w123:

salt:/srv # salt-ssh w123 grains.items | grep oooo  # --> no match

Versions Report

salt:/srv # salt --versions-report
Salt Version:
           Salt: 2015.8.3

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.22
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.1.0
         Python: 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC]
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.1
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: 0.21.2
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: SuSE 12.3 x86_64
        machine: x86_64
        release: 3.7.10-1.40-desktop
         system: openSUSE  12.3 x86_64

What can I do?

Dear experts, can you please advice me (new to salt) how to debug this?

[Ch3LL]

@guettli I am also seeing this behavior in 2015.8.9 and 2016.3.0. Looks like sync_modules is not working either. Looks like we need to get this fixed to be able to add custom modules/grains.

[guettli]

Is there a known work around? Some time later have seen my custom grains on minions. Maybe this bug happens, since we use salt-ssh and not salt. There is no minion daemon on our minions.

[jkramarz]

@danslimmon, is it sufficient to place custom.py in library (eg. /usr/lib/python2.7/site-packages/salt/grains) of salt-ssh "master" or should I somehow notify loader about its presence, if I'm not willing to rebuild Salt from scratch?

[thatch45]

@guettli if you run salt-ssh with the -w flag does the problem go away? I bet this has to do with how the thin tarball is created

[thatch45]

@jkramarz that should be sufficient, although you will need to trigger recreation of the thin tarball, it is only recreated if the salt version changes or it is not present, so you can remove it from the "master" cache to trigger a rebuild, it is located in the cachedir under thin, /var/cache/salt/master/thin/ in the default config

[guettli]

salt:/srv # salt-ssh -w lkw-cips grains.get mygrain
lkw-cips:
    grain old-message

salt:/srv # vi salt/_grains/mygrain.py
modified code to this:

def mygrain():
    return dict(mygrain='grain new :-)')

salt:/srv # salt-ssh lkw-cips saltutil.sync_grains 

salt:/srv # salt-ssh -w lkw-cips grains.get mygrain
lkw-cips:
# No output?

salt:/srv # ssh root@lkw-cips
lkw-cips:~ # rm -rf /var/tmp/.root_dcdf8c_salt

salt:/srv # salt-ssh -w lkw-cips grains.get mygrain
lkw-cips:
    grain new :-)
 # Hurray! New message is visible

salt:/srv # salt-ssh --version
salt-ssh 2016.3.1 (Boron)

The -w option does not work in my case. Any hints?

[jkramarz]

@thatch45 thank you, I've removed ./cachedir/thin/thin.tgz and it works for me.

[thatch45]

@guettli could you try to force generating a new thin tarball as well? I think that is the solution and that we should update -w to also generate a new thin tarball, or add an option to do so. I see the problem we are having but I just want to figure out the best way to solve it.....

[guettli]

Ahhh I was reading too fast. I thought -w does a wipe... yes it does, but only after the run. I need to use the -W option with capital letter. Sorry I am trying it again.

Here is the current graint. Everything is in sync.

salt:/srv # salt-ssh lkw-cips grains.get mygrain
lkw-cips:
    grain ooooooooooooooooooo

I update the file. I add -WWW

salt:/srv # vi salt/_grains/mygrain.py

The next line works, nice: I see -WWW.

salt:/srv # salt-ssh -W lkw-cips grains.get mygrain
lkw-cips:
    grain -WWW ooooooooooooooooooo

I update it again.

salt:/srv # vi salt/_grains/mygrain.py

I sync grains

salt:/srv # salt-ssh lkw-cips saltutil.sync_grains
lkw-cips:

Next line: no output? That's a bug:

salt:/srv # salt-ssh lkw-cips grains.get mygrain
lkw-cips:

With salt-ssh -W the grains get installed.

salt:/srv # salt-ssh -W lkw-cips grains.get mygrain
lkw-cips:
    grain -WWW 222222222 ooooooooooooooooooo

My conclusion: sync_grains is broken in 2016.3.1, and the work-around is to use salt-ssh -W.

[edgan]

Can this be backported to 2016.3? I just ran into this

[austinpapp]

Seems even /etc/salt/grains is having the same issue here...

[alxwr]

This seems to be still an issue in 2018.3.2.

[ghost]

Persists in 2019.2.0.

[nix-power]

just to clarify: Is there any reasonable workaround ? salt-ssh 2018.3.3
should i put my custom python file under /etc/salt/grains or under /srv/salt/_grains ?

[nix-power]

I see my files under /var/tmp/./runnning_data/var/cache/salt/minion/extmods/grains and i was able to run at as python script, but when i run salt-ssh -iW nodename grains.items i don`t see it in output at all

[guettli]

I opened this issue three yeas ago. I do not know what the salt developers think about this. But I know what I will do. I will switch to ansible.

[nix-power]

actually it does work quite fine if calling salt-ssh with "iWt" keys

salt-ssh -iWt "nodename" grains.items
"t" is for rebuilding thin tgz package
"W" to remove all files after the deployment

[Ch3LL]

thanks for posting that @nix-power looks like this PR https://github.com/saltstack/salt/pull/34974 was added to address this issue. Are we okay to close the issue?

[nix-power]

I didn't open original issue, but I have checked now in 3 different environments, and it always works.
Custom grains to be placed under /srv/salt/_grains
salt-ssh -iWt
Salt versions checked: 2018.3 and 2019.2

[Ch3LL]

thanks i'll go ahead and close since its been confirmed, but can re-open if needed

[nergdron]

I'm not sure how it was fixed for others, but I'm still seeing this issue with 2019.2.0. I've got a very simple grain which just exposes the running user's home dir as a test, in _grains/dir.py:

from pathlib import Path

def home_dir():
    grains = {}
    grains['home_dir'] = str(Path.home())
    return grains

it works locally with salt-call, but with salt-ssh 'hostname' -Wt grains.get home_dir, I get nothing back, and inspecting the copied thin dir, I don't see it synced there at all with other basic grains.