salt-ssh fails to render pillar data in the latest develop code

[kt97679]

Hi folks,

I'm using salt-ssh from the develop branch because I need some latest features and fixes. I have snapshot from september 24 (salt-ssh 2015.8.0-454-g63c3cd3 (Beryllium)) which works ok for me. Today I tried latest develop code (salt-ssh 2015.8.0-684-g066c1df (Beryllium)) since it should contain another set of important fixes but discovered that pillars are not rendered at all. Is this a known issue? Any advise on the workaround?

Please let me know if you need any specific information I should provide.

Thanks,
Kirill.

[jfindlay]

@kt97679, thanks for the report. Is it possible for you to provide a minimal example? Are there any configs or other circumstances that might be unusual? Thanks.

[jfindlay]

@kt97679, also, what platforms are you working with? Is there a difference between the master and ssh minion?

[kt97679]

Hi @jfindlay, I'm using purely salt-ssh so unfortunately I can't compare with master-minion setup. I'm using ubuntu 14 with python 2.7.6 installed from the packages. I'm trying to identify root cause right now so if I'll be able to provide minimal example I'll definitely do it. At this point I can tell you that I see data in the salt/loader.py here:

        if '__pillar__' not in self.pack:
            self.context_dict['pillar'] = opts.get('pillar', {})
            self.pack['__pillar__'] = salt.utils.context.NamespacedDictWrapper(self.context_dict, 'pillar')

but don't see it in the raw() function in the salt/client/ssh/wrapper/pillar.py

[jfindlay]

@kt97679, thanks for the extra info. Does the ssh minion also run on ubuntu 14?

[kt97679]

Yes, client is also u14.

[kt97679]

I feel that something is wrong with the LazyLoader in the salt/loader.py. From what I see empty pillar is set there. Unfortunately logic is quite complicated, I'll try to isolate issue further.

[kt97679]

This seems to fix an issue, though I'm not completely sure because logic is really quite obscure.

*** 939,944 ****
--- 939,945 ----
          if '__context__' not in self.pack:
              self.pack['__context__'] = None

+         self.pack['__pillar__'] = opts.get('pillar', {})
          for k, v in six.iteritems(self.pack):
              if v is None:  # if the value of a pack is None, lets make an empty dict
                  self.context_dict.setdefault(k, {})

[kt97679]

I also had to change salt/client/ssh/state.py:

***************
*** 143,149 ****
          fp_.write(json.dumps(chunks))
      if pillar:
          with salt.utils.fopen(pillarfn, 'w+') as fp_:
!             fp_.write(json.dumps(pillar._dict()))
      for saltenv in file_refs:
          file_refs[saltenv].extend(sync_refs)
          env_root = os.path.join(gendir, saltenv)
--- 143,149 ----
          fp_.write(json.dumps(chunks))
      if pillar:
          with salt.utils.fopen(pillarfn, 'w+') as fp_:
!             fp_.write(json.dumps(pillar))
      for saltenv in file_refs:
          file_refs[saltenv].extend(sync_refs)
          env_root = os.path.join(gendir, saltenv)

[josuebrunel]

Hi guys, is this already fixed ? Is there a patch somewhere we can apply ?

[jfindlay]

@josuebrunel, not that I know of. @kt97679, did you get a chance to test your fixes?

[kt97679]

@jfindlay my fixes work for me, but unfortunately salt implementation uses very obscure and nontrivial techniques so I'm not 100% that my fix is absolutely correct.

[eradman]

I'm hitting this issue in salt 2017.7.2: salt-ssh pillar.items is correct, but the __pillar__ dict is empty in custom execution modules

The changes @kt97679 posted don't apply to the latest release of salt. Is there a known patch that I can test against 2017.7.2?

[braunse]

I'm hitting the same problem in 2018.3.3. Both __pillar__ dict and even the output of __salt__["pillar.get"](...) are empty in my modules.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[Skeen]

I think I'm hitting this on salt-ssh 3000.2.

Running salt-ssh target pillar.items yields:

target:
    ----------
    docker-registries:
        ----------
        hub:
            ----------
            password:
                REDACTED
            username:
                REDACTED

However, running salt-ssh salt-master-test docker.login hub yields:

target:
    ----------
    Errors:
        - No match found for registry 'hub'
    Results:
        ----------
    retcode:
        1

Not the expected succes.

Entering the server, and modifying /var/tmp/.root_cb71e2_salt/pyall/salt/modules/dockermod.py, to print pillar.items just before returning said error:

    results = ret.setdefault('Results', {})
    for registry in registries:
        if registry not in registry_auth:
            errors.append(                                           # ADDED
                 'Debug \'{0}\''.format(__salt__['pillar.items']())  # ADDED
            )                                                        # ADDED
            errors.append(
                'No match found for registry \'{0}\''.format(registry)
            )
            continue

Yields the following output on the next run of salt-ssh salt-master-test docker.login hub:

target:
    ----------
    Errors:
        - Debug '{}'
        - No match found for registry 'hub'
    Results:
        ----------
    retcode:
        1

Indicating that the pillar is indeed empty, thus explaning the failure.

[Skeen]

I have made a minimal example producing the issue here: https://github.com/Skeen/missing_pillar_minimal_example

[Ch3LL]

im going to remove the regression label as this is not a regression. Custom modules have not worked on salt-ssh previously if they were trying to access pillar.

This issue is going to require a bit more thought, because we cant just add custom modules to the wrapper function because it would require users to write their modules to be cognitive of running commands on the remote target.