Salt: Since you support {{ pillar.get('var') }} in pillar files, then please make {{ pillar['var'] }} work

[ryba-xek]

Bsed on this discussion: https://groups.google.com/forum/#!topic/salt-users/gH7DHC0Ck88

{{ pillar['var'] }} is not working when it is used in a pillar .sls file referencing other pillar .sls file.

Here is a demo I just made:

ubuntu@ubuntu:~$ tree
.
└── pillar
    ├── file1.sls
    ├── file2.sls
    └── top.sls

1 directory, 3 files

ubuntu@ubuntu:~$ cat pillar/top.sls
base:
  '*':
    - file1
    - file2

ubuntu@ubuntu:~$ cat pillar/file1.sls
foo: bar

ubuntu@ubuntu:~$ cat pillar/file2.sls
foo2: {{ pillar['foo'] }}

ubuntu@ubuntu:~$ sudo salt-call pillar.data

[INFO    ] Loaded configuration file: /etc/salt/minion
[CRITICAL] Pillar render error: Rendering SLS file2 failed, render error:
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 55, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 98, in render_jinja_tmpl
    output = jinja_env.from_string(tmplstr).render(**context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 894, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 2, in top-level template code
UndefinedError: 'dict object' has no attribute 'foo'

[CRITICAL] Pillar render error: Rendering SLS file2 failed, render error:
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 55, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 98, in render_jinja_tmpl
    output = jinja_env.from_string(tmplstr).render(**context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 894, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 2, in top-level template code
UndefinedError: 'dict object' has no attribute 'foo'

ubuntu@ubuntu:~$ vi pillar/file2.sls
ubuntu@ubuntu:~$ cat pillar/file2.sls
foo2: {{ pillar.get('foo') }}

ubuntu@ubuntu:~$ sudo salt-call pillar.data

[INFO ] Loaded configuration file: /etc/salt/minion
foo:
bar
foo2:
bar
master:
----------
acceptance_wait_time:
10
arg:
autoload_dynamic_modules:
True
backup_mode:

cache_jobs:
    False
cachedir:
    /var/cache/salt/minion
caller:
    True
clean_dynamic_modules:
    True
conf_file:
    /etc/salt/minion
config_dir:
    /etc/salt
cython_enable:
    False
default_include:
    minion.d/*.conf
disable_modules:
disable_returners:
dns_check:
    True
doc:
    False
environment:
    None
extension_modules:
    /var/cache/salt/minion/extmods
external_nodes:

failhard:
    False
file_client:
    local
file_roots:
    ----------
    base:
        - /home/ubuntu/pillar
fun:
    pillar.data
grains_run:
    False
hash_type:
    md5
id:
    ubuntu
ipc_mode:
    ipc
json_out:
    False
local:
    False
log_datefmt:
    %H:%M:%S
log_datefmt_logfile:
    %Y-%m-%d %H:%M:%S
log_file:
    /var/log/salt/minion
log_fmt_console:
    [%(levelname)-8s] %(message)s
log_fmt_logfile:
    %(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s] %(message)s
log_granular_levels:
    ----------
log_level:
    info
loop_interval:
    60
master:
    salt
master_finger:

master_port:
    4506
module_dirs:
multiprocessing:
    True
no_color:
    False
open_mode:
    False
outputter_dirs:
permissive_pki_access:
    False
pidfile:
    /var/run/salt-minion.pid
pillar:
    ----------
    foo:
        bar
    foo2:
        None
    master:
        ----------
        acceptance_wait_time:
            10
        arg:
        autoload_dynamic_modules:
            True
        backup_mode:

        cache_jobs:
            False
        cachedir:
            /var/cache/salt/minion
        caller:
            True
        clean_dynamic_modules:
            True
        conf_file:
            /etc/salt/minion
        config_dir:
            /etc/salt
        cython_enable:
            False
        default_include:
            minion.d/*.conf
        disable_modules:
        disable_returners:
        dns_check:
            True
        doc:
            False
        environment:
            None
        extension_modules:
            /var/cache/salt/minion/extmods
        external_nodes:

        failhard:
            False
        file_client:
            local
        file_roots:
            ----------
            base:
                - /home/ubuntu/pillar
        fun:
            pillar.data
        grains_run:
            False
        hash_type:
            md5
        id:
            ubuntu
        ipc_mode:
            ipc
        json_out:
            False
        local:
            False
        log_datefmt:
            %H:%M:%S
        log_datefmt_logfile:
            %Y-%m-%d %H:%M:%S
        log_file:
            /var/log/salt/minion
        log_fmt_console:
            [%(levelname)-8s] %(message)s
        log_fmt_logfile:
            %(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s] %(message)s
        log_granular_levels:
            ----------
        log_level:
            info
        loop_interval:
            60
        master:
            salt
        master_finger:

        master_port:
            4506
        module_dirs:
        multiprocessing:
            True
        no_color:
            False
        open_mode:
            False
        outputter_dirs:
        permissive_pki_access:
            False
        pidfile:
            /var/run/salt-minion.pid
        pillar_roots:
            ----------
            base:
                - /home/ubuntu/pillar
        pki_dir:
            /etc/salt/pki/minion
        providers:
            ----------
        raw_out:
            False
        recon_max:
            5000
        render_dirs:
        renderer:
            yaml_jinja
        retry_dns:
            30
        return:

        returner_dirs:
        root_dir:
            /
        saltversion:
            0.13.3
        selected_output_option:
            output_indent
        sls_list:
        sock_dir:
            /var/run/salt/minion
        startup_states:

        state_output:
            full
        state_top:
            salt://top.sls
        state_verbose:
            True
        states_dirs:
        tcp_keepalive:
            True
        tcp_keepalive_cnt:
            -1
        tcp_keepalive_idle:
            300
        tcp_keepalive_intvl:
            -1
        tcp_pub_port:
            4510
        tcp_pull_port:
            4511
        test:
            False
        text_out:
            False
        top_file:

        update_restart_services:
        update_url:
            False
        user:
            root
        verify_env:
            True
        whitelist_modules:
        win_repo_cachefile:
            salt://win/repo/winrepo.p
        yaml_out:
            False
pillar_roots:
    ----------
    base:
        - /home/ubuntu/pillar
pki_dir:
    /etc/salt/pki/minion
providers:
    ----------
raw_out:
    False
recon_max:
    5000
render_dirs:
renderer:
    yaml_jinja
retry_dns:
    30
return:

returner_dirs:
root_dir:
    /
saltversion:
    0.13.3
selected_output_option:
    output_indent
sls_list:
sock_dir:
    /var/run/salt/minion
startup_states:

state_output:
    full
state_top:
    salt://top.sls
state_verbose:
    True
states_dirs:
tcp_keepalive:
    True
tcp_keepalive_cnt:
    -1
tcp_keepalive_idle:
    300
tcp_keepalive_intvl:
    -1
tcp_pub_port:
    4510
tcp_pull_port:
    4511
test:
    False
text_out:
    False
top_file:

update_restart_services:
update_url:
    False
user:
    root
verify_env:
    True
whitelist_modules:
win_repo_cachefile:
    salt://win/repo/winrepo.p
yaml_out:
    False

[thatch45]

We should consolidate these, but this will not work the way proposed. These changes are going to take some serious thought, I am not sure yet how to implement this.

[ryba-xek]

I have just tried master-minion setup and what I want to tell you in not-stand-alone scheme even {{pillar.get('var')}} is not working. That's a pity ;(

[timcharper]

The example specified here in the docs would satisfy our requirements:

https://salt.readthedocs.org/en/latest/topics/pillar/#including-other-pillars

However, it appears that said functionality is not implemented. I've read the code https://github.com/saltstack/salt/blob/v0.16.0/salt/pillar/__init__.py#L294 and it looks like the only include support specified there is:

include:
  - pillar-file-1
  - pillar-file-2

where 'pillar-file-1' becomes the referenced sub_sls variable.

If you make any of the includes a dict, like below, it fails (beneath):

include:
  - pillar-file-1:
      default:
        - sudo: ['bob', 'paul']

the error:

Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/pymodules/python2.7/salt/scripts.py", line 76, in salt_call
    client.run()
  File "/usr/lib/pymodules/python2.7/salt/cli/__init__.py", line 255, in run
    caller = salt.cli.caller.Caller(self.config)
  File "/usr/lib/pymodules/python2.7/salt/cli/caller.py", line 47, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/pymodules/python2.7/salt/minion.py", line 207, in __init__
    self.gen_modules()
  File "/usr/lib/pymodules/python2.7/salt/minion.py", line 217, in gen_modules
    self.opts['environment'],
  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 369, in compile_pillar
    pillar, errors = self.render_pillar(matches)
  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 320, in render_pillar
    pstate, mods, err = self.render_pstate(sls, env, mods)
  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 298, in render_pstate
    if sub_sls not in mods:
TypeError: unhashable type: 'dict'

[dimasmjunior]

I'm having the same problem. Spent some time trying to run the example shown in the documentation, getting the exact same error. Ended up here after googling it.

[boltronics]

Confirming that I also get the same explody when including other pillars using the second documented form. Just spent a half an hour on it.

The docs say "New in version 0.16.0". Would be nice to either remove this from the docs, or implement it.

[dimasmjunior]

I just recently discovered "reclass", a piece of software that does parametrization in a different way, with decent class inheritance support. You may want to take a look: http://reclass.pantsfullofunix.net/

[basepi]

@boltronics Can you create a separate issue? If the include syntax isn't working, we need to look into that. Please include as much info as you can, including your pillar files and version info.

[timcharper]

@basepi please see my comment where I point out where I'd expect to see the "second documented form" implemented, but there is clearly no support for additional options other than vanilla pillar include, which does happen to work.

The action should be:

A) Find where the code went that was supposed to implement the advertised functionality. Write some tests for it.
B) Fix the documentation to not advertise functionality that simply does not work.

[basepi]

Yes, if the syntax in the docs doesn't work, that's definitely a problem. Thanks for all the info, everyone.

[basepi]

I'm thinking this issue may have been fixed since this issue was last referenced -- I think we had a bug that made pillar not always compile in order, but I'm pretty sure that's resolved. Can anyone still reproduce this issue?

[cachedout]

Since we've had no reply to the request from @basepi and we're fairly certain we have this issue resolved, I'm going to go ahead and closed this. If this proves not to be fully resolved, please leave a comment on this issue and we'll happily re-examine it. Thanks!

[colinp85]

Looks like the example code from the docs is working. Just wanted to get confirmation on one of the other points from this issue

@ryba-xek mentioned aboved that {{ pillar.get('var') }} doesn't work in master-minion. Can you confirm that's the case/is expected? That's what I'm seeing also - can raise a bug with recreation if should work

[cachedout]

It works correctly between a master and a minion for me on 2014.7:

mp@silver ~ % sudo salt silver pillar.get foo
silver:
    bar

root@silver:/srv/salt# cat issue_4326.sls 
test.echo:
  module.run:
  - text: {{ pillar.get('foo') }}

mp@silver ~ % sudo salt silver state.sls issue_4326
silver:
----------
          ID: test.echo
    Function: module.run
      Result: True
     Comment: Module function test.echo executed
     Started: 09:28:04.023682
    Duration: 1.01 ms
     Changes:   
              ----------
              ret:
                  bar

Summary
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1

[colinp85]

I may be trying to do something that's invalid/is not expected to work but - I am using pillar.get within the pillar as @ryba-xek did in the first post on this issue. Here's what I've got - expected output would be that variable_two would be 100 also:

[root@cp-nas pillar]# cat pillar_one.sls
variable_one: 100

[root@cp-nas pillar]# cat pillar_two.sls
variable_two: {{ pillar.get('variable_one', 'Not 100') }}

[root@cp-nas pillar]# cat top.sls
base:
    '*':
        - pillar_one
        - pillar_two

[root@cp-nas pillar]# salt '*' pillar.items
cp-nas:
    ----------
    variable_one:
        100
    variable_two:
        Not 100

[jfindlay]

I've reproduced and resynthesized @cachedout and @colinp85's last comments (the previous two comments) thus:

configuration:

centos-7-main ~ master # salt --versions
           Salt: 2014.7.0-163-gc31bcb3
         Python: 2.7.5 (default, Jun 17 2014, 18:11:42)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.4.2
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.4
centos-7-main ~ master # cat /srv/salt/echo.sls
test.echo:
  module.run:
  - text: {{ pillar.get('one') }}
centos-7-main ~ master # cat /srv/pillar/top.sls 
base:
    '*':
        - one
        - two
centos-7-main ~ master # cat /srv/pillar/one.sls 
one: 100
centos-7-main ~ master # cat /srv/pillar/two.sls 
two: {{ pillar.get('one', 'Not 100') }}

execution:

centos-7-main ~ master # salt centos-7-main state.sls echo
centos-7-main:
----------
          ID: test.echo
    Function: module.run
      Result: True
     Comment: Module function test.echo executed
     Started: 16:54:52.121020
    Duration: 0.422 ms
     Changes:   
              ----------
              ret:
                  100

Summary
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
centos-7-main ~ master # salt centos-7-main saltutil.refresh_pillar
centos-7-main:
    True
centos-7-main ~ master # salt centos-7-main pillar.get one
centos-7-main:
    100
centos-7-main ~ master # salt centos-7-main pillar.get two
centos-7-main:
    Not 100

The conclusion is that while pillar.get('one') will resolve as intended within a state file, it does not when used within a pillar file.

[basepi]

Oh, right, I completely misread @cachedout's example. It definitely doesn't test the same problem.

Now, the question is whether we guarantee the availability of previous pillar values during pillar compilation. I don't think we've written that feature in. ext_pillar does get the previous pillar data, but that's about it. I don't think pillar is designed to self-reference in that way, so the labels are correct on this issue, this is a new feature we should add.

[anlutro]

Has there been any work on this somewhere? I swear this was working for me yesterday on the 2015.2 branch, but today it's not working.

EDIT: Trying to find a commit where it does work, I can't, so I guess I must've been imagining things all this time :(

[sbreidba]

Same for me - this had been working, but no longer is (as of about your time frame as well). My guess is that we were just getting lucky in terms of pillar load ordering or something similar.

[simonclausen]

I would find it very useful to be able to reference other pillar data from within a pillar.

In my specific use case I need a higher level of abstraction to target configuration options from pillar data than matching on minion id (or grouping minions), where I would define a bunch of configuration groups (i.e. location) within a configuration type (i.e. specific type of service). An object defining the configuration would also include a list of minions to which the configuration should be targeted.

This can be done via a map in a state, as the targeting and configuration targeting descriptions are less confidential than the keys and access credentials to be distributed.

But I find it more elegant to leave the specific configuration and targeting of these in pillar, rather than states, to support modularity, low coupling and ability to apply different configurations across different environments via the same state files (think formulas).

[simonclausen]

Btw, another way to do this (or workaround, however you look at it), is to load the data structure in question as a dictionary with i.e. load_yaml in the same pillar file. This also supports targeting and selecting the data structure in question with jinja conditionals.

If course you don't get the benefits of using the delimiter functionality in salt['pillar.get'].

[rickh563]

ZD-911

[ixs]

Generally the ability for a pillar to fetch other pillar data is becoming more important nowadays. Think a pillar that fetches data from a database but needs login credentials. We can work around a lot of problems by using import_yaml but a simple __pillar__.get() in a py rendered pillar or a salt'pillar.get' as the jinja equivalent is the cleaner solution.

I have a few use-cases where having the pillar dunder dictionary around would make my life much easier. Especially coming from the ansible world where it is very common to have group_vars that reference other group_vars.

The problem with these self-referential variables is the dependency ordering to make sure that the accessed pillars have been rendered already when they are requested.
A simple approach, which might not be the most efficient one but works as it's based on observing Ansible:
Run the rendering in a loop. If a {{ jinja.variable }} does not resolve yet, reinsert the "{{ jinja.variable }}" tag unchanged and wait for the next loop iteration. That's basially lazy resolving and after a small number of iterations everything has been resolved. One could protect against endless runs by verifying if the number of unresolved vars is decreasing and if not, it's time for a break and throwing an error.

@thatch45: Maybe that's a quick way to get started on pillar-in-pillar support? Would this be an acceptable approach?

[Pourliver]

Any update on this? It would be really useful to be able to reference some pillar in other pillar, almost mandatory in some setup. (without doing hacky stuff)

[sbreidba]

Pillarstack is the best approach I've seen so far.

On Wed, Mar 7, 2018 at 2:07 PM, Maxime Carbonneau [e-mail]
wrote:

Any update on this? It would be really useful to be able to reference some
pillar in other pillar, almost mandatory in some setup. (without doing
hacky stuff)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/4326#issuecomment-371302165,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACX2Lm_YvVkM-lQMaVsEkodbntPcFLa3ks5tcFo5gaJpZM4AilyM
.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[mitar]

Still relevant.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[johnnybubonic]

seems to still be relevant

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.