Salt: [question] Optional pillar include. Handle error or check file existence

[ngortheone]

I have a complex infrastructure with multiple projects, multiple regions and environments. I need to include pillars based on grains for each host. But some files can be not present (if there are no variables at that level). There must be a way to do something like ansible's ignore_errors: True on include or to check file existence prior to adding it to the list. Here are examples:

pillar structure

├── env-setup.sls
├── projectA
│   ├── main.sls
│   └── regionX
│       ├── envY
│       │   └── main.sls
│       ├── main.sls
│       └── envZ
└── top.sls

top.sls

base:
  '*':
    - env-setup

env-setup.sls

{% set hierarchy  = [
    salt['grains.get']('project'),
    salt['grains.get']('region'),
    salt['grains.get']('env')
  ]
%}

include:
{% for level in hierarchy %}
  {% set  path = [] %}
  {% for x in range(hierarchy.index(level), -1, -1) %}
    {% do path.insert(0, hierarchy[x] + "/") %}
  {% endfor %}
  {% do path.append("main") %}
  - {{ path | join }}
{% endfor %}

When included pillar file is not present - salt fails:

Specified SLS 'projectA/regionX/envZ/main' in environment 'base' is not available on the salt master

Is there a way either to ignore error on include or to check file presence in jinja code before adding it to the list?

Versions Report

Salt demo environment
Salt Version:
Salt: 2015.8.8.2

[ngortheone]

The solution provided here:
http://grokbase.com/t/gg/salt-users/148m57601n/pillars-conditionally-add-pillar-file-in-top-sls

{% if salt['file.file_exists')('hosts/{0}.sls'.format(grains.host)) %}
'{{ id }}':
   - hosts.{{ grains.host }}
{% endif %}

does not work as file_exists requires absolute path

[jfindlay]

@ngortheone, file.file_exists should work as it uses builtin python path utils. The difficulty with relative paths is to know what they are relative to. In my testing, they seem to be relative to / when running with the minion daemon and to $CWD when run with salt-call.

# git diff
diff --git a/salt/modules/file.py b/salt/modules/file.py
index da7308f..3d2df65 100644
--- a/salt/modules/file.py
+++ b/salt/modules/file.py
@@ -3141,6 +3141,7 @@ def file_exists(path):
         salt '*' file.file_exists /etc/passwd

     '''
+    print(os.getcwd())
     return os.path.isfile(os.path.expanduser(path))

# cat /srv/salt/test.sls 
{% set found = True if salt['file.file_exists']('file') else False %}

test:
  cmd.run:
    - name: echo {{ found }}

[ngortheone]

@jfindlay. Setting absolute path seems to work.

    {% if salt['file.file_exists']('/srv/pillar/' + path|join + '.sls') %}
  - {{ path|join }}
    {% endif %}

But this is hardcode. Is there a proper way to get pillar_roots path and current pillarenv in jinja?

[jfindlay]

@ngortheone, you can try salt['config.get']('pillar_roots'), which is a python dictionary, so you'll have to take care to iterate through it. The disadvantage to this approach is that jinja logic is executed by the minion, so will not necessarily have access to the pillar opts unless you add it.