Salt: Include State only if it exists

[puneetk]

Hi,
I would like to be able to include an sls file only if it exists. The use case for this came up when working with Collectd saltstack formula pillar, and the way the formula is setup. What i would like to do is include the collectd formula in my top.sls file and then use the pillar to configure what states get included from the formula.

For eg the collectd/init.sls file would look like

include:
  - collectd.package
  - collectd.service
  {% for state in salt['pillar.get']('collectd:plugins:enabled') %}
  - collectd.{{state}}
  {% endfor %} 

And the corresponding pillar would be

collectd:
  FQDNLookup: true
  TypesDB: ['/usr/share/collectd/types.db']
  plugins: 
    enabled: [battery, cpu, entropy, load, memory, swap, users, write_graphite ]

Now this would require that we have states for each plugin that is enabled. Instead if i could do the following, I can now make extra configs for the plugins, if needed and include the one file in the top.sls by enabling the collectd plugin in a pillar.

include:
  - collectd.package
  - collectd.service
include_if_exists:
  {% for state in salt['pillar.get']('collectd:plugins:enabled') %}
  - collectd.{{state}}
  {% endfor %} 

Currently i would have to make two pillars and then iterate over them separately.

[basepi]

This seems like it would probably be fine. However, I don't think you'd be able to reference the included states via requisites. We can't make requisites optional, I don't think that makes sense.

[dragon788]

@basepi You are correct that the require wouldn't work in the main state, we are doing something similar with roles and looping across the files (and I'm trying to figure out a sneaky way to only do it if they exist using some jinja foo) and in order to do any requires you need to put the "require in" or "prereq in" inside the plugins that are included by the main application. This makes the plugins trigger service restarts but the service function normally without it.

[sjorge]

I ran into something this yes now I was trying the following

include:
  - host/{{ salt.grains.get('host') }}

Of course not every host has a sls file, I'd had hoped it would soft fail on those. But it bails hard.
Which makes sense in the 'require' contex, still and optional include would be nice.

Likewise for pillar's it seems, no 'require' problem with implementing a include_optional there.

[dragon788]

There is an "ignore_missing" option, but it only appears to work in certain
contexts.

On Sat, Jan 23, 2016 at 2:59 PM, Jorge Schrauwen [email protected]
wrote:

I ran into something this yes now I was trying the following

include:

• host/{{ salt.grains.get('host') }}

Of course not every host has a sls file, I'd had hoped it would soft fail
on those. But it bails hard.
Which makes sense in the 'require' contex, still and optional include
would be nice.

—
Reply to this email directly or view it on GitHub
https://github.com/saltstack/salt/issues/20010#issuecomment-174222087.

[twellspring]

It would be better to have the ignore_missing work globally, but until it does you can use a macro. I have this file

macros.jinja

{%- macro include_optional(sls_file) %}
  {%- for root in opts['pillar_roots'][saltenv] -%}
    {%- if salt['file.file_exists']('{0}/{1}/init.sls'.format(root, sls_file )  %}
    - {{ sls_file }}
    {% endif %}   
  {%- endfor %}
{%- endmacro %}

Then to optionally include something I do:

some_file.sls

{%- from "macros.jinja" import include_optional with context %}
{%- set host = salt.grains.get('host') %}

include:
    - existing.sls
    {{ include_optional(  "host/{0}".format(host )  ) }}

[tampakrap]

I tried to use the same macro for the /srv/salt/top.sls by using opts['file_roots'][saltenv] but it doesn't seem to work (ignore_missing doesn't work either). Any idea why, or any idea if there is another way to do it?

[MAHDTech]

@tampakrap, I believe it doesn't work with the state files as they are sync'd to the minion before being processed locally, so the actual file path to the state is something like this on the Minion.

{{ cachedir }}{{ saltenv }}/PATH_TO_SLS_FILE
/var/cache/salt/files/base/PATH_TO_SLS_FILE

So for an example if you had these states

/srv/salt/Roles/StateFiles/Deeper.sls
/srv/salt/Roles/StateFiles/init.sls

Which are referenced like:

• Roles.StateFiles
• Roles.StateFiles.Deeper

Maybe an adaption on this could work... (haven't tested, excuse the ugly jinja)

in your top.sls

{{ include_state("Roles", "StateFiles") }}
{{ include_state("Roles", "StateFiles.Deeper") }}

The Jinja macro

{#- A Jinja macro to see if the file exists before attempting to apply the state -#}
{%- macro include_state(SLS_FOLDER, SLS_FILE) -%}

  {# Set the path where the state files are located #}
  {%- set SLS_ROOT = opts['cachedir'] + "/files/" + saltenv -%}

  {# Replace any dots in the file name with forward slashes so sub-states work #}
  {%- set SLS_FILE = SLS_FILE | replace(".","/") -%}

  {%- set SLS_PATH_INIT = "{0}/{1}/{2}/init.sls".format(SLS_ROOT, SLS_FOLDER, SLS_FILE) -%}
  {%- set SLS_PATH = "{0}/{1}/{2}.sls".format(SLS_ROOT, SLS_FOLDER, SLS_FILE) -%}

  {# If the actual State file is init.sls #}
  {% if salt['file.file_exists'](SLS_PATH_INIT) -%}

    {%- do salt.log.info("Applying State file: " + SLS_PATH_INIT) %}
      - {{ SLS_FOLDER }}.{{ SLS_FILE }}

  {# If the State file is named something else #}
  {%- elif salt['file.file_exists'](SLS_PATH) -%}

    {%- do salt.log.info("Applying State file: " + SLS_PATH) %}
    - {{ SLS_FOLDER }}.{{ SLS_FILE }}

  {%- else -%}

    {%- do salt.log.error("State file does not exist: " + SLS_FILE) -%}

  {%- endif -%}

{%- endmacro -%}

EDIT: I got around to testing this and it was never going to work, as the file is only copied to the minion after a successful run, doh.

[gczuczy]

I think this perfectly makes sense in every scenario where the top sls is automatically generated. There are parts of an infrastructure that triggers generation into the top sls but might not have a salt state actually for it. The plugin example above was a very good one.

Personally I could also use such a feature.

[mchugh19]

See #45730 for our quick fix to this problem.

Allows for:

# cat top.sls
base:
  '*':
    - common

  {% set roles = salt['grains.get']('roles',[]) -%}
  {% for role in roles -%}
  {% if salt['state.exists']('roles.{0}'.format(role)) -%}
  'roles:{{ role }}':
    - match: grain
    - roles.{{ role }}
  {% endif -%}
  {% endfor -%}

# salt-call state.show_top
[ERROR   ] Template was specified incorrectly: False
local:
    ----------
    base:
        - common
# touch roles/saltmaster.sls
# salt-call state.show_top
local:
    ----------
    base:
        - common
        - roles.saltmaster

[basepi]

@gtmanfred Pinging you so a current salt employee is on this thread.

[OrangeDog]

The state module now (2019.2.0) has sls_exists and id_exists. That resolves this issue?

[cr1st1p]

Note: I also wanted today to use something on the line of 'state exists' - for role based setup.
Based on last comment, I tried sls_exists.
It works fine. Until... I observed that for sls files that are having issues (syntax ones), it returns false,
so instead of getting the file included and getting an error at execution time, I get some warning (even though it says "Critical" and file is "happily" skipped.

root@salt-master-dev:~# salt-call --local state.sls_exists roles.salt-master
local:
    True
root@salt-master-dev:~# salt-call --local state.sls_exists home.roles.computer
[CRITICAL] Could not render SLS home.roles.computer. Syntax error detected.
local:
    False

Running salt-call -l debug --local state.highstate
ends without any error, and without the home.roles.computer state file loaded :-(
I get this output :

....
[CRITICAL] Could not render SLS home.roles.computer. Syntax error detected.
....
Summary for local
-------------
Succeeded: 18
Failed:     0
-------------
Total states run:     18
Total run time:    4.818 s

[cr1st1p]

Opened #55072

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[johnnybubonic]

Please either keep open until #55072 is resolved or mark as related; thanks

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[johnnybubonic]

...Why am I thanking a bot? -_-