salt-ssh not providing jinja file for salt-formula

[TimJones]

When bootstrapping\setting up a VPS using salt-ssh, I use salt-formula to properly configure the minion after the bootstrap script has done its job.
When running over salt-ssh, the formula fails with the following error:

# salt-ssh new-machine state.sls salt.minion
new-machine:
----------
          ID: salt-minion
    Function: pkg.installed
      Result: True
     Comment: Package salt-minion is already installed.
     Started: 01:31:16.732499
    Duration: 424.187 ms
     Changes:
----------
          ID: salt-minion
    Function: file.recurse
        Name: /etc/salt/minion.d
      Result: False
     Comment: #### /etc/salt/minion.d/f_defaults.conf ####
              Unable to manage file: Jinja error: salt/formulas.jinja
              Traceback (most recent call last):
                File "/tmp/.root_9e7976__salt/salt/utils/templates.py", line 306, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/tmp/.root_9e7976__salt/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/tmp/.root_9e7976__salt/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 15, in top-level template code
                File "/tmp/.root_9e7976__salt/salt/utils/jinja.py", line 143, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: salt/formulas.jinja

              ; line 15

              ---
              [...]
              {{ configname }}: {{ cfg_salt[configname] }}
              {%- else -%}
              #{{ configname }}: {{ default_value }}
              {%- endif -%}
              {%- endmacro -%}
              {%- from 'salt/formulas.jinja' import file_roots, formulas with context -%}    <======================
              ##### Primary configuration settings #####
              ##########################################

              # minion includes
              {{ get_config('default_include', 'minion.d/*.conf') }}
              [...]
              ---
     Started: 01:31:17.215560
    Duration: 197.266 ms
     Changes:
----------
          ID: remove-old-minion-conf-file
    Function: file.absent
        Name: /etc/salt/minion.d/_defaults.conf
      Result: True
     Comment: File /etc/salt/minion.d/_defaults.conf is not present
     Started: 01:31:17.417099
    Duration: 0.406 ms
     Changes:
----------
          ID: salt-minion
    Function: service.running
      Result: False
     Comment: One or more requisite failed: salt.minion.salt-minion
     Started:
    Duration:
     Changes:

Summary
------------
Succeeded: 2
Failed:    2
------------
Total states run:     4

As you can see, line 15 of the salt-formula/salt/files/minion.d/f_defaults.conf includes the salt/formulas.jinja file, but looking around /tmp/.root_9e7976__salt/ shows nothing and opening /tmp/.root_9e7976__salt/salt_state.tgz shows that it never made it into the tar when being compiled.

Running the salt-formula unmodified works fine via ´salt new-machine state.sls salt.minion´, and removing the reference to salt/formulas.jinja (and all subsequent references to formulas) allows the formula to run via salt-ssh new-machine state.sls salt.minion.

[anlutro]

You need to specify included/extended jinja templates in your Saltfile with extra_filerefs:

salt-ssh:
  extra_filerefs:
    - salt://salt/formulas.jinja

[TimJones]

Thanks for that, the Saltfile documentation was pretty lacking but I guess I understand it now.

Though this to me seems very much a work-around rather than a fix. Should the compilation of the tarball not find all state dependencies to include them?

[anlutro]

It's not that easy for jinja includes/extends, apparently.

[jfindlay]

@TimJones, thanks for the report.

[TimJones]

No problem.

Just to add, if adding a Saltfile isn't feasabile for whatever reason (I'm actually doing all this from a runner script so using an instance of salt.client.ssh.client.SSHClient directly), the CLI arg would be --extra-filerefs e.g.

salt-ssh 'minion-id' state.sls xxx --extra-filerefs salt://salt/formulas.jinja

[babilen5]

This bug is a blocker for any salt-ssh usage for us. Are there nice workarounds that allow us to construct the "extra filerefs" list programmatically?

[babilen5]

Would it be an option to simply include all files in file_roots rather than pruning files that are needed?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[windowsrefund]

4+ years...

[baby-gnu]

4+ years...

Yes, to me the extra_filerefs is a workaround, not a feature. I agree with @babilen5, just make a tar of all file_roots.

If you want to optimize to avoid deploying a huge tar each time:

1. list files with their checksum on both sides
2. built a tar with files needing updates
3. deploy the tar

I tried to look at the way salt-ssh is working but it's not an easy beast.
I found several issues for this recurring problem.

I'll try to attend a salt-ssh metting, there is one tomorrow.