Salt: forumla's map.jinja not found in jinja-template-files on salt-ssh

[sgeisbacher]

Expected Behavior

the forumla's file-generation by jinja-template should load map.jinja also on salt-ssh.
this formula works with salt minion state.highstate like a charm, but not with salt-ssh minion state.highstate

Actual Behavior

when performing a state.highstate via salt-ssh i get the following error-msg: (see bottom)

Steps to Reproduce Issue

1. create a simple formula with map.jinja
2. add file generation by jinja (file.managed: - source: salt://.../file.jinja - template: jinja) to this formula
3. import the map.jinja to this file.jinja-template with {% from ... import ... %}
4. try to provision this formula to a minion with salt-ssh --> you will get the error from above
   

   Versions Report

Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: 0.9.1
         PyYAML: 3.10
          PyZMQ: 14.0.1
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-71-generic
         system: Ubuntu 14.04 trusty

Relevant Logs, Configs, or States

Error-Message: `Unable to manage file: Jinja error: jinjatest/map.jinja
              Traceback (most recent call last):
                File "/tmp/.root_09032a_salt/salt/utils/templates.py", line 366, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/tmp/.root_09032a_salt/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/tmp/.root_09032a_salt/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 1, in top-level template code
                File "/tmp/.root_09032a_salt/salt/utils/jinja.py", line 144, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: jinjatest/map.jinja

              ; line 1

              ---
              {% from "jinjatest/map.jinja" import jinjatest with context %}    <======================

              hello {{ jinjatest.name }}
              you are {{ jinjatest.age }} old
              Traceback (most recent call last):
                File "/tmp/.root_09032a_salt/salt/utils/templates.py", line 366, in render_jinja_tmpl
              [...]`

here is the simple-test-formula:
salt-jinjatest.zip

[jfindlay]

@sgeisbacher, thanks for reporting.

[JubbaSmail]

@jfindlay can I work on this issue ?

[edgan]

The know workaround is using extra_filerefs. The salt-ssh code doesn't pick up files that aren't referenced by include or source. The import doesn't count. I have tried a system of having a state to source the map.jinja in cases of salt-ssh, but it proved to be more work than using the extra_filerefs method. See #9878 for history.

Saltfile:
extra_filerefs:
  - salt://jingatest/map.jinja

[rallytime]

@Ismail-AlJubbah Certainly!

[awerner]

Another possible workaroung is to use the import statement in the init.sls and then pass the jinjatest variable as context to the file.managed state. This is also more clean IMO.
@Ismail-AlJubbah any progress?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[publicarray]

Any update?

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[awerner]

If using the context does not work for you, I have another workaround.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[max-arnold]

Bump

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[max-arnold]

I made an ugly patch that seemingly fixes this issue by packing the whole salt:// directory into the salt-ssh state tarball:

diff --git a/salt/fileclient.py b/salt/fileclient.py
index d606bea99e..64ae1b0a63 100644
--- a/salt/fileclient.py
+++ b/salt/fileclient.py
@@ -238,7 +238,7 @@ class Client(object):
         # the target directory and caching them
         for fn_ in self.file_list(saltenv):
             fn_ = salt.utils.data.decode(fn_)
-            if fn_.strip() and fn_.startswith(path):
+            if fn_.strip() and (fn_.startswith(path) or path == '/'):
                 if salt.utils.stringutils.check_include_exclude(
                     fn_, include_pat, exclude_pat
                 ):

To activate it add --extra-filerefs 'salt://' command line option to salt-ssh. Also it is possible to do the same through /etc/salt/master config snippet:

ssh_minion_opts:
  extra_filerefs: salt://

[afletch]

This was not an issue in salt-2019.2.0-1 but since updating to salt-3000.2-1 it has become so.

[edgan]

diff --git a/salt/fileclient.py b/salt/fileclient.py
index d606bea99e..64ae1b0a63 100644
--- a/salt/fileclient.py
+++ b/salt/fileclient.py
@@ -238,7 +238,7 @@ class Client(object):
         # the target directory and caching them
         for fn_ in self.file_list(saltenv):
             fn_ = salt.utils.data.decode(fn_)
-            if fn_.strip() and fn_.startswith(path):
+            if fn_.strip() and (fn_.startswith(path) or path == '/'):
                 if salt.utils.stringutils.check_include_exclude(
                     fn_, include_pat, exclude_pat
                 ):

I tested @max-arnold 's patch. It works on salt-ssh 3001. It let me remove 78 lines from my Saltfile. I would love to see this become part of official Salt via a PR.

[louwers]

Doesn't work for me, and I feel that this should be the default behavior (e.g. Salt SSH should not require this tweak). If it's included it should be documented as well.

[max-arnold]

@louwers I'm puzzled why the patch doesn't work for you. Can you try the following salt-ssh command:

salt-ssh HOSTNAME -t -W -w --extra-filerefs 'salt://' state.apply YOURSTATE

And I haven't tried it with Gitfs.

Submitting a PR is still on my todo list (it requires salt-ssh integration tests which I'm unfamiliar with).

[max-arnold]

I'm hesitant to enable this behaviour by default, because it transfers the whole state tree which is slower. And implementing the proper solution (only transfer what is needed) is beyond my abilities and knowledge of salt-ssh internals.

@Ch3LL What do you think about including the above patch in Salt? Also, are there any viable approaches to implement a proper solution (that will be able to find the map.jinja files referenced from templates)? Is it possible to do that via salt-ssh wrappers, or it requires rearchitecting the way salt-ssh does file transfer (pull instead of push)?

[daks]

I tested @max-arnold patch and it works fine. I can put

salt-ssh:
  extra_filerefs:
    - salt://

and no more problem with missing files (map.jinja, defaults.yaml...). I understand that it may not be desirable to sync all salt:// files by default, but, with enough warnings, it could be possible to let people do it.

[foudfou]

extra_filerefs workaround doesn't seem to work for me in salt-3002 :sob:

See also #58762

[daks]

extra_filerefs workaround doesn't seem to work for me in salt-3002 sob

See also #58762

Broken for me too on 3002.1, works fine on 3001.3

[alxwr]

@foudfou @daks same here for me on FreeBSD 12.1, salt 3002

[alxwr]

@foudfou @daks I found a workaround. This patch adds just the bit of information required for salt-ssh to use extra_filerefs again. I made sure not to alter data structures which are used somewhere later in the code. (Consider this to be a hacky solution from an uninformed person though.)

From 052b72c0bf8040493f714b54775a50edd8d1062d Mon Sep 17 00:00:00 2001
Date: Wed, 11 Nov 2020 12:59:16 +0100
Subject: [PATCH] add extra_filerefs to SSH high state

---
 salt/client/ssh/wrapper/state.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/salt/client/ssh/wrapper/state.py b/salt/client/ssh/wrapper/state.py
index bbe8385ae0..ccf31d6bbf 100644
--- a/salt/client/ssh/wrapper/state.py
+++ b/salt/client/ssh/wrapper/state.py
@@ -5,6 +5,7 @@ Create ssh executor system
 import logging
 import os
 import time
+from copy import deepcopy

 import salt.client.ssh.shell
 import salt.client.ssh.state
@@ -656,15 +657,20 @@ def highstate(test=None, **kwargs):
     st_kwargs = __salt__.kwargs
     __opts__["grains"] = __grains__
     opts = salt.utils.state.get_sls_opts(__opts__, **kwargs)
+    highstate_opts = deepcopy(opts)
+    highstate_opts["extra_filerefs"] = opts["__master_opts__"].get("extra_filerefs", "")
+    highstate_opts["cachedir"] = opts["__master_opts__"].get("cachedir", "")
     st_ = salt.client.ssh.state.SSHHighState(
-        opts, __pillar__, __salt__, __context__["fileclient"]
+        highstate_opts, __pillar__, __salt__, __context__["fileclient"]
     )
     st_.push_active()
     chunks = st_.compile_low_chunks()
     file_refs = salt.client.ssh.state.lowstate_file_refs(
         chunks,
         _merge_extra_filerefs(
-            kwargs.get("extra_filerefs", ""), opts.get("extra_filerefs", "")
+            kwargs.get("extra_filerefs", ""),
+            opts.get("extra_filerefs", ""),
+            opts.get("__master_opts__", {}).get("extra_filerefs", "")
         ),
     )
     # Check for errors
-- 
2.25.1

Please test with

salt-ssh HOSTNAME -t -W -w state.apply YOURSTATE test=True