salt-ssh jinja import_text breaks gitfs backend (also can't import from gitfs)

[afletch]

Using salt-ssh:
When using import_text in jinja states, referencing files stored on a gitfs remote, the file cannot be found[1]. This works fine when executed via the minion, but fails with salt-ssh.

This seems to have a cascading effect that other files stored on the gitfs, for subsequent IDs in the same state (possibly other states too) also become unavailable even for salt:// references.

For example, file.managed with source: salt://will fail to find the source file which is located on the gitfs backend. When not run subsequent to an import_text, these file.managed states work fine and pull the file from the gitfs just fine. It's as if the import_text breaks something.

In addition to this, if the import_text file is made available on a local file_root so that this loads fine, the subsequently executed IDs using file.managed and source: salt:// still fail to find the gitfs source files. Again, it appears as if once the import_text has happened, the gitfs backend is gone and the only place being checked is the local file_root[2]

[1]

    - Rendering SLS 'base:mystates.teststate' failed: Jinja error: mystates/files/myfile.txt
      Traceback (most recent call last):
        File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl
          output = template.render(**decoded_context)
        File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
          return self.environment.handle_exception(exc_info, True)
        File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "<template>", line 3, in top-level template code
        File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 135, in get_source
          raise TemplateNotFound(template)
       TemplateNotFound: mystates/files/myfile.txt

[2]

[DEBUG   ] Could not find file 'salt://mystates/files/managed_file' in saltenv 'base'
[INFO    ] Caching directory 'mystates/files/managed_file/' for environment 'base'
...
          ID: test_managed_state
    Function: file.managed
        Name: /tmp/managed_file
      Result: False
     Comment: Source file salt://mystates/files/managed_file not found
     Started: 10:52:16.657665
    Duration: 77.732 ms
     Changes:  

Example state:

test_import_state:
{% import_text "mystates/files/myfile.txt" as myfile %}
  cmd.run:
    - name: echo {{ myfile }}
test_managed_state:
  file.managed:
    - name: /tmp/managed_file
    - makedirs: True
    - dirmode: 700
    - user: root
    - group: root
    - source: salt://mystates/files/managed_file

# salt --versions-report
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.25.1
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.25.0
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-229.4.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

[Ch3LL]

Looks like i'm able to replicate this. Here is my test case:

test_import_state:
{% import_text "text.txt" as myfile %}
  cmd.run:
    - name: echo {{ myfile }}
test_managed_state:
  file.managed:
    - name: /tmp/managed_file
    - makedirs: True
    - dirmode: 700
    - user: root
    - group: root
    - source: salt://test-git.sls

And here is my gitfs:

gitfs_remotes:
  - https://github.com/Ch3LL/Ch3LLScripts.git:
    - root: salt/states

It works when running against a normal minion but when using salt-ssh it breaks it and i see that template not found error. We will need to get this fixed up thanks.

[afletch]

@Ch3LL Is there any progress on this? It's making salt-ssh unusable with gitfs backend and some months now since it was reported (indeed 2017.7 has come out since then).
Thanks.

[Ch3LL]

we do not currently have any progress of the issue as other issues are higher priority. Maybe my priority analysis of the issue was incorrect but I viewed this issue as an issue that would not bee seen by most users ( hence the P4 label) because it only did not work when including a import_text that makes a call to a gitfs file while using salt-ssh. Did i not entirely understand the use case? Do I need to change the priority?

ping @terminalmage might also have some ideas on this issue. Might be more in the salt-ssh realm, but do you see any easy solution here?

[VShell]

@Ch3LL If I recall correctly, the same bug happens with Jinja's import directive, and thus breaks a number of the salt-formula repositories (I believe I ran into the issue with nginx-formula). My workaround was to vendor the repositories within my own Salt config repo, rather than referencing them with the gitfs backend.

[Ch3LL]

Looks like you are right. This does affect just a typical import. This seems to be affected by 2016.11 releases as well so not a regression.

➜  salt git:(0931281ebd) cat /srv/salt/issues/39881.sls 


{% import 'install.sls' as test with context %}

echo test:
  cmd.run

h3ll1:
    - Rendering SLS 'base:issues.39881' failed: Jinja error: install.sls
      Traceback (most recent call last):
        File "/home/ch3ll/git/salt/salt/utils/templates.py", line 368, in render_jinja_tmpl
          output = template.render(**decoded_context)
        File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
          return self.environment.handle_exception(exc_info, True)
        File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "<template>", line 3, in top-level template code
        File "/home/ch3ll/git/salt/salt/utils/jinja.py", line 135, in get_source
          raise TemplateNotFound(template)
      TemplateNotFound: install.sls

      ; line 3

      ---


      {% import 'install.sls' as test with context %}    <======================

      echo test:
        cmd.run

      Traceback (most recent call last):
      [...]
      ---

ping @thatch45 any ideas here as to why we can't import gitfs file from a local file when using salt-ssh?

[joni1993]

I don't know my problem (https://groups.google.com/forum/#!topic/salt-users/JgE2IVB7U2I) is related to this, but for me it seems that jinja imports also don't work with gitfs if the mountpoint option is used.

It looks like the import statement in the jinja does not respect the mountpoint option and therefore tries to import from the wrong directory.

[sunthera]

Got same issue.
Salt: 2017.7.2

Looks like this problem due to dynamic file loading and Jinja filesystem loaded do not have correct context for include.
If you referencing sls file directly in top.sls then in appear in cache directory

in master:

gitfs_remotes:
   - https://github.com/saltstack-formulas/mysql-formula.git

salt/top.sls:

base:
  '*':
    - mysql

Run salt-ssh '*' state.apply test=True

Result is:

var/cache/files/base
├── mysql
│   └── init.sls
└── top.sls

Salt output:

    - Specified SLS mysql.server in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS mysql.database in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS mysql.user in saltenv base is not available on the salt master or through a configured fileserver
    - Cannot extend ID 'mysqld' in 'base:mysql'. It is not part of the high state.
      This is likely due to a missing include statement or an incorrectly typed ID.
      Ensure that a state with an ID of 'mysqld' is available
      in environment 'base' and to SLS 'mysql'

Modify top.sls

base:
  '*':
    - mysql.server
    - mysql

Run salt-ssh '*' state.apply test=True again:

var/cache/files/base
├── mysql
│   ├── init.sls
│   └── server.sls
└── top.sls

Salt:

    - Specified SLS mysql.config in saltenv base is not available on the salt master or through a configured fileserver
    - Specified SLS mysql.python in saltenv base is not available on the salt master or through a configured fileserver
    - No matching sls found for 'mysql' in env 'base'
    - Cannot extend ID 'mysql_config' in 'base:mysql.server'. It is not part of the high state.
      This is likely due to a missing include statement or an incorrectly typed ID.
      Ensure that a state with an ID of 'mysql_config' is available
      in environment 'base' and to SLS 'mysql.server'

Unfortunately, kludge with adding all required templates into top.sls is not work.

[Ch3LL]

I was able to find this issue: https://github.com/saltstack/salt/issues/9878 when re-analyzing this issue.

So the current solution is using extra_filerefs. Any files required via jinja need to be added to this option.

There is a feature request here: https://github.com/saltstack/salt/issues/19564 so once anyone here verifies this option works for them i'll close this issue in favor of this feature request so we can track there.

[afletch]

@Ch3LL This would probably work, but the fix detailed in #19564 doesn't appear to work for gitfs backends, so can't be used to solve this problem.

[joni1993]

Is there anything planned for the upcoming release regarding this problem (or #19564)?

This issue makes it impossible to use 99.9% of all public formulas in combination with the gitfs mountpoint option (as they all use jinja imports), which make the mountpoint option essentially useless.

[infoveinx]

This issue (extra_filerefs ignored on gitfs) still exists on salt 2017.7.2 (Nitrogen). I had managed several systems remotely from my master using salt-ssh earlier last year prior to moving the state and repos to gitfs. Now nothing works anymore. What's even more bizarre is that I swear I wasn't passing the extra_filerefs anywhere on the prior Salt release when I wasn't using gitfs, and it was working! Either way using gitfs doesn't work with exta_filerefs as of now. I can no longer remotely manage my systems because I import map.jinja at the top of my init.sls.

[mbochenk]

Same behavior here: cannot import in Jinja. It works on gitfs if I am using salt and salt-call commands however the same state fails in salt-ssh. extra_filerefs appears to be ignored - either in Saltfiile or as a command line option.

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: 1.3.8
         Jinja2: 2.10
        libgit2: 0.26.3
        libnacl: 1.6.1
       M2Crypto: 0.28.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.4
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: 0.4.3
         PyYAML: 3.13
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 4.4.163-1.el7.elrepo.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

[david-pulkowski]

It seems to be working for us. We call the exta_filerefs in the roster file. And point directly to our git location.

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.28.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.7.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 11 2019, 12:19:05)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.6 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.35.2.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.6 Maipo

For example:

extra_filerefs:
  - https://github.com/saltstack-formulas/openssh-formula.git
  - https://github.com/saltstack-formulas/apache-formula.git

[david-pulkowski]

Nevermind. Formulas musta been cached when we added the repos local to verify local works vs. gitfs fails. Still having the same issue.