Salt: TemplateNotFound: salt/formulas.jinja when running salt-ssh
I used salt-formula to install salt-minion on other ubuntu 14.04 servers.
master file_roots config:
file_roots:
base:
- /srv/salt
- /data/salstack_forumals/salt-formula
version:
# salt-ssh --version
salt-ssh 2014.7.1 (Helium)
command:
salt-ssh 'es4log*' state.sls salt.minion
error:
----------
ID: salt-minion
Function: pkg.installed
Result: True
Comment: Package salt-minion is already installed.
Started: 16:02:52.474110
Duration: 1210.034 ms
Changes:
----------
ID: salt-minion
Function: file.recurse
Name: /etc/salt/minion.d
Result: False
Comment: #### /etc/salt/minion.d/_defaults.conf ####
Unable to manage file: Jinja error: salt/formulas.jinja
Traceback (most recent call last):
File "/tmp/.root_76a713__salt/salt/utils/templates.py", line 286, in render_jinja_tmpl
output = template.render(**unicode_context)
File "/tmp/.root_76a713__salt/jinja2/environment.py", line 969, in render
return self.environment.handle_exception(exc_info, True)
File "/tmp/.root_76a713__salt/jinja2/environment.py", line 742, in handle_exception
reraise(exc_type, exc_value, tb)
File "<template>", line 15, in top-level template code
File "/tmp/.root_76a713__salt/salt/utils/jinja.py", line 132, in get_source
raise TemplateNotFound(template)
TemplateNotFound: salt/formulas.jinja
; line 15
---
[...]
{{ configname }}: {{ cfg_salt[configname] }}
{%- else -%}
#{{ configname }}: {{ default_value }}
{%- endif -%}
{%- endmacro -%}
{%- from 'salt/formulas.jinja' import file_roots, formulas with context -%} <======================
##### Primary configuration settings #####
##########################################
# Per default the minion will automatically include all config files
# from minion.d/*.conf (minion.d is a directory in the same directory
[...]
---
Started: 16:02:53.684336
Duration: 108.172 ms
Changes:
----------
ID: salt-minion
Function: service.running
Result: False
Comment: One or more requisite failed: {'salt.minion.salt-minion': '#### /etc/salt/minion.d/_defaults.conf ####\nUnable to manage file: Jinja error: salt/formulas.jinja\nTraceback (most recent call last):\n File "/tmp/.root_76a713__salt/salt/utils/templates.py", line 286, in render_jinja_tmpl\n output = template.render(**unicode_context)\n File "/tmp/.root_76a713__salt/jinja2/environment.py", line 969, in render\n return self.environment.handle_exception(exc_info, True)\n File "/tmp/.root_76a713__salt/jinja2/environment.py", line 742, in handle_exception\n reraise(exc_type, exc_value, tb)\n File "<template>", line 15, in top-level template code\n File "/tmp/.root_76a713__salt/salt/utils/jinja.py", line 132, in get_source\n raise TemplateNotFound(template)\nTemplateNotFound: salt/formulas.jinja\n\n; line 15\n\n---\n[...]\n{{ configname }}: {{ cfg_salt[configname] }}\n{%- else -%}\n#{{ configname }}: {{ default_value }}\n{%- endif -%}\n{%- endmacro -%}\n{%- from \'salt/formulas.jinja\' import file_roots, formulas with context -%} <======================\n##### Primary configuration settings #####\n##########################################\n\n# Per default the minion will automatically include all config files\n# from minion.d/*.conf (minion.d is a directory in the same directory\n[...]\n---'}
Started:
Duration:
Changes:
Summary
------------
Succeeded: 1
Failed: 2
------------
Total states run: 3
LYY
All 39 comments
@LYY Have you been able to get this working via a regular salt command (instead of using salt-ssh)? I suspect this might be a problem with the formula itself.
rallytime
on 9 Mar 2015
@rallytime I'm a beginner of salt user, I really don't know how to use this formula via a regular salt command. If you can give me some help, I think I can give more info.
LYY
on 10 Mar 2015
@LYY Welcome! Now that I think about it, I am not sure if I was asking a useful question. At a glance from the info you've provided here, it looks like you have everything set up correctly.
ping @nmadhok - I haven't used this formula before. Have you seen this particular error? I can try and reproduce this tomorrow.
rallytime
on 10 Mar 2015
@rallytime I haven't come across this error in tests before. Seeing this for the first time. I will have to test this using salt-ssh since I've never tested this formula using salt-ssh
Also, salt/formulas.jinja is a part of the salt-formula. Look here: https://github.com/saltstack-formulas/salt-formula/blob/master/salt/formulas.jinja
It's probably not able to find the path to the jinja template. @LYY make sure that you have cloned the salt-formula and have specified the correct path to the jinja template in your state file.
BTW, you can use the bootstrap script to install salt-minion on the servers using salt-ssh.
nmadhok
on 10 Mar 2015
@nmadhok Thank you very much!
I cloned salt-formula to /data/salstack_forumals/salt-formula, and setted the path to /etc/salt/master with file_roots config. Is there any path setting for use?
LYY
on 10 Mar 2015
@rallytime We're getting the same thing over here: https://github.com/saltstack-formulas/mysql-formula/issues/92 where @ogabrielsantos has completed some additional testing for us. If you want to take a look at that there is additional information that could help resolve this.
gravyboat
on 28 Sep 2015
Thanks for the updates everyone. I've edited the labels so that we'll hopefully get some more eyes on this one.
@basepi FYI.
rallytime
on 28 Sep 2015
Any files required by jinja templating must be synced down as extra_filerefs, which is just a list of salt:// paths that should be synced with the rest of the state data. Salt tries to find many of these references, but it cannot find jinja references. You can pass in extra_filerefs as a kwarg to your state run, or just set it in the master config and have those extra files synced down with every state run.
basepi
on 28 Sep 2015
Is this supposed to work without "extra_filerefs" only in certain scenarios?
ogabrielsantos
on 28 Sep 2015
Basically, the only refs we find are references to salt://, I can't remember off the top of my head how we're parsing those out. So if your file.managed has a source argument such as salt://myfile.txt then it should be picked up.
basepi
on 29 Sep 2015
@basepi I think this would be something worth investigating as files that end in .yaml clearly aren't being picked up. @ogabrielsantos was able to get this working via adding the following to his /etc/salt/master file for salt-ssh:
extra_filerefs:
- salt://mysql/defaults.yaml
- salt://mysql/supported_sections.yaml
Seems both unnecessary and confusing to have to add that especially for people who may just be testing out Salt via salt-ssh and a formula.
gravyboat
on 30 Sep 2015
I can't reproduce the same issue with masterless / master-slave operation. Currently only salt-ssh has had this extra_filerefs requirement.
Any files required by jinja templating must be synced down as extra_filerefs,
ogabrielsantos
on 30 Sep 2015
Right. This is not a problem with normal salt, because minions can request new files at any time. With salt-ssh, however, we have to anticipate all files that will be needed _before_ sending those files to the minion to be evaluated.
It has nothing to do with file extension. Rather, the master does a best effort job of finding file references to send down so the minion will have a complete picture when it runs the states.
That said, if I remember correctly, states are compiled down to low data master-side for salt-ssh, so any references in the jinja should be incorporated before the files are sent down to the minion. The only time you should have to use the extra filerefs is for templates and the like, unless I'm misremembering.
basepi
on 30 Sep 2015
(I know the above conflicts with what I said before, I'm trying to resolve that discrepancy now, going through the code)
basepi
on 30 Sep 2015
Awesome thanks for taking a look!
gravyboat
on 30 Sep 2015
Came across this whilst having the same issue. For clarity, if you're encountering this issue you should run salt-ssh like so:
salt-ssh '*' state.sls salt.minion extra_filerefs=salt://salt/formulas.jinja
Xiol
on 25 Jan 2016
Hello,
Is there any progress on this issue?
@Xiol workaround works, but its not practical o use if you have many formulas that use default values like myformula/map.jinja.
alfredopalhares
on 16 Mar 2016
You can also set extra_filerefs in your master config so you don't have to pass them on the command line. Those files will be sent with all state runs.
basepi
on 16 Mar 2016
I really need this fixed.
extra_filerefs is a bad workaround. You have to remember to add a new map.jinja to the list every time you add a new one.
My solution is to include the map.jinja as a file sent to /dev/null, and I only include it for salt-ssh. I set the is_ssh grain in the roster per host. But this solution is getting out of hand. I am heavily using salt-ssh for deploys.
nginx/init.sls:
{% if salt['grains.get']('is_ssh') >= 1 %}
- nginx.map
{% endif %}
nginx/map.sls:
nginx_map_jinja:
file.managed:
- name: /dev/null
- source: salt://nginx/map.jinja
- replace: False
edgan
on 17 Jun 2016
I'm finding that this works if I use extra_filerefs= on the command line, but adding extra_filerefs to my master config has no effect.
beverlycodes
on 18 Aug 2016
It goes in Saltfile not master. salt-ssh always looks for a Saltfile in the current directory. So if you run salt-ssh from /foo, then you want to have a /foo/Saltfile.
Saltfile:
salt-ssh:
config_dir: .
max_procs: 1
ssh_wipe: True
extra_filerefs:
- salt://accounts/map.jinja
- salt://apache2/map.jinja
edgan
on 18 Aug 2016
@edgan Ah, okay. I'll give that a try.
beverlycodes
on 18 Aug 2016
Just trying salt-ssh out and ran into this. I was able to get it to work using extra_filerefs but I can see this getting out of hand quickly. Any idea if or when salt-ssh will be able to handle this by itself? Thanks.
chandy
on 30 Apr 2017
I have came through the exact same issue. there is a way i followed before in a similar case file.managed but the normal minion (there was not any issues i just needed to pass a variable to the jinja template). here is the solution for similar cases according to the documentation
If using a template, any user-defined template variables in the file defined in source must be passed in using the defaults and/or context arguments. The general best practice is to place default values in defaults, with conditional overrides going into context
So All you have to do is passing the variables from the states ( which will be able to call the map.jinja normaly) using context parameter. I have solved this issue on salt-ssh using the exact same way when I needed to pass a variable from map.jinja to a managed file
Also I found the same suggestion in this issue #31531
mostafahussein
on 5 Jun 2017
I just had this issue with the exact same issue as the reason this issue was reported: tried salt via salt-ssh, missed formulas.jinja and failed. Strange to not see this fixed yet.
johnkeates
on 12 Sep 2017
+1 with extra_filesfs worked
dreampuf
on 13 Oct 2017
Another workaround, that can be triggered from a sls-file and can be used without impact such as the /dev/null approach is to use file.cached :
cache_map_jinja:
file:
- cached
- name: salt://accounts/map.jinja
- require_in:
- file: file_that_needs_map
file_that_needs_map:
file:
- managed
- ...
awerner
on 20 Sep 2018
Does any one know if Saltfile is ignored when calling salt-ssh over salt-api?
ClaudiuLSH
on 26 Oct 2018
I am seeing this error when running a formula over salt-ssh
when I run the same formula with salt agent, no issues. It appears a nested Jinja cant be read by salt-ssh
root@nemesis /s/s/s/s/f/n/f/hosts# salt --versions-report
Salt Version:
Salt: 2018.3.2
Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.30.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Jul 13 2018, 13:06:57)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-514.16.1.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core
Heres the formula, its a simple /etc/hosts formula that has a hosts_file.j2 which includes a shared template
root@nemesis# cat /srv/saltstack/salt/state/formula/network/files/hosts/aws_usa.j2
###########################################
## this file is managed by Saltstack
## do not modify this file manually!!!
##
## AWS - N. Virginia
###########################################
127.0.0.1 {{ grains.id }}.company.com {{ grains.id }}
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# AWS
{% include 'formula/network/files/hosts/aws_shared.j2' %}
# NY4 datacenter
123.12...etc
root@nemesis# cat /srv/saltstack/salt/state/formula/network/files/hosts/aws_shared.j2
34.xxx github xxxx.compute-1.amazonaws.com
172.xxx tableau xxxx.compute-1.amazonaws.com
172.xxxx jenkins xxx.compute-1.amazonaws.com
running this with salt-ssh causes TemplateNotFound
Name: /root/.bashrc - Function: file.managed - Result: Clean Started: - 11:05:29.066854 Duration: 58.499 ms
Name: /root/.vimrc - Function: file.managed - Result: Clean Started: - 11:05:29.125597 Duration: 12.175 ms
----------
ID: aws_usa
Function: file.managed
Name: /etc/hosts
Result: False
Comment: Unable to manage file: Jinja error: formula/network/files/hosts/aws_shared.j2
Traceback (most recent call last):
File "/var/tmp/.centos_9e4306_salt/py2/salt/utils/templates.py", line 389, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/var/tmp/.centos_9e4306_salt/py2/jinja2/environment.py", line 969, in render
return self.environment.handle_exception(exc_info, True)
File "/var/tmp/.centos_9e4306_salt/py2/jinja2/environment.py", line 742, in handle_exception
reraise(exc_type, exc_value, tb)
File "<template>", line 14, in top-level template code
File "/var/tmp/.centos_9e4306_salt/py2/salt/utils/jinja.py", line 140, in get_source
raise TemplateNotFound(template)
TemplateNotFound: formula/network/files/hosts/aws_shared.j2
; line 14
---
[...]
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# AWS
{% include 'formula/network/files/hosts/aws_shared.j2' %} <======================
# NY4 datacenter
[...]
---
Started: 11:05:29.137979
Duration: 33.545 ms
Changes:
running same exact formula via salt agent, no issues
Name: /etc/hosts - Function: file.managed - Result: Clean Started: - 11:15:38.569152 Duration: 115.477 ms
perfecto25
on 20 Dec 2018
Is this issue still relevant? With salt-2018.3.4 the following state works for me:
example/init.sls:
{% from "example/map.jinja" import example with context %}
Install the package:
pkg.installed:
- name: {{ example.pkg }}
example/map.jinja:
{% set example = salt['grains.filter_by']({
'Debian': {'pkg': 'mtr'},
}, merge=salt['pillar.get']('example:lookup')) %}
% salt-ssh -W -w -t HOSTNAME state.apply example
HOSTNAME:
Summary for HOSTNAME
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
Total run time: 781.308 ms
max-arnold
on 2 Jul 2019
Still happens with 2019.2.0, I'm trying to use docker formula and immediately get
localhost:
- Rendering SLS 'base:docker' failed: Jinja error: docker/map.jinja
Traceback (most recent call last):
File "/home/non7top/salt/lib/python3.6/site-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/home/non7top/salt/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/home/non7top/salt/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/home/non7top/salt/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/non7top/salt/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/home/non7top/salt/lib/python3.6/site-packages/salt/utils/jinja.py", line 171, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: docker/map.jinja
; line 1
---
{% from "docker/map.jinja" import docker with context %} <======================
{%- set docker_pkg_name = docker.pkg.old_name if docker.use_old_repo else docker.pkg.name %}
{%- set docker_pkg_version = docker.version | default(docker.pkg.version) %}
{%- set docker_packages = docker.kernel.pkgs + docker.pkgs %}
[...]
non7top
on 2 Jul 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
![stale[bot] picture](https://avatars.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 8 Jan 2020
Thank you for updating this issue. It is no longer marked as stale.
stale[bot]
on 8 Jan 2020
Hello.
This is a long standing issue without proper fix, the --extra-filerefs workaround is really not useful when you have lots of YAML/jinja files (it's not working for me and others.
So, instead of parsing files to look for possible include or import, which is broken for computer file paths, when building the archive to send to the minion, salt-ssh could:
- list all files in the file server for the environment with checksums and send that to the minion
- the minion list all files in its cache with checksums, make a diff, remove old files from the cache and report the wanted files to
salt-ssh salt-sshmake an archive with the wanted filessalt-sshsend the archive to the minion
If extra-filerefs would works for me, I'll finish by putting everything in it.
baby-gnu
on 28 Apr 2020
@baby-gnu Could you try the following one-line patch? https://github.com/saltstack/salt/issues/31531#issuecomment-615253644
It packs the whole salt:// file tree and requires you to specify just one --extra-filerefs arg.
max-arnold
on 28 Apr 2020
I ran into this with the postgresql-formula with salt-ssh 3001rc1.
louwers
on 25 Aug 2020
@louwers Recent versions of the postgres-formula do work with salt-ssh -- would you mind opening an issue in the formula's repo so we can figure out what's going on?
myii
on 26 Aug 2020
I will @myii!
louwers
on 26 Aug 2020
The extra_filerefs workaround does not work with gitfs.
If you're not succesful in using it and you are using gitfs, try copying the specified files to your fileroot as a temporary fix, and then defining that location as your kwarg.
jul-reb
on 12 Oct 2020
Related issues
sfozz
路
3Comments
allyunion
路
3Comments
layer3switch
路
3Comments
udf2457
路
3Comments
mooperd
路
3Comments
Most helpful comment
Still happens with 2019.2.0, I'm trying to use docker formula and immediately get