Salt: Backslash is doubled in templates

[alexmorozov]

Having a pillar data like this:

options:
  force_cache:
    - ^.+\.(?:css|js|jpe?g|gif|ico|png)$

and a template like this:

{% for location in options.get('force_cache', []) %}
location ~* {{ location }} {
}
{% endfor %}

We get the backslash doubled in the resulting file:

  location ~* ^.+\\.(?:css|js|jpe?g|gif|ico|png)$ {
  }

How to get rid of this?
P.S.

$ salt --versions-report
           Salt: 0.15.3
         Python: 2.7.3 (default, Aug  1 2012, 05:14:39)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.4.1
         PyYAML: 3.10
          PyZMQ: 13.0.0
            ZMQ: 3.2.2

[terminalmage]

@alexmorozov The template you referenced, it was used in a file.managed state I assume?

[terminalmage]

@alexmorozov I am unable to reproduce (see below). 0.15.3 is an older release, it's possible that this _was_ a bug, and has since been fixed. 0.16.2 should be out in a couple days, and 0.16.0 is available now for you to test. I personally tested using an up-to-date git checkout, here are the results:

(saltdev)erik@virtucentos:~% cat ~/pillar/test.sls
options:
  force_cache:
    - ^.+\.(?:css|js|jpe?g|gif|ico|png)$

(saltdev)erik@virtucentos:~% cat ~/states/test.sls
/tmp/foo.txt:
  file.managed:
    - source: salt://foo.txt
    - template: jinja
(saltdev)erik@virtucentos:~% cat ~/states/foo.txt
{% for location in pillar.get('options', {}).get('force_cache', []) %}
location ~* {{ location }} {
}
{% endfor %}

(saltdev)[root@virtucentos saltdev]# salt -c etc/salt virtucentos state.sls test
virtucentos:
----------
    State: - file
    Name:      /tmp/foo.txt
    Function:  managed
        Result:    True
        Comment:   File /tmp/foo.txt updated
        Changes:   diff: New file


Summary
------------
Succeeded: 1
Failed:    0
------------
Total:     1

(saltdev)[root@virtucentos saltdev]# cat /tmp/foo.txt

location ~* ^.+\.(?:css|js|jpe?g|gif|ico|png)$ {
}

[alexmorozov]

Oh, I'm sorry, it`s my bad.

The issue was that I passed an array in the file.managed context like - options: {{ options }}, and it got escaped somehow.
Totally off-topic, but what's the recommended way to pass list/dict variables to the file`s context?

@terminalmage , thank you for the fast and detailed response!

[terminalmage]

@alexmorozov This may need to be better documented, but as you saw in the example I posted, pillar is available from a jinja template file in Salt. So, since you already have the data in pillar, you can refer to it directly, like I did:

{% for location in pillar.get('options', {}).get('force_cache', []) %}
location ~* {{ location }} {
}
{% endfor %}

Additionally, grains and any salt function are available within any jinja block, like so:

{{ grains['host'] }}

{% for ip in salt['network.ip_addrs']('em1') %}
...
<do something here>
...
{% endfor %}

[steverweber]

this is still a bug. running develop branch as of.
Fri Aug 15 13:59:57 EDT 2014
templates render vars with a backslash as double slashed.

[steverweber]

humm well if using pillar it works....

{{ salt['pillar.get']('auth:ldap:bind:login_user') }}
'MATHADTEST\mfsaltad'

{{ pillar.get('auth').ldap.bind.login_user }}
'MATHADTEST\mfsaltad'

# vars is passed in context
{{ vars.ldap.bind.login_user }}
'MATHADTEST\\mfsaltad'

it seems like the issue is my own... I'll keep digging...

workaround: {{ vars.ldap.bind.login_user.replace('\\\\','\\') }}

[steverweber]

when passing vars through context like:

autofs auto.home:
    file.managed:
        - name: /etc/auto.home
        - source: salt://auth/etc/auto.home
        - mode: 0700
        - template: jinja
        - context: { vars: {{ pillar.get('auth') }} }

the vars will be rendered with dubble slash in template.

[terminalmage]

Thanks, we'll give this another look.

[jorgenschaefer]

I seem to be seeing this issue, too. Was it intended to keep the issue closed?

salt-minion 2014.7.0+ds-2trusty1

pillar/repro.sls

vars:
  test: "foo\\bar"

pillar/top.sls

base:
  '*':
    - repro

salt/repro.sls

/tmp/foo.txt:
  file.managed:
    - source: salt://foo.txt
    - template: jinja
    - context:
        vars: {{salt["pillar.get"]("vars")}}

salt/foo.txt

test: {{vars["test"]}}

salt/top.sls

base:
  '*':
    - repro

This results in the following /tmp/foo.txt:

test: foo\\bar

It seems impossible in this context to get a foo\bar content. Removing one backslash from pillar/repro.sls causes YAML to render a \x08 character (for \b). Two backslashes result in two backslashes in the output file.

I would expect two backslashes to result in a single backslash in the output file.

[steverweber]

try:

        vars: {{salt["pillar.get"]("vars")|json}}

[jorgenschaefer]

… that's embarrassingly simple. Thank you.

[invsblduck]

@jorgenschaefer @steverweber That's not embarrassingly simple—it's embarrassingly counterintuitive. I spent the last several hours combing through SaltStack code trying to figure out why this was happening.

Long story short- it's caused by Jinja, not Salt.

You can't accurately pass lists or dictionaries to a template (via defaults or context) using {{ }}. This is because {{ }} is Jinja's "print" function and uses Python repr() underneath to supply a printable representation of the object (in addition to other magic). Don't forget it was originally intended for HTML.

This is why when @terminalmage did a pillar.get(), he got the raw data instead of the repr()'ed version of it.

For example:

>>> str = "foo\n"
>>> str
'foo\n'
>>> len(str)
4
>>> repr(str)
"'foo\\n'"

Using Salt's json Jinja filter is an interesting hack workaround. Looking at the code, it uses Jinja's Markup class to wrap the string, which, "_Marks a string as being safe for inclusion in HTML/XML output without needing to be escaped_," according to Jinja API docs.

This is an interesting solution to the problem, but also points to the fact that maybe You're Doing It Wrong if you need to template a complex data structure into the SLS as a value to a state parameter, instead of doing an {% import %} in the template (there are plenty of reasons I'm sure...).

;)

HTH.

[invsblduck]

^ grumpy much, tracking that one down in the middle of the night... hee-hee. :relieved:

[steverweber]

it was a good read... I always kinda wondered what was causing it.

[tmandry]

I have to agree with @invsblduck here.. this feels like a major hack. I ran into this problem in a (IMO) pretty well-written formula. Now I have to convince the author to use this | json trick. Is there a cleaner way to do this?

Here's how the config data is obtained in the state file. I can see why the author didn't want to duplicate it across two files...

{% set datamap = salt['grains.filter_by'](rawmap_osfam, grain='os_family', merge=salt['pillar.get']('elasticsearch:lookup')) %}

[invsblduck]

Sorry to keep commenting on a closed/old thread. @tmandry The cleaner way to do that is to import the variable into the template with Jinja and _not_ pass it as a string in the SLS (ie, not via context or defaults or other params). For example, if that formula instead used a map.jinja pattern as documented here, all the map filtering and pillar merging logic is handled in that file alone (not in the SLS) and the hash variable can be imported into all the Jinja templates and SLS files with:

{% from 'elasticsearch/map.jinja' import datamap with context %}

Some people will not prefer this approach (and it may not work in salt-ssh scenarios?), but this is how you access a complex data structure reliably—not by templating its value into the SLS with Jinja's print function {{ }}.

[ata-sql]

Looks like similar issue returned after few years...

salt --versions-report:
Salt Version:
           Salt: 3002.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr 16 2020, 01:36:27)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: centos 8 Core
         locale: UTF-8
        machine: x86_64
        release: 5.8.16-200.fc32.x86_64
         system: Linux
        version: CentOS Linux 8 Core

pillars/a.sls:

{%- import_yaml "configuration.yaml" as configuration -%}
xxx: {{ configuration.params }}

configuration.yaml:

params:
  - os-to-pg-accounts-mapping /^(.*)$ \1

result (pillar.get xxx):

local:
    - os-to-pg-accounts-mapping /^(.*)$ \\1

Backslash is double quoted.
It's just symbolic, didn't paste my complex configuration.
When putting param straight to pillar (without yaml include), work fine.