Salt: Newlines may be rendered as literal \n for multi-line scalar variables in some circumstances

[kartsm]

Template file bar.jinja:

{% for k, v in site.quux.items() %}
{{ k }}
{{ v.garply }}
{% endfor %}

State file bar.sls:

/tmp/bar:
  file.managed:
    - user: root
    - group: root
    - mode: 644
    - source: salt://bar.jinja
    - template: jinja
    - context:
        site:
          quux:
            blurfl:
              garply: |
                corge
                wibble
                wobble

State file baz.sls:

{% import_yaml "baz.yaml" as site %}
/tmp/baz:
  file.managed:
    - user: root
    - group: root
    - mode: 644
    - source: salt://bar.jinja
    - template: jinja
    - context:
        site: {{ site }}

And imitation pillar data baz.yaml:

quux:
  blurfl:
    garply: |
      corge
      wibble
      wobble

So let's run these two states:

[root@l-tsmsalt-101 states]# salt '*salt*' state.sls bar
[root@l-tsmsalt-101 states]# salt '*salt*' state.sls baz

And compare output.

[root@l-tsmsalt-101 states]# diff -u /tmp/bar /tmp/baz
--- /tmp/bar    2016-01-27 23:16:01.839356994 +0000
+++ /tmp/baz    2016-01-27 23:16:10.282133493 +0000
@@ -1,7 +1,4 @@

 blurfl
-corge
-wibble
-wobble
-
+corge\nwibble\nwobble

It looks like it makes a rather large difference how the variable site is passed into the template. Since import_yaml is a Salt thing, and not a Jinja thing, I'm wondering if the newlines are getting mangled there.

I'm trying to take advantage of the public apache-formula. It encourages you to store information about all your sites in the pillar (thus "imitation pillar data" above). It allows arbitrary bits of config to be passed as multi-line scalars, for when the provided templates don't do what you want. Look for Formula_Append. I'm not interested in Salting every possible bit of Apache configuration syntax when I only need to use this feature occasionally. Only 3 sites out of several dozen need it, and only for about 4 short lines at a time.

About the machine where my tests were run:

[root@l-tsmsalt-101 states]# uname -a
Linux l-tsmsalt-101.bhyve.local 2.6.32-573.7.1.el6.x86_64 #1 SMP Tue Sep 22 22:00:00 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@l-tsmsalt-101 states]# cat /etc/centos-release
CentOS release 6.7 (Final)
[root@l-tsmsalt-101 states]# cat /etc/system-release-cpe
cpe:/o:centos:linux:6:GA
[root@l-tsmsalt-101 states]# salt --version
salt 2015.8.3 (Beryllium)
[root@l-tsmsalt-101 states]# python --version
Python 2.6.6

Salt was installed via salt-bootstrap on 19 January 2016, if that helps any.

[jfindlay]

@kartsm, thanks for the report.

[stefansundin]

I'm seeing this issue in 2014.7.0rc2 as well.

[JonathanThorpe]

Still appears to be happening with 2016.3.3+ds-1 and Jinja 2.7.3-1 .

Gist of how I reproduce: https://gist.github.com/JonathanThorpe/c5b5ad5853c3465e1bbc9765030a4b5d

[ppmathis]

Experiencing the same issue with Salt 2016.3.3 as @JonathanThorpe
A temporary, but ugly workaround: {{ variable|replace('\\n', '\n') }}

[Armadill0]

Still present in 2016.11.

@ppmathis Thanks for the workaround.

[feinu]

Still present in 2017.7.0.

An alternative workaround is to use {%- set foo = salt.pillar.get('foo') -%} in the template itself. It's still ugly as it's not using context, but at least it won't accidentally mangle anything.

[hkbakke]

To avoid messing with the template I do this instead:

- context:
    # Pipe via json to avoid issue where newlines are mangled to literal \n
    # for multiline strings (https://github.com/saltstack/salt/issues/30690)
    config: {{ config|json }}

[ppmathis]

I've discovered that even using {{ config|yaml }} is sufficient within defaults or context to not mess with any new lines - all the content ends up correctly within the template.

[SEJeff]

@ppmathis I was literally about the say the exact same thing as I ran into this problem as well.

The correct solution is to use the |yaml jinja filter. If you want the exact formatting you can also disable the flow control via |yaml(None), depending on your use case.

cc: @cachedout this issue should be closed with some user documentation or something.

[hkbakke]

Just a side question, but I use |json instead of |yaml because json is normally more efficient than yaml. Is there any reason to use the yaml filter instead of json for the use case in this issue?

[Armadill0]

@hkbakke I could only think of readability in the debug logs.

Anyway thanks for this "workaround". I also heard about it at SaltConf17. :smile:

[danielmorlock]

I have the same problem with in 2017.7.7 and the workaround with |yaml and or |json does not work:

Having the following pillar:

context:
  config: |
    [foo]
    line1
    line2

and the following in my state:

/tmp/test
  file.managed:
    - source: salt://{{ slspath }}/template.yml
    - template: jinja
    - defaults:
        context: {{ salt.pillar.get('context') }}

template.yml:

config: |
  {{ context.config | yaml }}

Result:

config: |
  [foo]\\nline1\\nline2\\n

The only workaround that actually works is using |replace('\\\\n', '\n').

[nevetS]

Seeing the same problem in 2019.2.0 (Fluorine)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.