Salt: List variables do not parse into templates properly

[Kaydub00]

Description of Issue/Question

Setup

Manage the file:

random-config:
  file.managed:
    - name: /etc/random.conf
    - source: salt://module/templates/random.conf.j2
    - template: jinja
    - defaults:
        host_list: ["host-1.domain.com","host-2.domain.com","host-3.domain.com"]

The line in the template module/templates/random.conf.j2:

random.property: {{ host_list }}

Steps to Reproduce Issue

Create a salt and pillar module, set up a file.managed in the module and set one variable in the file. Then set the variable with something like:
["host-1.domain.com","host-2.domain.com","host-3.domain.com","host-4.domain.com"]

If the String ["host-1.domain.com","host-2.domain.com","host-3.domain.com"] is present in the template where it is supposed to be this is recognized as a List. Using test=True I can see that Salt sees the value in the template as [u"host-1.domain.com",u"host-2.domain.com",u"host-3.domain.com"] even though the value is what it's supposed to be. Salt then tries to change this value to ["host-1.domain.com","host-2.domain.com","host-3.domain.com"] but instead changes it to
[u"host-1.domain.com",u"host-2.domain.com",u"host-3.domain.com"]

I have tried multiple methods of circumventing this, but nothing has worked. (tried piping to a join(''), tried changing the variables to be between quotes, double and single.) I believe this may be a Jinja issue?

Versions Report

Master:
Salt Version:
Salt: 2018.3.0

Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.1
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: 3.4.3
pygit2: Not Installed
Python: 2.7.5 (default, Aug 4 2017, 00:39:18)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.4.1708 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-693.21.1.el7.x86_64
system: Linux
version: CentOS Linux 7.4.1708 Core

Minion:
salt-minion 2018.3.0 (Oxygen)

[gtmanfred]

Json is syntactically correct yaml, so what you are actually setting is

random-config:
  file.managed:
    - name: /etc/random.conf
    - source: salt://module/templates/random.conf.j2
    - template: jinja
    - defaults:
        host_list:
          - host-1.domain.com
          - host-2.domain.com
          - host-2.domain.com

If you want the host list to be an actual string, then you will need to quote it.

random-config:
  file.managed:
    - name: /etc/random.conf
    - source: salt://module/templates/random.conf.j2
    - template: jinja
    - defaults:
        host_list: '["host-1.domain.com","host-2.domain.com","host-3.domain.com"]'

Can you try this?
Thanks,
Daniel

[Kaydub00]

Yes, I did try this and I just re-attempted this. I get the same results.

Salt sees the String in the template as a List and tries to update it to the string, but instead changes it to the list.

So when I run state.apply I can see that salt sees the string in the file as
'[u"host-1.domain.com",u"host-2.domain.com",u"host-3.domain.com"]

The output from the Salt state.apply says the above string will be changed to
["host-1.domain.com","host-2.domain.com","host-3.domain.com"]

This is what the string in the managed file already is...

Then Salt will change the string in the file to
'[u"host-1.domain.com",u"host-2.domain.com",u"host-3.domain.com"]

I've tried a couple of different ways to do this, from quoting the pillar variable/default variable, using some jinja methods like join('') to try to force a string, etc.

I'd like to add that this DID work before on salt-2017.7.4-1.el7.noarch and salt-master-2017.7.4-1.el7.noarch. After upgrading to salt-2018.3.0-1.el7.noarch this stopped working as it did.

[gtmanfred]

This is probably caused by our change to use unicode_literals in 2018.3, so that we could have better support for unicode, and an easier transition to python3 support.

The fastest way to get around this is to just pass a list in, and do this in your file.managed template.

["{{host_list|join('", "')|string()}}"]

@terminalmage i know you worked on removing the 'u' prefix for strings when dumping stuff to yaml, would you happen to know what to do to fix this here? When outputting a list in jinja, it is now including it as unicode.

Thanks,
Daniel

[Kaydub00]

@gtmanfred That change did work so I'm updating this in my code.

[terminalmage]

When you replace a jinja placeholder with a list/dict, Jinja just calls repr() on it under the hood, hence the u prefixes on the strings. There's not a lot we can do about that. It's possible that we may be able to use a jinja filter to on-the-fly convert the strings in the list back to str types. I'll investigate doing this.

[terminalmage]

Also, I tried @gtmanfred's suggestion to quote the list, and it works just fine. I've created a docker image to demonstrate this, which will run Salt out of a git clone of the Salt source code. To use this image, first clone the Salt repo, then run git checkout v2018.3.0 to check out the 2018.3.0 release. Then, run the docker run command from the root of the git clone.

As you can see, it works just fine when quoting the JSON:

% docker run --rm -it -v $PWD:/testing terminalmage/issues:47001
[root@00568760ef8b /]# salt-call test.version
local:
    2018.3.0
[root@00568760ef8b /]# cat /srv/salt/foo.sls 
/tmp/foo.txt:
  file.managed:
    - source: salt://foo.txt
    - template: jinja
    - defaults:
        host: '["foo","bar","baz"]'
[root@00568760ef8b /]# cat /srv/salt/foo.txt
{{ host }}
[root@00568760ef8b /]# salt-call state.apply foo
local:
----------
          ID: /tmp/foo.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/foo.txt updated
     Started: 14:20:12.942282
    Duration: 115.216 ms
     Changes:   
              ----------
              diff:
                  New file
              mode:
                  0644

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 115.216 ms
[root@00568760ef8b /]# cat /tmp/foo.txt
["foo","bar","baz"]

It's only when I remove the quotes that PyYAML loads it as a list and Jinja uses repr() when templating, resulting in the u prefix on the strings:

```
[root@00568760ef8b /]# vim /srv/salt/foo.sls
[root@00568760ef8b /]# cat /srv/salt/foo.sls
/tmp/foo.txt:
file.managed:
- source: salt://foo.txt
- template: jinja
- defaults:
host: ["foo","bar","baz"]
[root@00568760ef8b /]# salt-call state.apply foo

local:

      ID: /tmp/foo.txt                                                                                                                                       
Function: file.managed                                                                                                                                       
  Result: True
 Comment: File /tmp/foo.txt updated
 Started: 14:22:02.074853
Duration: 36.339 ms
 Changes:   
          ----------
          diff:
              --- 
              +++ 
              @@ -1 +1 @@
              -["foo","bar","baz"]
              +[u'foo', u'bar', u'baz']

Summary for local

Succeeded: 1 (changed=1)

Failed: 0

Total states run: 1
Total run time: 36.339 ms
[root@00568760ef8b /]#

[Kaydub00]

Hmmm, not sure why it didn't work in my instance. I've already updated my code and I'm happy with how this works (I think this is cleaner anyways). I'd be happy with closing this ticket, I just wasn't sure if this was a problem for others or something you guys would want to look into.

[AlonSh]

The solution with the quotes does not work in my instance as well. (same salt version)

I'm using my list that get's converted to Unicode to look for other values inside my configuration.
So I'm having the problem:
'dict object' has no attribute u'some_string'
But it does have the attribute 'some_string'.

Any ideas?

(@noaginzbursky)

[terminalmage]

and then when I want to use the values inside the list and compare them to other strings in my code - it doesn't work cause 'some_type1' != u'some_type1'

I don't understand what you mean. The u prefixes on the values in your screenshot are only there because it's a repr() representation of a list, which is what you get in Python when you format a data structure as a string. What shows in the changes field from the state return shows that the individual strings in the list do not have the u prefix.

Can you elaborate on how you are comparing them to "other strings in your code"? For example, what are you doing, what do you expect to happen, and what actually happens?

[terminalmage]

@AlonSh Please provide SLS examples and examples of the error, thanks.

[chrispetsos]

Same problem here with Salt version 2018.3.2.
Imagine this pillar,

a_pillar:
  a_list: '["item1","item2","item3"]'

I am referencing this pillar inside a state file as follows,

{% set the_pillar = salt['pillar.get']('a_pillar') %}
a_state:
  file.recurse:
    - name: /target/folder
    - source: salt://source/folder
    - template: jinja
    - context:
        list_items: {{ the_pillar.a_list }}

and this is rendered as,

[u'item1',u'item2',u'item3']

within the files that reference the context var 'list_items' inside the folder where I am recursing.
I have managed to overcome this by playing a bit with where brackets are added. Note the following changes in the pillar and state files,

a_pillar:
  a_list: '"item1","item2","item3"'

{% set the_pillar = salt['pillar.get']('a_pillar') %}
a_state:
  file.recurse:
    - name: /target/folder
    - source: salt://source/folder
    - template: jinja
    - context:
        list_items: '[{{ the_pillar.a_list }}]'



md5-70710aeedc59fd14532ec029f83fa0db



["item1","item2","item3"]

Note the change from single quote ' to double quotes ".
Hope this is helpful in debugging this...

[terminalmage]

Since JSON is a subset of YAML, you can dump a Python data structure to JSON using Jinja's tojson filter and the u prefix will be removed from each string:

{{ myvar | tojson }}

This filter is new in Jinja 2.9, so you may need to upgrade Jinja to use it. Alternatively, I have ported this filter into Salt for the upcoming 2018.3.3 release, which will make it available to those who have older Jinja installed.

[eric-anderton-at-sony]

I'd like to ask that this bug be raised in priority.

My team is still seeing this bug, as of SaltStack 2018.3.2. Jinja appears to be using repr() somewhere under the hood to serialize types like list and OrderedDict, which is causing python-style u prefixed strings to show up in various places. Sending all our mapped pillar data (e.g. map.jinja) through the json filter just in case we happen to have Unicode values is usable workaround, but not really a solution.

I'm guessing that the problem may be more fundamental to Jinja itself. If so, please let us know - I can't speak for everyone, but I would be happy to file a bug against Jinja to make this behavior configurable somehow.

[terminalmage]

Given that I ported Jinja's tojson filter to Salt for the 2018.3.3 release (and later), and that Jinja >= 2.9 already has this filter, in my opinion this issue is fixed.

What you are expecting is for invalid YAML to be treated as valid YAML, which is problematic to say the least. Salt has tried for a while to do this via adding several hacks to our custom YAML loader, but this has in turn caused several bugs in the loader. Not only that, but removing the hacks has allowed us to support using the libyaml versions of PyYAML's SafeLoader in the upcoming Fluorine release, resulting in a significant performance boost.

Given that removing these hacks has not only simplified our custom YAML loader, but also made it more reliable and faster, I don't see us ever going back and attempting to make non-YAML be processed as YAML.

Yes, Jinja uses repr() on data structures. This is not likely to ever change, and I honestly don't think it should, but you're welcome to take it up with the Jinja maintainers. Jinja and YAML are two separate projects, and it's not necessarily in Jinja's interests to make rendered data structures parse as valid YAML.

[OrlandoArcapix]

There are |yaml and |json filters available under 2018.3.2 with python-jinja2 2.7.2 which seem to do the job nicely as well. Out of interest - how do these compare with |tojson?

[terminalmage]

@OrlandoArcapix Those are not part of Jinja and look like they were added by someone else within salt/utils/jinja.py. I was not aware of that particular JSON filter. Both seem to do similar work, so either should work fine.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Paulo-Nunes]

I am experiencing similar problems here.

Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.11.1
        libgit2: 0.20.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.5
      pycparser: Not Installed
       pycrypto: 3.9.0
   pycryptodome: Not Installed
         pygit2: 0.20.3
         Python: 2.7.14 (default, Jan 31 2018, 02:12:13)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: redhat 6.10 Santiago
         locale: UTF-8
        machine: x86_64
        release: 2.6.32-754.28.1.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.10 Santiago

Similar to the above I am using file.managed with template: jinja and context.
Unlike the above I am passing in to context a dictionary with sub dictionaries and lists:

I create a variable that pulls from pillar:
{% set someVar = pillar_dictionary %}

pillar_dictionary is defined here:

a:
  b:
    c:
      x: 1
      y: 2
      z: 3
      d: 
        - foo
        - bar
        - baz

So someVar contains the above dictionary and I pass it into the template file

ID:
  file.managed:
    - template: jinja
    - context: 
        a: {{ someVar | json }}

In the template I might refer to the variables like:

templateProperty = {{ a.b.c.d }}

I expect the rendered template to look like:

templateProperty = ["foo","bar","baz"]

Instead I see the below:

templateProperty = [u'foo',u'bar',u'baz']

I am not sure what to do here. I have tried wrapping {{ someVar | json }} in single quotes, but that leads to an error.
I have tried wrapping the pillar data like:

...:
  - '"foo"'
  - '"bar"'
  - '"baz"'

but the output becomes:

[u'"foo"', u'"bar"', u'"baz"']

I updated Jinja2 on the master to the latest available via pip.

Not sure what else I can do.

I think the only thing that did work was modifying the template itself:

templateProperty = {{ a.b.c.d | json}}

But that seems wrong. I don't want to modify the template. I want to pass a data structure to the template and just have it work.

[terminalmage]

I think the only thing that did work was modifying the template itself:

templateProperty = {{ a.b.c.d | json}}

This is the correct solution.

As noted elsewhere in this comment thread, when a data structure is dumped using a jinja placeholder, jinja simply runs repr() on the variable:

>>> from __future__ import unicode_literals
>>> x = ['foo', 'bar', 'baz']
>>> x
[u'foo', u'bar', u'baz']
>>> print(repr(x))
[u'foo', u'bar', u'baz']

Using the json or tojson filter is equivalent to running json.dumps() on the data structure, resulting in a representation of that data without the u prefixes:

>>> import json
>>> print(json.dumps(x))
["foo", "bar", "baz"]

[Paulo-Nunes]

As noted elsewhere in this comment thread, when a data structure is dumped using a jinja placeholder, jinja simply runs repr() on the variable:
Using the json or tojson filter is equivalent to running json.dumps() on the data structure, resulting in a representation of that data without the u prefixes:

Even if I already did a json dumps in the state?

ID:
  file.managed:
    - template: jinja
    - context: 
        a: {{ someVar | json }}

I use the json filter in the state that passes the context to the jinja template. Why would I have to do a dump of the data again from within the template? I guess I was just expecting all the data I pass to the json filter to be ... filtered.

[gtmanfred]

the result is that is reread back into python and turns into unicode, and then gets dumped again with the u prefixes. The other solution is to move to using the python3 version of salt, and you lose this weird artifact of legacy python, where strings were bytes and not unicode.

[terminalmage]

As noted elsewhere in this comment thread, when a data structure is dumped using a jinja placeholder, jinja simply runs repr() on the variable:
Using the json or tojson filter is equivalent to running json.dumps() on the data structure, resulting in a representation of that data without the u prefixes:

Even if I already did a json dumps in the state?

ID:
  file.managed:
    - template: jinja
    - context: 
        a: {{ someVar | json }}

I use the json filter in the state that passes the context to the jinja template. Why would I have to do a dump of the data again from within the template? I guess I was just expecting all the data I pass to the json filter to be ... filtered.

You are misunderstanding how this works. You are using the json filter to coax a Python data structure to JSON so that it can be loaded as YAML back into a Python dictionary (where the strings internally will be unicode).

These are _two independent Jinja templates_ (the SLS file and the file being managed), and they are compiled separately. If you were passing JSON into the Jinja context, you wouldn't be able to treat it like a nested dictionary (i.e. {{ a.b.c.d }}).