Salt: Jinja renderer doesn't work on loaded data / OrderedDict

[autocracy]

Description of Issue/Question

Get python state representation rather than json

Setup

test.sls:

{% load_yaml as consul_config %}
  data_dir: /var/run/consul
  server: true
  bootstrap_expect: 1
{% endload %}

/etc/consul/server.conf:
  file.managed:
    - makedirs: True
    - contents: {{ consul_config|json }}

State after execution:

$ cat /etc/consul/server.conf 
OrderedDict([('bootstrap_expect', 1), ('data_dir', '/var/run/consul'), ('server', True)])

Versions Report

  salt-call --versions-report                                                                                                                                
  Salt Version:
           Salt: 2016.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jul  1 2016, 15:12:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:                                                                                                                                           
           dist: Ubuntu 16.04 xenial
        machine: x86_64   
        release: 4.4.0-31-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[Ch3LL]

@autocracy i'm able to replicate this with this example:

{% set data = {
    'foo': True,
    'bar': 42,
    'baz': [1, 2, 3],
    'qux': 2.0
}%}

/etc/consul/server.conf:
  file.managed:
    - makedirs: True
    - contents: 
        {{ data|yaml }}

[ch3ll@thecakeisalie ~]$ cat /etc/consul/server.conf 
OrderedDict([('bar', 42), ('baz', [1, 2, 3]), ('foo', True), ('qux', 2.0)])

If you look at the logs though it does show it rendering correctly:

[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/issues/35375.sls:


/etc/consul/server.conf:
  file.managed:
    - makedirs: True
    - contents: 
        {bar: 42, baz: [1, 2, 3], foo: true, qux: 2.0}

[DEBUG   ] LazyLoaded config.get

So it seems there is an issue with the contents

[autocracy]

I love your hostname.

So staring at this a bit and trying it, I realize it is not rendering correctly in your example. To be valid json, they keys would have to be quoted.

... this change actually worked:

<     - contents: {{ data|json }}
---
>     - contents: {{ data|json|json }}

[Ch3LL]

@autocracy hehe glad you like the hostname.

Also thanks for pointing that out. I was also able to change your test case to {{ consul_config|json|json }} and it seems to be workign as well:

[ch3ll@thecakeisalie ~]$ cat /etc/consul/server.conf 
{"bootstrap_expect": 1, "data_dir": "/var/run/consul", "server": true}

[autocracy]

A nudge. While the workaround works, this is a bug. Is the priority lower than stated, or otherwise lost?

[muravitskiy]

any changes?

[Ch3LL]

i've been re-looking at this and i noticed that the state formatted like this works:

/etc/consul/server.conf:
  file.managed:
    - makedirs: True
    - contents: |
        {{ consul_config|json }}

The pipe tells yaml this will be a multiline statement string. Without the pipe its interpreting the data type as a dict. I believe this might be a yaml idiosyncrasy but not certain.

As seen in python interpreter:

>>> yaml.safe_load("""
... |
...  {"bootstrap_expect": 1, "data_dir": "/var/run/consul", "server": true}
... """)
'{"bootstrap_expect": 1, "data_dir": "/var/run/consul", "server": true}\n'

>>> yaml.safe_load("""{"bootstrap_expect": 1, "data_dir": "/var/run/consul", "server": true}""")
{'data_dir': '/var/run/consul', 'bootstrap_expect': 1, 'server': True}

so its a string vs a dictionary. To prove this point further:

>>> type(yaml.safe_load("""
... |
...  {'data_dir': '/var/run/consul', 'bootstrap_expect': 1, 'server': True}
... """)
... 
... )
<type 'str'>

>>> type(yaml.safe_load("""{'data_dir': '/var/run/consul', 'bootstrap_expect': 1, 'server': True}""")
... )
<type 'dict'>
>>> 

So i'm wondering if this would be expected behavior but i want to get @saltstack/team-core 's opinions here whether this is a bug or expected behavior and the intended behavior is to use the pipe here.

[steverweber]

there is no issue here. Remember json is valid yaml so you need to double wrap it in some cases when you dont want it read as yaml / state.

using {{var|json|json}} is a good solution if you want to pass json into a file as well as:

contents: |
   {{var|json}}

{{var|json}} converts the var into json, the extra |json encodes the json data as a larger json string that yaml will read without interpreting it as the original dict

[autocracy]

1.5 years later, self-closing per @steverweber's comment.