Salt: Using yaml serializer inside jinja template results in unicode being prepended by '!!python/unicode'

[favoretti]

Given the following example:

files/elasticsearch.yml:

{%- from 'elasticsearch/settings.sls' import es_config_defaults with context %}# MANAGED BY SALT
{{ es_config_defaults|yaml(False) }}

settings.sls:

{% set master_nodes = ["bla", "bla", "bla"]|join(",") %}

…
{% set es_config_defaults = {
        'unicast': {
          'hosts': master_nodes
        }
%}

Resulting rendered file will look like:

unicast:
  hosts: !!python/unicode bla,bla,bla

Looking around this looks like it's not really salt's state per se, rather pyyaml's, as stated, for instance in http://stackoverflow.com/questions/1950306/pyyaml-dumping-without-tags

https://github.com/saltstack/salt/blob/develop/salt/utils/jinja.py#L387 uses yaml.dump, which according to the documentation will always prepend unicode objects. yaml.safe_dump will not, although it can't dump arbitrary python objects, when yaml.dump, in fact, can.

A "workaround" might be {{ es_config_defaults|yaml(False)|replace("!!python/unicode", "") }}, which is so dirty - it hurts.

Anyone else stumbled on this? Did I miss anything? Am I overlooking something obvious?

[basepi]

My vote is still to just create another filter for safe_dump. I mean, I'd probably vote to just make the existing filter safe_dump, but that could break existing setups.

[basepi]

I added the filter really quick. I didn't have a chance to test it, so if you have a minute that would be awesome. #30481

[favoretti]

Awesome, thank you, we will test and report back.

[mdevreugd]

I quick patched the salt master code with #30481 to see if it works, and unfortunately it turns out it didn't:

~# grep 'yaml_safe' /usr/lib/python2.7/dist-packages/salt/utils/jinja.py
            'yaml_safe': self.format_yaml_safe,
    def format_yaml_safe(self, value, flow_style=True):

~# cat /srv/salt/states/elasticsearch/files/etc/elasticsearch/elasticsearch.yml
{%- from 'elasticsearch/settings.sls' import config with context %}# MANAGED BY SALT
{{ config|yaml_safe(False) }}

~# salt '*' state.highstate
Total states run:     36

~# salt 'node003*' file.grep /etc/elasticsearch/elasticsearch.yml 'python'
node003:
    ----------
    pid:
        27329
    retcode:
        0
    stderr:
    stdout:
                hosts: !!python/unicode bla,bla,bla
                hosts: !!python/unicode bla,bla,bla

[favoretti]

@basepi I can't re-open the issue, but apparently safe_dump doesn't fix the issue :(

[basepi]

Hmmm, I guess I was under the impression you guys had tried changing the existing filter and it had worked. Well, back to the drawing board.

[hemebond]

What is the reason for not adding the python/unicode constructor to the SaltYamlSafeLoader class?

[multani]

I'm facing the same problem here. I made a small patch which allows to deserialize from YAML these values properly, as suggested by @hemebond .

It also reverts the #30481 as it never worked in the first place.

[pjcreath]

I ran into a cousin of this issue (in 2017.7.2) when serializing a Jinja variable into a state parameter:

{% set oldstring = "oldstring" %}
{% set olding = 123 %}
{% set mystring = "%s:%d"|format(oldstring, oldint) %}

mystate:
  file.managed:
    ...
    - default:
        parameter: {{ mystring }}

This gets rendered as parameter: u'oldstring:123', which fails in two ways: (1) pyyaml doesn't know how to parse u'. (2) As a result, the state fails to compile because pyyaml finds the : separator outside of a string.

It looks like all of Jinja's built-in string formatting functions promote everything to unicode. That includes at least join(), format(), and string(). ("string()" is at least documented to convert to unicode.) As a result, if I do any formatting in Jinja, I end up with broken substitutions like the above. My workaround was:

        parameter: {{ mystring|yaml }}

This gets rendered as !!python/unicode "oldstring:123", which, while ugly, at least works.

I'm mentioning it here mostly so that the workaround will save someone some time and headache in the future.