Salt: yaml load unicode error salt3000.2 python2.7.5[BUG]

Created on 12 May 2020  ·  4Comments  ·  Source: saltstack/salt

Description
when we nested import_yaml for a SLS file, this had a multi-objects such as sequences or maps as a value for a yaml node.

Setup
SLS file like this:

{% import_yaml "config/xxx/xxx.yaml" as xxx %}

for SLS's yaml file like this:

{% import_json "config/xxx/xxx.json" as aaa with context %}
conf:
    nodes: {{ aaa.get('nodes') }}

for yaml's json file like this:

"nodes": {
                "servers": [
                    "app1:10790",
                    "app2:20790",
                    "app3:30790",
                    "app4:40790",
                    "app5:50790"
                ]
            }

Steps to Reproduce the behavior
command line

salt-call state.sls server.test_1 test=true

Expected behavior
if this useful, we can get xxx info in sls
but log error is:

[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 169, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 443, in render_jinja_tmpl
    trace=tracestr)
SaltRenderError: Jinja error: 'NoneType' object has no attribute 'splitlines'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 955, in load_yaml
    marker='    <======================')
  File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 527, in get_context
    template_lines = template.splitlines()
AttributeError: 'NoneType' object has no attribute 'splitlines'

---
{% import_yaml "config/xxx/xxx.yaml" as xxx %}
<======================

; line 1

Versions Report

salt --versions-report: 3000.2
python --versions-report: 2.7.5

Additional context
we change little code for py, we found error ouccered from yaml pakge(version=3.1.1)
scanner.py from_line: 1304

 if (self.flow_level and ch == u':'
                    and self.peek(length+1) not in u'\0 \t\r\n\x85\u2028\u2029,[]{}'):
                self.forward(length)
                raise ScannerError("while scanning a plain scalar", start_mark,
                    "found unexpected ':'", self.get_mark(),
                    "Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.")

then we write xxx.yaml to local
it seems unicode had been rendered to yaml file:

conf:
  nodes: {u'servers': [u'app1:10790', u'app2:20790', u'app3:30790', u'app4:40790', u'app5:50790']}
Bug cannot-reproduce info-needed

Most helpful comment

Yes, I see. Well, anyway. we solved this problem.
Thanks.

All 4 comments

Cannot reproduce.
I'm using Salt 3000.2 with python 2.7.17.
I've created the following files:

(salt27) $ cat /srv/salt/base/issue_yaml.sls 
{% import_yaml "config/xxx/xxx.yaml" as xxx %}
2020-05-12 12:41:09 dimm@kuzmenko /home/dimm/projects/salt/git/salt (⌐■_■) (HEAD detached at v3000.2)
(salt27) $ cat /srv/salt/base/config/xxx/xxx.yaml 
{% import_json "config/xxx/xxx.json" as aaa with context %}
conf:
    nodes: {{ aaa.get('nodes') }}
2020-05-12 12:41:22 dimm@kuzmenko /home/dimm/projects/salt/git/salt (⌐■_■) (HEAD detached at v3000.2)
(salt27) $ cat /srv/salt/base/config/xxx/xxx.json 
{"nodes": {
                "servers": [
                    "app1:10790",
                    "app2:20790",
                    "app3:30790",
                    "app4:40790",
                    "app5:50790"
                ]
            }}

Then run this:

(salt27) $ ./scripts/salt-call -l info state.sls issue_yaml test=true
[INFO    ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO    ] Loading fresh modules for state activity
local:

Summary for local
-----------
Succeeded: 0
Failed:   0
-----------
Total states run:    0
Total run time:  0.000 ms

Thanks for your comment!

I solved this problem by add code into xxx.yaml , next line after the import_json .
code is:

{% set new_content = aaa  | json_encode_dict %}

use the filter to encode this content。
It seems like used for our task.

But for your test, you can use the yaml in issue_yaml.sls to write in your local yaml file.
Maybe can reproduce.

I don't know if the python version is related to this test.
My python version: 2.7.5

@LeoDemoniac sounds legit to me. Just one suggestion: use tojson instead of the deprecated json_encode_dict.
Can we close this issue now?

Yes, I see. Well, anyway. we solved this problem.
Thanks.

Was this page helpful?
0 / 5 - 0 ratings