Salt: 2018.3 regression file.managed.context parsing

[noelmcloughlin]

Description of Issue/Question

With 2018.3 release, there is regression in file.managed states WRT found unexpected ':' issue.
Sorry I have no logs available.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

These values have embedded Colon" and file.managed context parsing fails.

etcd:
  service:
    initial_cluster: etcd0=https://127.0.0.1:2380
    initial_cluster_state: new
    initial_cluster_token: etcd-cluster-1
    initial_advertise_peer_urls: https://127.0.0.1:2380
    listen_peer_urls: https://127.0.0.1:2380
    listen_client_urls: https://127.0.0.1:2379
    advertise_client_urls: https://127.0.0.1:2379


..

  file.managed:
    - template: jinja
    - context:
      etcd: {{ etcd }}

Workaround:

    - context:
      etcd: {{ etcd|json }}

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

A colleague verbally reported same issue with different formula so looks like regression.

Reverting to 2017.7 stable resolved any problems.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

2018.3 (not available, I have downgraded to 2017.7).

[Ch3LL]

can you include how you are setting: {{ etcd }} and the info for it, its not clear to me how that is set in this example.

[noelmcloughlin]

Hi @Ch3LL See example defaults.yaml below where the context dict data originated.

defaults.yaml

etcd:
  a: b
  c: d

  lookup:
     e: f
     g: h

  service:
    initial_cluster: etcd0=https://127.0.0.1:2380
    initial_cluster_state: new
    initial_cluster_token: etcd-cluster-1
    initial_advertise_peer_urls: https://127.0.0.1:2380
    listen_peer_urls: https://127.0.0.1:2380
    listen_client_urls: https://127.0.0.1:2379
    advertise_client_urls: https://127.0.0.1:2379

With 2018.3 the following values were tested in defaults.yaml.

• https://127.0.0.1:2380 failed
• https{{ ':' }}127.0.0.1{{ ':' }}2380 failed
• https//127.0.0.12380 worked (not very useful)

I can try to replicate next week if necessary.

[Ch3LL]

apologies i should have been more clear. Are you importing that etcd data from yaml file like so ? {% import "test.yaml" as blah %} or pillar data, etc?

[noelmcloughlin]

NP, I checked the etcd PR code.

• The install.sls has {% from "etcd/map.jinja" import etcd with context -%}.
• The map.jinja has {% import_yaml 'etcd/defaults.yaml' as defaults %} before merging
• My only pillar data was

etcd:
  lookup:
    version: 3.2.18

[pgporada]

We're seeing this even with quotes around the value.

etcd:
  config_path: /etc/etcd/etcd.conf.yml
  name: my_etcd_node0
  service:
    initial_cluster: 'etcd0=https://127.0.0.1:2380'

The workaround solution was to change from this

etcd_config:
  file.managed:
          - name: {{ etcd.config_path }}
          - template: jinja
          - mode: 644
          - user: root
          - group: root
          - defaults:
              etcd: {{ etcd }}

to putting the pillar values directly into the template instead which works
/etc/etcd/etcd.conf.yml

{%- from "etcd/map.jinja" import etcd with context %}
# This is the configuration file for the etcd server.

# Human-readable name for this member.
name: '{{ etcd.name }}'
...
...

The error looks like

my-host-box:
    - Rendering SLS 'base:etcd' failed: found unexpected ':'; line 50

      ---
      [...]
          - template: jinja
          - mode: 644
          - user: root
          - group: root
          - defaults:
              etcd: {u'etcd': {u'config_path': u'/etc/etcd/etcd.conf.yml', u'name': u'my_etcd_node0........<======================

[rmarchei]

I have a similar issue when importing from a yaml file

# salt-call --versions-report
Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: 0.2.4
        Tornado: 4.2.1
            ZMQ: 4.2.3

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-87-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[Martin819]

I have run into the same problem when importing yaml file which contains colon ':' in the key or value. It fails with unexpected ':' even if it's surrounded with quotes.

{%- from "ceilometer/map.jinja" import server with context %}

server_config:
  file.managed:
    - name: /path/to/file
    - defaults:
        values: {{ server }}

Pillar:

ceilometer:
  server:
    policy: 'rule:user'
    'telemetry:resource': 'value'

$ salt-call --versions-report
Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.9.1
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.13.0-38-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[binocvlar]

Same regression found here.

Everything works as expected under 2017.7.5 (Nitrogen).
Upon upgrading to 2018.3.0 (Oxygen), the bug appears.

Further info:

I was also importing a dictionary from a YAML file. The dictionary values were double-quoted strings containing a colon (':') character.

The workaround was to port the datastructure from the YAML file into pillar. This change allowed my statefile to render correctly.

[Ch3LL]

okay thanks for the examples. I was able to replicate this and bisect it to: 9ee5b417f775ea307f508459b037514cdf67021f

and see this error:

[CRITICAL] Rendering SLS 'base:etcd.install' failed: found unexpected ':'; line 51

For anyone that wants to easily replicate this issue here is a docker container:

docker run -it -v ~/git/salt/:/testing/ ch3ll/issues:46929 /bin/bash -c "salt-call --local state.show_sls etcd.install -ldebug" (where ~/git/salt is a local cloned git repo of salt)

[terminalmage]

Fixed in https://github.com/saltstack/salt/pull/47104