Salt: Using YAML anchors/references in Pillars causes conflicting IDs

[eliasp]

How to reproduce:

1. Create a Pillar like this:

``` YAML
yamltest:
defaults: &defaults
group: users
path: /var/lib/app
mask: '0644'

   definitions:
       main:
           <<: *defaults
       test:
           <<: *defaults
           path: /var/lib/testapp
       staging:
           <<: *defaults
           mask: '0555'

```

1. Use it in an SLS like this:

YAML {% for definition, data in salt['pillar.get']('yamltest:definitions').iteritems() %} yamltest-{{ definition }}: test.succeed_without_changes: - name: {{ definition }} {% endfor %}
3. Apply it to a Minion:

``` YAML
MN1221-C0013:
Data failed to compile:

---

   Pillar failed to render with the following messages:

---

   Rendering SLS 'pillars.yamltest' failed, render error:

Conflicting ID 'path'
```

The problem seems to be, that the keys duplicated by using the anchor/reference aren't merged before being further evaluated.

[cachedout]

Interesting report. I'm not sure we've ever had anybody try to use YAML anchors before so this has certainly been overlooked. We'll examine the feasibility of supporting this fully.

[eliasp]

Just did the following test to verify whether it's an issue within pyyaml itself, but it doesn't seem to be. The resulting data-structure is merged as expected.

Testscript:

#!/usr/bin/python
'''
Test merging of anchors/references in YAML data structures
'''

import yaml

YML = """
yamltest:
    defaults: &defaults
        group: users
        path: /var/lib/app
        mask: '0644'

    definitions:
        main:
            <<: *defaults
        test:
            <<: *defaults
            path: /var/lib/testapp
        staging:
            <<: *defaults
            mask: '0555'
"""

print yaml.dump(yaml.load(YML), default_flow_style=False)

Result:

yamltest:
  defaults:
    group: users
    mask: '0644'
    path: /var/lib/app
  definitions:
    main:
      group: users
      mask: '0644'
      path: /var/lib/app
    staging:
      group: users
      mask: '0555'
      path: /var/lib/app
    test:
      group: users
      mask: '0644'
      path: /var/lib/testapp

[cachedout]

Very interesting. Good news that it isn't an upstream bug. Seems like we should be able to solve this. Thanks for the good detective work here!

[mjgorman]

Hello, We use the same things. It was not an issues in 0.17.1 and prior. Worked fine, but now in trying to upgrade to 2014.7.0 we're really hitting this. For the moment to test other things in my development environment in yamlloader.py i've just commented out:

 79             #if key in mapping:
 80             #    raise ConstructorError('Conflicting ID {0!r}'.format(key))

What this really means right now is that we can't upgrade realistically.

[eliasp]

As a workaround, in some cases it might be possible to use the use requisite instead of YAML anchors/refs.

[mjgorman]

That won't work in pillar though(Right?), which is where we've run into render issues. We have a macro that render's defaults and then using anchors merges environmental specifics ontop of them. Rendered it looks like this:

owl-autoscale-sqs_default: &owl-autoscale-sqs_default
  sqs_region: us-west-1
  sqs_queue_name: php-autoscale-qa-queuePHPAutoscaleTerminationQueueQA
owl-autoscale-sqs_stg: &owl-autoscale-sqs_stg
  sqs_region: us-east-1
  sqs_queue_name: php-autoscale-stg-queuePHPAutoscaleTerminationQueueSTG
owl-autoscale-sqs_prd: &owl-autoscale-sqs_prd
  sqs_region: us-east-1
  sqs_queue_name: php-autoscale-prd-queuePHPAutoscaleTerminationQueuePRD

owl-autoscale-sqs_base: &owl-autoscale-sqs_base
  <<: *owl-autoscale-sqs_default
owl-autoscale-sqs_local: &owl-autoscale-sqs_local
  <<: *owl-autoscale-sqs_default
owl-autoscale-sqs_bottle: &owl-autoscale-sqs_bottle
  <<: *owl-autoscale-sqs_default
owl-autoscale-sqs_dev: &owl-autoscale-sqs_dev
  <<: *owl-autoscale-sqs_default
owl-autoscale-sqs_ci: &owl-autoscale-sqs_ci
  <<: *owl-autoscale-sqs_default
owl-autoscale-sqs_cid: &owl-autoscale-sqs_cid
  <<: *owl-autoscale-sqs_default
owl-autoscale-sqs_qa: &owl-autoscale-sqs_qa
  <<: *owl-autoscale-sqs_default

owl-autoscale-sqs: &owl-autoscale-sqs
  <<: *owl-autoscale-sqs_default
  <<: *owl-autoscale-sqs_base

As you said above it evaluated before merging so the pillar complains conflicting ID.

[eliasp]

That won't work in pillar though(Right?)

Uh, yeah… you're right. May brain wasn't awake yet when I wrote the suggestion regarding use requisites, sorry :)

[mikeywaites]

Any news on this?

[hugochinchilla]

Please, fix this soon.

[hemebond]

I'm currently testing out https://gist.github.com/hemebond/623f0cc1f14392d62d89 which discards duplicate keys when merging (done at the very bottom of the method). Feedback very welcome.

Also, #8053 appears to be the original reason for the duplicate detection. I've just tested that and I still get an error if I create states with the same name.