Sceptre: config files included with {% include %} does not format correctly

[heikkima]

Scpetre version: Sceptre, version 2.3.0
Issue: {{ stack_group_config }} does not format correctly when included with {% include %}

Demo:

.
├── config
│   ├── config.yaml
│   └── dev
│       ├── config-templates
│       │   └── tags.yml
│       ├── config.yaml
│       └── s3.yaml
└── templates
    └── cf-s3.yaml.j2

We have shared config files in config-templates/ directory and we use these config files with {% include %} in stack's main config -file.

In this demo our stack's main config -files path is ./config/dev/s3.yaml

# ./config/dev/s3.yaml

template_path: cf-s3.yaml.j2

parameters:
  BucketName: {{ stack_group_config.s3_bucket_name }}-{{ stack_group_config.environment }}

{% include 'config-templates/tags.yml' %}

# ./config/dev/config-templates/tags.yml

sceptre_user_data:
  {% if stack_group_config.variables[stack_group_config.environment].tags is defined %}
  tagKeyPairs:
    {{ stack_group_config.variables[stack_group_config.environment].tags }}
  {% endif %}



md5-08200a9b7acf3e8326ccb439727a6fbc



# ./config/config.yaml

project_code: 'sceptre-demo'
region: 'eu-west-1'

s3_bucket_name: sceptre-demo-bucket-name

variables:
  dev:
    tags:
      FOO: 'barDev'
      BAZ: 'bazDev'
      BAR: 'fooDev'



md5-5c5699e76976f582a761e79790bb3ca7



expected alphabetic or numeric character, but found '#'
  in "<unicode string>", line 9, column 7:
        {&#39;FOO&#39;: &#39;barDev&#39;,  ... 
          ^



md5-2cebc76754a0132d3975b672333b1ac2



# ./config/dev/s3.yaml

template_path: cf-s3.yaml.j2

parameters:
  BucketName: {{ stack_group_config.s3_bucket_name }}-{{ stack_group_config.environment }}

sceptre_user_data:
  {% if stack_group_config.variables[stack_group_config.environment].tags is defined %}
  tagKeyPairs:
    {{ stack_group_config.variables[stack_group_config.environment].tags }}
  {% endif %}

Running command sceptre status dev again will not throw an error.

This setup was also tested with Sceptre, version 2.2.1 and it worked fine with {% include %}

[ngfgrant]

Just as a quick elimination thing - can you change tags.yml to tags.yaml and try again? There was a Jinja vulnerability which we patched between sceptre 2.2.1 and 2.3.0 so that might be something we need to look at as well.

[heikkima]

Changing tags.yml to tags.yaml result in different error:

"Required attribute 'template_path' not found in configuration of 'dev/config-templates/tags'."

[ngfgrant]

Ok that second error makes sense - I see that you are making use of the fact we don't use .yml extension and importing that. This is very likely to be a Jinja thing.

[ngfgrant]

Here is the only change that happening to our handling of Jinja between 2.2.1 and 2.3.0 https://github.com/Sceptre/sceptre/commit/6d4124bdc9d4d5277c3f5cdc5eb7d6db9696b667 - does this look like something that would have affected your setup?

[heikkima]

I believe select_autoescape() function is the reason why this doesn't work in 2.3.0.
I tried overriding autoescape with {% autoescape false %} and it fixed our problem (link to jinja2 documentation)

This works:

# ./config/dev/config-templates/tags.yml

sceptre_user_data:
  {% if stack_group_config.variables[stack_group_config.environment].tags is defined %}
  tagKeyPairs:
    {% autoescape false %}
    {{ stack_group_config.variables[stack_group_config.environment].tags }}
    {% endautoescape %}
  {% endif %}

[ngfgrant]

@heikkima Thanks for follow up with this. If you had time would you be willing to add a note into our documentation for this?