Aws-cdk: cdk synth generates only json files when used with -o

[ghidi]

We don’t have plans to output YAML to cdk.out

The cdk.out directory is a build artifact and generally intended to be read by machines and not by humans. As such we want to make sure it’s contents are deterministic and consistent and YAML is less stable then JSON (there are quirks related to yaml version, parsing, multi-line, etc).

Any yaml parser should be able to read json out of the box because YAML is designed as a superset of json (all json is valid yaml).

If you need to convert json to human readable yaml, there are many many tools you could use.

Original post

Describe the bug
cdk synth ignores -j option when used with -o. It will always produce json files even if -j is false or not present. As a result, it is not possible to synthesize yaml files in a given directory.

To Reproduce
cdk synth -o stacks
cdk synth -j false -o stacks

Expected behavior
There should be yaml files in the stacks dir.

Version:

• OS MacOS
• CDK Version 0.34.0

[mhuebner]

Related to #2848

[rix0rrr]

Looking at #2848, there might be some conflation going on between the -o output directory and the cdk.out CloudAssembly directory.

[markusl]

Is there now a way to synthesize yaml files of all stacks with cdk synth command instead of specifying each of them manually? We'd like to produce yaml's of all stacks during a buildspec execution.

[eladb]

Currently there isn’t. Can you further describe your use case? Why isnt JSON sufficient? Are these templates inspected by humans?

[markusl]

@eladb what we do is we use CDK in codepipeline to generate cloudformation templates. Our buildspec contains something like this:

      - cdk synth stack1-test > stack1-test-template.yaml
      - cdk synth stack2-test > stack2-test-template.yaml
      - cdk synth stack1-prod > stack1-prod-template.yaml
      - cdk synth stack2-prod > stack2-prod-template.yaml

These templates are then used in different stages of codepipeline for updating different parts of our infra. For us it would also make sense that there would be cdk synth --all or similar command to just output everything using their default names in _same_ format cdk synth already uses (YAML).

I'm also interested if our approach has something that could be improved :) For now, this has worked quite well while parts of the infra are YAML and parts are CDK which is synthetized as part of pipeline process.

[eladb]

Hey @markusl! Thanks for the details.

The command cdk synth -o OUTDIR (and actually any call to cdk synth) will now emit a cloud assembly which, practically speaking, includes a template file for all the stacks in your app (in JSON format).

The default output directory is now cdk.out.

So the only difference for you would be:

$ cdk synth
$ ls cdk.out
stack1-test.template.json
stack2-test.template.json
stack1-prod.template.json
stack2-prod.template.json

YAML is JSON, so you could use these files anywhere that accepts YAML.

As for how to set up CI/CD for CDK apps, we have some initial experimental work in the app-delivery module, and a pending PR.

Closing for now. Please reopen if you see fit.

[robertd]

Honestly, current workflow is somewhat confusing (reference: https://github.com/awslabs/aws-cdk/pull/2636#issuecomment-496970443).

If I have a single stack and I run cdk synth I get YAML in the stdout. But if I have multiple stacks, then I get JSON in cdk.out. I agree with @markusl that It would be nice to have --all option and YAML option as well (more readable for humans imho). For example cdk deploy "*" uses wildcards, perhaps cdk synth should too.

Also, in our CI/CD pipeline we have cdk synth and cdk diff step that shows user CloudFormation template and proposed changes before moving forward (cdk deploy step is manual trigger in Gitlab CI/CD). Therefore it would be really nice if we can have YAML option back.

[robertd]

After chatting with @eladb on Gitter... we talked about having the ability to pass the -y flags to cdk synth (-j already exists)

i.e.: cdk synth -y -o stdout prints all stacks (even multiple ones) to a stdout as yaml. This would be very beneficial for our CI/CD pipelines where we have ops folks review the changes coming from cdk diff and cdk synth.

[antonfelich]

+1 our ops team review the changes before the pipeline runs out the new infrastructure. Everything they do is in yaml so it would be helpful for them if it was in the same "language".

[eladb]

This is not something we will likely support. cdk.out is designed to be machine-readable and there is value in keeping it "agnostic" of visual representation. YAML is not an ideal format for that.

It should trivial to convert the templates from cdk.out to YAML using any standard tool if you desire to review them as YAML.

[coding-jj]

Never the less what you want to do. CDK help is stating it wrong:

> cdk synth --help

cdk synthesize [STACKS..]

Synthesizes and prints the CloudFormation template for this stack

Options:
[...]
  --json, -j            Use JSON output instead of YAML when templates are
                        printed to STDOUT            [boolean] [Standard: false]
[...]
  --output, -o          Emits the synthesized cloud assembly into a directory
                        (default: cdk.out)                              [string]
[...]

There is nothing stated about ...:

• ... multiple Stacks always go to cdk.out
• ... multiple files will always be json
• ... there is nothing you can do about it not being json

[logemann]

Furthermore on this issue. If i want to use SAM together with CDK, does SAM accept JSON and if so... how to tell it that its not template.yaml but some stack json in cdk.out.? The documentation (https://docs.aws.amazon.com/cdk/latest/guide/tools.html#sam) about SAM <-> CDK Interop says nothing about JSON.

Update: Forget my question. Its possible via:
sam local invoke MyFunction1233456 -t lambdaStack.template.json
where lambdaStack.template.json is part of the cdk.out assembly. Just in case anybody wants to know ;-)

[eladb]

Furthermore on this issue. If i want to use SAM together with CDK, does SAM accept JSON and if so... how to tell it that its not template.yaml but some stack json in cdk.out.? The documentation (https://docs.aws.amazon.com/cdk/latest/guide/tools.html#sam) about SAM <-> CDK Interop says nothing about JSON.

Update: Forget my question. Its possible via:

sam local invoke MyFunction1233456 -t lambdaStack.template.json

where lambdaStack.template.json is part of the cdk.out assembly. Just in case anybody wants to know ;-)

JSON is valid YAML for all intents and purposes.

[evmin]

YAML is much easier to read for the human eye. Could be useful to have an explicit YAML option for the situations when the app needs to be distributed as a template.