Aws-cli: Merge cloudformation --cli-input-json parameters and --parameters

[atheiman]

Given this cloudformation JSON input file input.json:

{
  "StackName": "MyStack",
  "Parameters": [
    { "ParameterKey": "A", "ParameterValue": "a" },
    { "ParameterKey": "B", "ParameterValue": "b" }
  ],
  // ...
}

I would expect this command

aws cloudformation create-stack --stack-name 'MyStack' \
  --template-body "file://$PWD/template.json" \
  --cli-input-json "file://$PWD/input.json" \
  --parameters 'ParameterKey=B,ParameterValue=B' \
  --parameters 'ParameterKey=C,ParameterValue=C'

to merge the --parameters overriding the --cli-input-json "Parameters" key to make something like

[
  { "ParameterKey": "A", "ParameterValue": "a" },
  { "ParameterKey": "B", "ParameterValue": "B" },
  { "ParameterKey": "C", "ParameterValue": "C" }
]

The use case for this is I have a cli input json stored in source control for a database stack. There are parameters for dns record, db user name, and db password. I dont want to keep the password in source control with the other params, so I want to deploy with the command I specified above and merge --parameters overriding Parameters in the input json file.

The documentation I was able to find (http://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html) seems to imply this works, but I was not able to get it working:

--cli-input-json (string) Performs service operation based on the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, the CLI values will override the JSON-provided values.

[atheiman]

I'd be happy to submit the pr to fix this, but my Python is a bit rusty and I have no idea where in this repo the logic for this is.

[kyleknap]

Interesting. I can see the use case for this, but unfortunately to implement it would be a breaking change to --cli-input-json. Interpreting the docs you posted, if you provide --parameters it will just clobber the "Parameters:" key in your json file for --cli-input-json. To start merging these changes into the list instead of clobbering them may cause issues for existing users.

Otherwise, I am not entirely sure how you would do this via the AWS CLI. You would probably need some script that handles the merging for you in the final --cli-input-json parameter so you do not keep the sensitive material in source code.

I am not sure if there is much we can do from our side until the next major version bump of the AWS CLI. Let us know if you have anymore questions/comments.

[atheiman]

Yea I guess if I reread the doc I posted it is more clear than how I originally read it.

Could perhaps a new option be added like --extra-parameters which merges over "Parameters" found in the input json? There could be --extra-<any-top-level-key> for merging over the Key in the input json.

I really think there is a use case for this functionality in the CLI... In the org I work in we have a script that does this - it merges --parameters over --cli-input-json, and passes the merged hash / dictionary to the api via the Ruby SDK.

[kyleknap]

I think something along those lines could make sense. It may be tricky to implement especially for detecting mergeable parameters and ensuring there is no clobbering of the parameters created directly from the API. I will switch it back to just a feature request label.

[reavon]

I would like to see something along these lines as well. I would like to override (not merge) parameters already set in --cli-input file with what is in the --parameters that I list on the CLI.

[atheiman]

@reavon if I'm understanding you correct, that is part of the functionality I was trying to describe. In the description I showed:

1. { "ParameterKey": "B", "ParameterValue": "b" } in the input.json
2. ParameterKey=B,ParameterValue=B in the --parameters option (overriding the input.json)
3. resulting in { "ParameterKey": "B", "ParameterValue": "B" } being submitted to the CF api

If you wanted something different I'd like to hear an explanation of it because I'm curious of the use case. Just want to make sure we're on the same page.

[reavon]

@atheiman we are on the same page. That is exactly my use case.

[cervantek]

I think this would be a great feature as well. I would prefer any --parameters specified to override the input-json parameters. I don't see how this would break anyone that is currently using either --parameters or input-json parameters as attempting to do this currently results in errors unless all needed parameters are specified on the command line.

[tiadobatima]

Yep. This is a super handy feature.
But also mixing and matching file and cli provided params (cli wins) would be even nicer. Ie:

# cat my-params.json 
{  "ParameterKey": "A", "ParameterValue": "a", "ParameterKey": "B", "ParameterValue": "b" }

# aws cloudformation create-stack \
    --stack-name some-stack \
    --template-body file://some-template.json \
    --parameters file://my-params.json ParameterKey=B,ParameterValue=B 
           ParameterKey=C,ParameterValue=c

Resulting in these final params:

{  "ParameterKey": "A", "ParameterValue": "a", "ParameterKey": "B", "ParameterValue": "B",  "ParameterKey": "C", "ParameterValue": "c",}

And I agree with @johnthomasmorton above... Nobody could be using --parameters this way now, so I imagine nothing would break for existing users.

Thanks! :)

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[rupertbg]

18 months later and still no progress?

[gene1wood]

Here's a workaround to this missing feature

aws cloudformation create-stack --stack-name 'MyStack' \
  --template-body "file://$PWD/template.json" \
  --parameters $(python -c 'import json,sys;f=open(sys.argv[1]);print(" ".join([",".join(["%s=\\\"%s\\\""%(k,v) for k,v in x.items()]) for x in json.load(f)]));f.close()' input.json) \
  'ParameterKey=B,ParameterValue=B' \
  'ParameterKey=C,ParameterValue=C'

This takes the parameter JSON file, input.json in this example, parses it and outputs each parameter as a --parameters argument on the command line. This way aws cli will use all of the parameters passed.

This doesn't do anything in regards to overriding one passed value with another, as the end result is the same as if two parameters with the same ParameterKey were passed as command line arguments. This solves for wanting to combine parameters in a json file and on the command line.

Some explanations for why the format looks this way :

• The value of each key is quoted like this (\\\"%s\\\") is to deal with cases where the value is a comma delimited list, like a list of subnets. This is explained here. The triple slashes resolve to \" surrounding the value.
• The JSON is converted into "aws cli shorthand syntax" because aws cli won't accept
  
  
  
  • a file:///input.json parameters argument combined with shorthand syntax
  
  
  • raw JSON combined with shorthand syntax
  
  

[dattran92]

Looking forward for this feature to be implemented.

I need it to inject new env when updating the stack. I'm working around by adding all the keys in the file into the parameter list which the UsePreviousValue=true to work around. Something like this:

aws cloudformation update-stack --stack-name 'MyStack' \
  --template-body "file://$PWD/template.json" \
  'ParameterKey=Exitsted,UsePreviousValue=true \
  'ParameterKey=C,ParameterValue=NewValue'

[ernestm]

I could use this too. I have parameters I want to be stable and in source control and parameters I want to specify at build time (e.g. environment) and not having this means "parse your parameter json yourself" is the go-to option.

[ThomasStubbe]

I would really love this possibility as well

[RichardForshaw]

Adding my 2c.

I think this would be a good feature. I have rolled out a few CD pipelines (using CodePipeline), which can be nicely parameterised (and turned into a pattern). But you don't want to put things like GitHub OAuth keys in the parameter file.

So a solution where I can parameterise most of my CD-pipeline pattern in a parameter file, and pass it to the CLI _together with_ specifying the OAuth key(s) separately so that it is combined with the parameters in the parameter file would be nice.

I realise that you can work around this with using Secrets Manager or Parameter Store, but this makes the cloudformation much more complicated for something that is probably only going to be set up once.

[rupertbg]

@RichardForshaw - I agree. I often want to supply my templates to others but the "setup up a Secrets Manager secret yourself" part is a bit cumbersome.. at best you can provide a CF for the Secret creation and then run a CLI command to replace the value of the secret.

Unfortunately, putting things like OAuth tokens in to the Cloudformation parameters via the CLI would still expose the values in the parameters in CF... NoEcho helps here though.

[tmillner]

+1

[danielwhatmuff]

+1