Aws-cli: cloudformation deploy --parameter-overrides parses parameters with spaces in a wrong way

[seeebiii]

Description:
I want to hand over parameters to the deployment of my cloudformation template using cloudformation deploy --parameter-overrides. Some parameters might contain spaces which are parsed in a wrong way. Even by using escape characters an error is thrown.

For example: cloudformation deploy [...] --parameter-overrides Foo=B a r or Foo=B\ a\ r or Foo="B a r" do not work.

Expected Behaviour:
The parameters are parsed as they would not contain any space, e.g. like Foo=Bar.

Actual Behaviour:
An error is thrown: ['a'] value passed to --parameter-overrides must be of format Key=Value

Debug infos:

• aws --version returns: aws-cli/1.15.0 Python/3.6.5 Darwin/17.4.0 botocore/1.10.0

Here is the --debug output (I have removed some params as I had no time to prepare a simple example yet).

2018-04-19 11:26:00,017 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.15.0 Python/3.6.5 Darwin/17.4.0 botocore/1.10.0
2018-04-19 11:26:00,018 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['cloudformation', 'deploy', '--debug', '--template-file', 'cfn.packaged.yml', 'Foo=B', 'a', 'r']

...

2018-04-19 11:26:00,063 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 208, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/clidriver.py", line 345, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/customizations/cloudformation/deploy.py", line 259, in _run_main
    self.PARAMETER_OVERRIDE_CMD)
  File "/usr/local/Cellar/awscli/1.15.0/libexec/lib/python3.6/site-packages/awscli/customizations/cloudformation/deploy.py", line 381, in parse_key_value_arg
    value=key_value_pair)
awscli.customizations.cloudformation.exceptions.InvalidKeyValuePairArgumentError: ['a'] value passed to --parameter-overrides must be of format Key=Value
2018-04-19 11:26:00,066 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

If you need more information to reproduce it, let me know.

[dstufft]

This is going to come down to the rules of your shell, you can see what values the CLI thinks it recieved from the shell on the second line of your ouput above.

What I would suggest trying is:

$ cloudformation deploy [...] --parameter-overrides "Foo=B a r"

Notice that the " is quoting the Entire Key=Value and not just the value. In most shells that will have it pass that entire string in as one argument to the CLI.

[seeebiii]

@dstufft Indeed! Sorry, I didn't thought about quoting the whole key value pair and also searched a lot for docs about it, but could not find them.. Thanks for the hint! I'm closing it ;-)

[ispyinternet]

Having problems with this when trying to parse a json file with values that have whitespace

E.G:

$ array=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "'\''\(.[0])=\(.[1])'\''"' parameters.json)
$ aws cloudformation deploy --template-file deploy.yaml --parameter-overrides "${array}" --stack-name TEST

$ echo "${array}"
//
    'param1=val1'
    'param2=value with space'
//

$ [u'with'] value passed to --parameter-overrides must be of format ParameterKey=ParameterValue

This isn't the only iteration of my efforts. I have tried many combinations / approaches all in vain.

I can't for the life of me get bash or aws cli to keep the ParameterKey=ParameterValue parts quoted.

Very frustrating as, if you want to parse a large parameter file programmatically it appears impossible.

Can anybody offer a solution?

[ispyinternet]

Ok, should anybody stumble on this and want a solution, this is what worked for me:

// package.json...

.
scripts: {
   "deploy": "array=( \"aws\" \"cloudformation\" \"deploy\" \"--template-file\" \"deploy.yaml\" \"--capabilities\" \"CAPABILITY_NAMED_IAM\" \"--parameter-overrides\" \"$(jq -r '.[] | [.ParameterKey, .ParameterValue] | \"'\\''\\(.[0])=\\(.[1])'\\'' \"' parameters.json)\" \"--stack-name\") && eval $(echo ${array[@]} $*)"
}
.
// usage:
$ npm run deploy MY_STACK_NAME

handles parameters with spaces.

[PiyushMittl]

@ispyinternet thanks for your solution.
i am using shell and have used below command:
$ array=$(jq -r '.[] | [.ParameterKey, .ParameterValue] | "'''(.[0])=(.[1])'''"' parameters.json)
$ aws cloudformation deploy --template-file deploy.yaml --parameter-overrides "${array}" --stack-name TEST

and getting same error.
you have given solution in which using npm. is there any way to achieve the same in shell.

[ispyinternet]

I think in your case you need to do something like:

$ array=(aws cloudformation deploy --template-file deploy.yaml --parameter-overrides $(jq -r '.[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\(.[1])"' parameters.json) --stack-name TEST)
$ eval $(echo ${array[@]})

You could probably 1 line that, but Im not sure off the top of my head

[cmaxwellau]

Try this first:

export IFS=$'\n'

It will stop bash from expanding on anything other than a line return.

4055

[default50]

This worked for me:

#!/bin/bash

STACK_NAME="mystack"
TEMPLATE="mytemplate.yaml"
PARAMETERS_FILE="parameters.json"
PARAMS=($(jq -r '.Parameters[] | [.ParameterKey, .ParameterValue] | "\(.[0])=\(.[1])"' ${PARAMETERS_FILE}))

aws cloudformation deploy \
    --template-file "${TEMPLATE}" \
    --stack-name "${STACK_NAME}" \
    --parameter-overrides ${PARAMS[@]}

With parameters.json having this structure:

{
  "Parameters": [
    {
      "ParameterKey": "MyParameter",
      "ParameterValue": "SomeValue"
    }
 ]
}

[szmulder]

{
  "Parameters": [
    {
      "ParameterKey": "MyParameter",
      "ParameterValue": "SomeValue"
    }
 ]
}

but when the value has space it will not work{
"Parameters": [
{
"ParameterKey": "MyParameter",
"ParameterValue": "Some Value ha space"
}
]
}

[Fectorez]

--parameter-overrides "key=my\ value"
Works on PowerShell

[Ochaun]

Reopen this. I'm having the same problem even when quoting the key value pair.
"Key=Value" "Key=Value with space"

[elreydetoda]

so, instead of escaping spaces and quotes there is a built in function for bash called mapfile which will handle parsing of the jq command properly and quote things when necessary.

here is an example on my blog where we got it working for my work: https://blog.elreydetoda.site/cool-shell-tricks/#bashawscloudformationparameteroverrides specifically these are the commands, but checkout the blog to know what they do.

# pull secrets from AWS secrets manager service by the name
infra_params=$( aws secretsmanager get-secret-value --secret-id "${secret_name}" )

# parse our secrets format (json) and add it to paramz array
mapfile -t paramz < <(jq -r '.SecretString' <<< "${infra_params}" | jq -r '.params[] | "\(.ParameterKey)=\(.ParameterValue)"')

# deploy cloudformation stack w/expanding the paramz array
aws cloudformation deploy --stack-name "${stack_name}" --template <template_name> --parameter-overrides "${paramz[@]}"

let me know if you have anymore questions around the commands (probably comment on my blog to not take over this issue)