Aws-cli: --parameters URI of an s3 bucket does not authenticate, expected to behave like --template-url

[bbrummer]

Uploading both my cloud formation template and a parameters json file to the same S3 bucket, I'm not able to reference the parameters file. It doesn't appear the CLI authenticates the S3 call for the parameters file, while it does for the template?

$ aws cloudformation create-stack --stack-name Test --template-url https://s3.amazonaws.com/mybucket/cf_template.json --parameters https://s3.amazonaws.com/mybucket/my_params.json

Error parsing parameter '--parameters': Unable to retrieve https://s3.amazonaws.com/mybucket/my_params.json: received non 200 status code of 403

$ aws --version
aws-cli/1.10.51 Python/2.7.10 CYGWIN_NT-10.0/2.5.1(0.297/5/3) botocore/1.4.41

[kyleknap]

Looks like you are getting a 403 for this. Are you sure you have unauthenticated read access to the my_params.json object? We will not use your aws credentials when retrieving from an s3 bucket.

[jamesls]

Following up on this. The CLI has a general feature, where, if you specify a paramter value that starts with a special prefix, the CLI will take certain steps to retrieve the actual value used:

• file://<path> - Read a file from disk and use the contents of the file as the value
• http[s]://<path> - Make an http/https request and use the response body as the param value

However, there are certain parameters, such as --template-url, where we want the literal string https://... as the parameter value. The CLI special cases these parameters to not retrieve the HTTP request, --template-url being one of them. This is why you're seeing different behavior between --parameters and --template-url.

Hope that helps, and if you still have questions feel free to stop by our gitter channel to chat.

[bbrummer]

That's just it, I wouldn't want unauthenticated access to my parameters object.

--template-url uses my current AWS credentials to access the S3 bucket my template is in, no need to be unauthenticated. The expectation is that --parameters when operation on a similar remote resource would behave the same. Especially considering a parameters file is more likely than a template to contain sensitive information such as raw credentials.

I understand _why_ the behavior is different, I just disagree that it _should_ be different. ;)

[DoubleDensity]

why was this closed? I agree with @bbrummer and don't understand the logic here. are we to understand that the parameters file can only be retrieved from S3 when authentication is disabled? why would we want Everyone access for something with sensitive details like our CloudFormation parameters?

[estahn]

@jamesls Why was this closed? Would it make sense to introduce s3://?

[fleeco]

+1 on having these require authentication. Was banging on the CLI wondering why it wouldn't work, then made it world read and everything worked. Mainly was confused why this would ever be the default behavior - having this be treated like template-url would be ideal :)

[lorengordon]

If I understand what @jamesls is saying, the value of --template-url is passed literally as a string to the CloudFormation API, and CloudFormation itself retrieves the template from S3. CloudFormation does not support that feature for a parameters file, so in that case the awscli is doing the work to retrieve the json file and convert it to a structure supported by the CloudFormation API. Entirely different mechanisms.

To support the feature being requested, I am seeing two options:

1. Submit a feature request to the CloudFormation team to add support for retrieving a parameters json file from an S3 bucket. The awscli team could then also special-case --parameters in the same way as --template-url.
2. The awscli team could add a general feature where the cli supports a parameter value specified in the form s3://<path> and attempts to retrieve the file from s3 and use the contents as the value. This would be handy in the general use case, but slightly suboptimal for this specific use case, because then you would specify --template-url as https://<path> (to take advantage of the larger file size supported when CloudFormation itself retrieves the template from S3) and --parameters as s3://<path>.

[abtheninja]

i couldn't yet make it work, what needs to be done to be able to pass the parameters ?
aws cloudformation create-stack --stack-name my-new-stack --template-url https://s3-us-west-2.amazonaws.com/path/to/the/template --capabilities CAPABILITY_NAMED_IAM --region us-west-2 --parameters https://s3-us-west-2.amazonaws.com/path/to/the/json

Error parsing parameter '--parameters': Unable to retrieve https://s3-us-west-2.amazonaws.com/omf-wfm-artifacts/wfmcftemplate/wfm.json: received non 200 status code of 403

[justnance]

Reopening issue.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[justnance]

All - This issue is being resolve with the release of CLI v 2.

[austindimmer]

Running into this issue today. Is there a timeline for CLI v 2 release?

I have tried

pip install -e git://github.com/aws/aws-cli.git@v2#egg=awscli

(virtualenv) PS C:\Dev\LB\cloudformation> aws --version
aws-cli/2.0.0dev0 Python/3.7.0 Windows/10 botocore/1.12.38

When I installed CLI v2 I observed an error in my virtualenv

pip3 : awscli 2.0.0.dev0 has requirement botocore==1.12.38, but you'll have
botocore 1.12.45 which is incompatible.
At line:1 char:1

• pip3 install boto3
• ~~~~~~
  
  
  
  • CategoryInfo : NotSpecified: (awscli 2.0.0.de...s incompatible.:St
    
    ring) [], RemoteException
  
  
  • FullyQualifiedErrorId : NativeCommandError
  
  

The error I observe when trying to pass a parameters file located on S3 when using CLI v2 is the following,
Error parsing parameter '--parameters': Expected: '=', received: ':' for input:
https://s3-eu-west-1.amazonaws.com/s3bucket/RedShiftPoc/Parameters.json