Aws-cli: Ability to include external file contents in cli parameters json

[tpakula-fender]

For instance, to create a new CloudFormation stack, I would like to do something like

aws cloudformation create-stack --cli-input-json file://mystack.cliparameters

where the contents of mystack.cliparameters is

{ "TemplateBody": "file://mystack.cloudformationtemplate", "Parameters": [ { "ParameterKey": "UserDataParam", "ParameterValue": "file://userdata.sh", "UsePreviousValue": false } }

mystack.cloudformationtemplate then contains the actual CloudFormation template, and userdata.sh contains the script used to bootstrap the instance at launch time.

Currently, I have to do

aws cloudformation create-stack --cli-input-json file://mystack.cliparameters --template-body file://mystack.cloudformationtemplate

...and specify the userdata explicitly within mystack.cliparameters which can get very messy with newlines if the userdata script is more than just a few lines.

[jamesls]

There was a similar request a while back here: https://github.com/aws/aws-cli/issues/1097

In that scenario, they wanted to be able to specify binary contents as a value, which isn't possible in a JSON document. The most reasonable way to do this would be to reference a file in the JSON document.

We've explored several options, but the main constraint we're running into is that we'd need this to be backwards compatible.

The solution you've proposed could work. However, if there's existing users that want to specify the _literal_ string value of file://mystack.cloudformationtemplate and we change this behavior to now interpret this as mean "insert the contents of this file", this will be a breaking change.

Another solution proposed was to support something besides JSON. YAML would allow for custom tags (i.e !file), one of which could be a "file" tag. Another option would be to extend JSON to support something like:

{"foo": file:///path/to/file}     # Note there's no quotes around the file path
{"foo": File("/path/to/file")}

If we can come up with something that's backwards compatible, I think this is something we should do.
I'm interested to hear what others have to say.

[tpakula-fender]

Ahh, I see where my suggestion might break things. CloudFormation itself does something like the JSON extension you've proposed, so maybe similar to the following in the CLI parameters JSON:

{ "ParameterKey": "UserDataParam", "ParameterValue": { "Fn::ExtFile": "/path/to/file" } }

Bonus: the door is then open to adding other functions without breaking backwards compatibility by introducing new.

[jamesls]

Very interesting idea. If we only apply this for scalar values (strings, integers, booleans, floats), then it should be entirely backwards compatible. Previously the value had to be a string, so providing a dictionary type would cause a validation error, so all we would do now is pre-process the data, look for the {"Fn::ExtFile" ....} where a string type was previously used and replace that value with a string.

[hsupu]

Could this bug be fixed soon? It has exists for one year.

I found this issue from https://forums.aws.amazon.com/thread.jspa?threadID=225033

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[salmanwaheed]

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a temporary error. The following address(es) deferred:

[email protected]
Domain salmanwaheed.info has exceeded the max emails per hour (172/150 (114%)) allowed. Message will be reattempted later

------- This is a copy of the message, including all the headers. ------
Received: from github-smtp2-ext5.iad.github.net ([192.30.252.196]:60133 helo=github-smtp2a-ext-cp1-prd.iad.github.net)
by box1177.bluehost.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256)
(Exim 4.89_1)
(envelope-from [e-mail])
id 1ej0Qr-001bk7-It
for [email protected]; Tue, 06 Feb 2018 03:24:57 -0700
Date: Tue, 06 Feb 2018 02:24:43 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=github.com;
s=pf2014; t=1517912683;
bh=tqTIm5tCw5vaBvyleO8ZJxQTcNGh4HuAA3sOn2llpwA=;
h=From:Reply-To:To:Cc:In-Reply-To:References:Subject:List-ID:
List-Archive:List-Post:List-Unsubscribe:From;
b=hMpxoYpDt4cgnoBejHY3pY4mrkFoLmlyk6l+RGnKPtbFoGAfukLkXSmQCldlWz74g
qIlJIva5Xwv6n8/explqarzvDjmm1cWzyOtgGt9+pJ1BwxT2QiPU3wCMUE/oKC8g2l
+7AGVY6bd4EyC6lysaEJc2TwIi7SBm7AKJsmD304=
From: Andre Sayre [e-mail]
Reply-To: aws/aws-cli [e-mail]
To: aws/aws-cli [e-mail]
Cc: Subscribed [e-mail]
Message-ID:
In-Reply-To:
References:
Subject: Re: [aws/aws-cli] Ability to include external file contents in cli
parameters json (#1803)
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="--==_mimepart_5a79826bee30b_55d33fbb60110f3431538";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: list
X-GitHub-Sender: ASayre
X-GitHub-Recipient: salmanwaheed
X-GitHub-Reason: subscribed
List-ID: aws/aws-cli
List-Archive: https://github.com/aws/aws-cli
List-Post: [e-mail]
List-Unsubscribe: ,
https://github.com/notifications/unsubscribe/AO8bOGnceRxAIw5z9qu_loNRNDVQjaFlks5tSChrgaJpZM4HdY7p
X-Auto-Response-Suppress: All
X-GitHub-Recipient-Address: [email protected]
X-Spam-Status: No, score=0.5
X-Spam-Score: 5
X-Spam-Bar: /
X-Ham-Report: Spam detection software, running on the system "box1177.bluehost.com",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
root\@localhost for details.

Content preview: Closed #1803. -- You are receiving this because you are subscribed
to this thread. Reply to this email directly or view it on GitHub: https://github.com/aws/aws-cli/issues/1803#event-1459792366
Closed #1803. [...]

Content analysis details: (0.5 points, 5.0 required)

pts rule name description

---

0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked.
See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information.
[URIs: github.com]
-0.5 SPF_PASS SPF: sender matches SPF record
0.0 HTML_MESSAGE BODY: HTML included in message
0.7 HTML_IMAGE_ONLY_20 BODY: HTML: images with 1600-2000 bytes of words
-0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
2.5 DCC_CHECK No description available.
-0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
domain
0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid
-2.1 AWL AWL: Adjusted score from AWL reputation of From: address
X-Spam-Flag: NO

----==_mimepart_5a79826bee30b_55d33fbb60110f3431538
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 7bit

Closed #1803.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/aws/aws-cli/issues/1803#event-1459792366
----==_mimepart_5a79826bee30b_55d33fbb60110f3431538
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit

Closed #1803.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

----==_mimepart_5a79826bee30b_55d33fbb60110f3431538--