Aws-cli: Auto-fetching URL's in KMS encrypt command

[evansolomon]

When I run something like aws kms encrypt --key-id my-key --plaintext "http://example.com", the url (example.com) is fetched and its response body is encrypted. This seems like a very odd behavior. I'm guessing it's probably related to the special case handling for fileb:// stuff.

I'm wondering if it's intended, and if so whether there's a preferred way to encrypt things that look like URL's? I'd personally argue that it should not be intended because it seems very confusing, particularly for an argument called "plaintext".

[rayluo]

Good point. We can fix that.

To be precise, it is not a bug, it is a feature to support loading data from a location with either "file://", "fileb://", "http://" and "https://" as prefix.

We already noticed the occasional confusion and record it as a known issue. We are still trying to find a better solution. Let's track this in that linked issue.

Before that, a workaround would be to put your content http://example.com into a local file, and then call the CLI by something like:

aws kms encrypt --key-id my-key --plaintext fileb:///path/to/my/data.txt

Closing this one as duplicated feature request.

[skwp]

Just ran into this, know this is a closed issue but it's incredibly confusing. This "feature" is very magical.

[skwp]

I would have expected it to be something like --fetch "url" rather than --plaintext auto-fetching

[AlJohri]

^ @skwp agreed

[nkayurov-newforma]

This is an issue, not a feature. There has to be a reserved key word for a value which would allow to avoid such behaviour. The suggested "workaround" is not really practical for automation.

[mipearson]

Can this please be classified instead as a "bug we can't fix" rather than a "feature"?

"It's a feature, not a bug" is dismissive of the confusion this causes.

[commander]

Even I am facing the same issue. Rather than saying this is a feature, AWS team should really provide another workaround to this as if you search on google, almost everyone who use parameter store has faced the same issue. so this is not the feature that everyone wants.
Possible solution:
Solution 1. add another datatype called URL, apart from supported datatype of String, SecureString etc,.
Solution 2. add a flag --nofollow to denote that URL itself is value rather than get the value by following the url

[rosswilson]

This has now been fixed. A new configuration option called cli_follow_urlparam can be set to disable this auto-fetch URL behaviour across the entire AWS CLI.

In your ~/.aws/config file, add the following line:

cli_follow_urlparam = false

See the documentation for more details.

I've cross-posted this comment from https://github.com/aws/aws-cli/issues/630#issuecomment-472832254 too.

[bisdavid]

This setting is also documented in the CLI User's Guide at https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-global

[jmervine]

Simplest solution is run

aws configure set cli_follow_urlparam false

[LorensK]

Simplest solution is run

aws configure set cli_follow_urlparam false

This disables the "feature" for all aws commands until you revert (it's stored in your user settings and it is per-profile).

If you wish to avoid changes to your configuration that may introduce inconsistencies between profiles or impact scripts you may be running that make use of this "feature", but you also wish to avoid the hassle of creating and deleting a temporary file for each command, you can (as suggested in a comment on #630) use bash process substitution to transparently and temporarily create something that the aws command sees as a file containing the data:

aws kms encrypt --key-id my-key --plaintext fileb://<(echo -n 'http://example.com')

[LorensK]

Note that this behavior is removed across the board in AWS CLI v2: https://github.com/aws/aws-cli/pull/3590/commits/b8f2d1a697025cf4edcc104fdf3f80c2dcd7dc6e

Only file:// and fileb:// remain special prefixes.

People who were using this behavior (hopefully in cases other than KMS secrets!) can for example (if using some common Unix shells) replace https://urlwithmydata with "$(curl -f https://urlwithmydata)". The -f is important to detect when the URL is inaccessible; the curl exit code is unfortunately not taken into account but kms will complain that the secret is empty.

For the more general case using v2, DATA="$(curl -f https://urlwithmydata)" && aws somecommand --someargument "$DATA" will provide some error detection in case the curl request fails.

HTH

[rushikeshkalyani]

Awesome...!!
This helped me a lot, Thanks @rosswilson !!

cli_follow_urlparam = false

https://github.com/aws/aws-cli/issues/1475#issuecomment-472832509