Aws-cli: Support generation of signed URL's for S3 access

[garnaat]

[jamesls]

Yeah I think this would be a great feature. I'm thinking this would be a new s3 subcommand? I've been working on some internal changes to the s3 command code to make it easier to add subcommands. I've been using this to start to refactor the ls command to fix several of the reported issues with ls.

Not entirely done yet, but here's what I have so far:
https://github.com/jamesls/aws-cli/compare/s3-ls-permissions#diff-b88a66f4bd148577a9390cb980d7eeb9R321

[vlcinsky]

Yes, this I was looking for. Please, allow to specify expiration not only by duration, but also by a specific datetime. Requiring UTC timezone, ISO format, full seconds, the string ending with "Z" seems to me well usable (e.g. "2014-05-30T00:00:00Z").

[vlcinsky]

In case, you really need to generate tmpurl from command line, I could recommend my command line tool s3tmpgen

Anyway, I would really invite such functionality in AWS CLI, which became valuable tool for my daily work with AWS S3.

[eterps]

+1

[johnboxall]

+1

I'm looking to use S3 to host code bundles to deployed to Heroku and this would be an awesome feature :)

https://blog.heroku.com/archives/2014/5/22/introducing_the_app_json_application_manifest

[leobessa]

+1

[modius]

+1 would be great to have this in AWS CLI

[johnboxall]

Just a note for anyone who ends up here, you can do this yourself trivially by reaching down into the boto library: https://boto.readthedocs.org/en/latest/ref/s3.html#boto.s3.key.Key.generate_url

[jamesls]

Thanks for the feedback. I'll take a look at this, I agree that this would be a great feature to add to the AWS CLI.

[davidski]

Thanks, @johnboxall. Boto is certainly an option, though for folks deploying the cli binary without native Python deployments (think Windows users), going that route is just as much work as pulling down the PowerShell SDK and doing things via the .NET SDK. Would still like to see this native in the CLI. :grin:

[isleshocky77]

Just for anyone looking to use boto directly for now until this is added to aws-cli I figured I would add a quick sample instruction. I've now went looking for this a couple of times and would rather have it along with this ticket and @johnboxall comment. This may be extremely obvious for some, but not for non-python devs

On a box which already has python installed

$ python --version
Python 2.7.6
$ sudo pip install boto
$ python
>>> import boto
>>> s3 = boto.connect_s3()
>>> bucket = s3.get_bucket('your-bucket-name')
>>> key = bucket.get_key('the-prefix/the-name-of-the-object.mp4')
>>> key.generate_url(3600)
'https://your-bucket-name.s3.amazonaws.com/the-prefix/the-name-of-the-object.mp4?Signature=CgDfFa45DBXFiMfASxSTpiSuHKM%3D&Expires=1415913273&AWSAccessKeyId=ABCDEDKSY344ACVDG'

[joech4n]

Building on @isleshocky77's example... adding arguments and minor error checking:

Script

$ cat boto-get-signed-url.py
#!/usr/bin/python
import boto
import argparse

parser = argparse.ArgumentParser(description='Generate an S3 signed URL')
parser.add_argument('-b', '--bucket', help='bucket name')
parser.add_argument('-k', '--key', help='prefix/key')
parser.add_argument('-s', '--seconds', type=int, help='time in seconds until the URL will expire')
args = parser.parse_args()

s3 = boto.connect_s3()
bucket = s3.get_bucket(args.bucket)
key = bucket.get_key(args.key)
if bucket.get_key(args.key):
  print key.generate_url(args.seconds)
else:
  print 's3://' + args.bucket + '/' + args.key + ' does not exist'

Example usage and help

$ ./boto-get-signed-url.py -b superbucket -k "test" -s 60
https://superbucket.s3.amazonaws.com/test?Signature=n6cO8RH%2FbNwQhuZVNNazo3q04x0%3D&Expires=1416695021&AWSAccessKeyId=AKIEXAMPLEKEYNOTREAL

$ boto-get-signed-url.py --help
usage: boto-get-signed-url.py [-h] [-b BUCKET] [-k KEY] [-s SECONDS]

Generate an S3 signed URL

optional arguments:
  -h, --help            show this help message and exit
  -b BUCKET, --bucket BUCKET
                        bucket name
  -k KEY, --key KEY     prefix/key
  -s SECONDS, --seconds SECONDS
                        time in seconds until the URL will expire

[vlcinsky]

Hi all
After another check about status of this issue I made an update to Python package ttr.aws.utils.s3 providing the tool s3tmpgen. The update allows to generate the url not only using https, but (with option an -http) also http.

Still looking forward to replace this with awscli solution.

[farrellit]

:+1:

[sukrit007]

+1 for s3 signed url to be part of cli

[tableware]

:+1:

[memory]

+1

[vlcinsky]

I think, I understand, why it takes so long time: things must happen in order and good thing is, it looks like being on the roadmap.

As AWS CLI is based on botocore, it shall be resolved there first. boto/botocore#291 is asking for it.

Pull request boto/botocore#504 was already merged into branch clients-only and roadmap (mentioned on README at https://github.com/boto/botocore ) claims, botocore is currently just one step before merging the branch clients-only into develop (then beta, GA and AWS CLI has "native" tool to provide give functionality for us).

[kyleknap]

@vlcinsky

Yep once the clients-only branch gets merged into botocore, we will be able to pick up the generation of signed url's in the CLI. Then, the main amount of work that would have to be done on the CLI side is to create a good API that exposes the feature.

[yermulnik]

https://github.com/boto/botocore/issues/291 was already closed on 23 Jun: "The feature is currently available in botocore."

[bzz]

:+1: would be a handy feature indeed

[pcolazurdo]

+1

[vlcinsky]

As @kyleknap noted, one of things to do is to design good API for this feature.

I see following use cases:

• "tmpGET": create tmpurl for GET request
• "tmpPOST": create tmpurl for POST request
• "tmp???": is there any more method to support? I am not aware of (and have never used other than
  GET and POST)

The "tmpGET" is really easy as the only output is url and the only input is bucket/key and expiration
date (expiration time).

The "tmpPOST" is much more complex as one has to define a policy for posting.

One question to resolve is, where to put these actions, there seem to be two alternatives:

• aws s3
• aws s3api

I will not deal today with the more complex "tmpPOST" and will focus on simpler "tmpGET"

Quick solution: add aws s3 tmpurl

Such a solution would be rather easy to implement, would help in 80% of real
use situations (publish an object to AWS S3 and provide temporary url to share
with someone).

The concept would build on existing aws s3 ls with following differences:

• add an option --expires to define date and time at which the url expires or --expires-in to
  specify number of seconds till the expiration
• would output only tmpurls, one line per listed object

I am afraid, there is currently no easy way to create tmp url for exact moment in time, so initial
version would offer only the --expires-in option with default value of 3600 seconds.

Pros and cons

Pros

It is very easy to create bunch of tmp urls for number of existing objects on AWS S3. It saves the
hurdle with getting exact bucket/key name values.

Cons

Disadvantage of this aws s3 ls based approach is, one cannot create tmp url for an object, which
does not exist yet.

Anyway, this could be resolved later on by aws s3api solution.

[farrellit]

Very nicely put there. Whereas a post is something that would almost certainly be built around some server side handling of the posted data, the get method use cases are usually quite a lot easier. When you say that would cover most use cases, I would strongly agree.

[farrellit]

I meant to add though, I don't think the temp URL should have anything to do with locating the files on s3. If somebody wants to generate entire sets of signed URLs, that's logic the cli doesn't need to do more to wrap. The tmpurl primitive is the minimum viable product in my mind .

[quiver]

I've implemented(not fully) pre-sgined URL for s3 objects in my local branch.

https://github.com/quiver/aws-cli/tree/s3-presigned-url

This is a thin-wrapper for botocore.generate_presigned_url, and it's not production ready yet. As @vlcinsky noted, we need a good API design as AWSCLI. Once that's fixed I can implement that.

One thing I note is that generate_presigned_url requires client_method parameter, which specifies S3 API(e.g. get_object) to sign for. In my current implementation, it's exposed to the user, but it would be better to be user-friendly, like providing different sub-commands(aws s3 geturl, aws s3 uploadurl) or switch options(--type upload)

Usage

upload objects to S3

$ echo hello world > test.txt

$ aws s3 url s3://BUCKET/test.txt --client-method put_object --expires-in 180
https://BUCKET.s3.amazonaws.com/test.txt?AWSAccessKeyId=AKIAIXXXXXXXXXXXXXXX&Expires=1451449621&Signature=KgwO9lBx942fFvln0JW0NX7mKS0%3D

$ URL=`aws s3 url s3://BUCKET/test.txt --client-method put_object --expires-in 180`

$ curl -D - -X PUT --upload-file test.txt $URL
HTTP/1.1 100 Continue

HTTP/1.1 200 OK
x-amz-id-2: /90B1axPysBg3P8kv8BlR8RoqdO1JfajCN5BM5/TxIT3VjGphKmyGX8EgCQEtCXYhuNkVne5+GM=
x-amz-request-id: 685F03CA6C84FAC0
Date: Wed, 30 Dec 2015 05:18:38 GMT
ETag: "6f5902ac237024bdd0c176cb93063dc4"
Content-Length: 0
Server: AmazonS3

$ aws s3 cp s3://BUCKET/test.txt -
hello world

get objects from S3

$ URL=`aws s3 url s3://BUCKET/test.txt --client-method get_object --expires-in 180`

$ curl -D - -X GET $URL
HTTP/1.1 200 OK
x-amz-id-2: WuRokcBm9wnDMaRkD8kNeGijuKEzVp3eagi7JbpPXmmchEljsiP4wZX5w1TaeuK94n2526FGKMI=
x-amz-request-id: 1EBCAA7A691A577D
Date: Wed, 30 Dec 2015 05:20:14 GMT
Last-Modified: Wed, 30 Dec 2015 05:19:15 GMT
ETag: "6f5902ac237024bdd0c176cb93063dc4"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 12
Server: AmazonS3

hello world

[MrAnderson7]

The above python generates a url like: 'https://.s3.amazonaws.com/dir/dir/file

When I attempt to curl the file, I get SSL certificate problem: Invalid certificate chain
If I curl with option -k, I can get past this, but I also know that if the url looks like: 'https://s3-.amazonaws.com//dir/dir/file'
the cert is valid, is there a way to change the url, or some other fix?
Thanks,
Garry

eg: 'https://s3-us-west-1.amazonaws.com/bucket/dir/dir/file'

Update, using boto3 was able to generate the correctly signed url.

[fwisehc]

+1

[bzz]

Impressive - in 3 years official CLI tool did not get support for signed url implemented, allthough it exists in boto and s3cmd

[michael-wirth]

1+

[lafraia]

+1

[PsyTae]

+1

[jason-riddle]

Any update on this? Surprised this isn't available yet.

[knap1930]

+1

[gdbtek]

I wrote one up as a workaround and works as expected: https://github.com/gdbtek/aws-tools

[jamesls]

Hi everyone, thanks for the feedback. This something that is on our backlog. I don't have exact dates yet, but I'll link to this issue once we have a pull request up.

[yermulnik]

Great! Thanx!

[tommedema]

Is it possible to get a presigned url to an entire s3 folder? Including a web interface to navigate through the folder?

[vlcinsky]

@tommeda Not possible. pre-signed url is always related to single stored object. What you talk about is similar to static web site but to control access to it (if based on AWS S3), one has to write some proxy. Few attempts already exists, none seemed to me (researched about a year ago) easy peasy.

[farrellit]

You could always generate a webpage that provided an interface to and included pre signed urls for each object, and then put the interface into s3 and return a pre signed url to the interface. Not exactly easy peasy, and the interface would like be usage specific