Boto3: method generate_url (get temporary url)

[vlcinsky]

With boto library (not boto3), there is method generate_url which generates temporary url for given object.
This method is availabe on connection, bucket and key objects.

import boto
con = boto.connect_s3()
tmpurl_c = con.generate_url(...)
bucket = con.get_bucket("mybucket")
tmpurl_b = bucket.generate_url(...)
key = bucket.get_object("key/name.txt")
tmpurl_k = key.generate_url(...)

I was unable to find corresponding method in boto3. Is there such a method? Or is it planned?

I am aware, that actually creating the temporary url is not a call to AWS web service (so I do not expect it to be found in AWS serices model), but for practical use it is very important.

Fortunatelly, today we may se boto to do this work, but if boto3 is supposed to be complete replacement to today boto, ability to generate temporary url seems to be important.

[kyleknap]

@vlcinsky

Yeah that is one feature that we have identified as necessary for boto3. We actually do have it implemented. It is implemented in botocore, which is a low level library that boto3 uses. It is just not merged into the develop branch yet: https://github.com/boto/botocore/tree/clients-only

If you want to to try it out, I would pull down that branch. To make the call, you can do something like this:

import botocore.session

session = botocore.session.get_session()
client = session.create_client('s3')

 presigned_url = client.generate_presigned_url(
    'get_object', Params={'Bucket': self.bucket_name, 'Key': self.key})

Let me know if you have any questions. This feature will be available in boto3 once that branch gets merged into the develop branch of botocore.

[vlcinsky]

@kyleknap Perfect, as it is on roadmap (and it looks like getting close), it make me feel safer.

Anyway, I did try to use existing solution (using botocore branch clients-only) and see, that the expiration can be currently specified only in relative value - how many seconds from now it shall require.

It would be very convenient, if it would also allow specifying absolute time. I will add this request to botocore.

[kyleknap]

Noted. We will look into this. Will mark this as a feature request.

[dannguyen]

A couple of questions:

One: is there anyway to generate plain HTTP URL? Thie following command will always create an expiring https get_object URL, even if object is set to public-read:

client.generate_presigned_url('get_object', Params = {'Bucket': 'www.mybucket.com', 'Key': 'hello.txt'}, ExpiresIn = 0)
'https://s3.amazonaws.com/www.mybucket.com/hello.txt?Signature=BLAHBLAHBLAH%3D&AWSAccessKeyId=MYKEY&Expires=1435013699'

Two: Is this kind of thing not best practice? Other S3 clients I've used (which I assume used the previous AWS API) seem to just generate the HTTP URL without any problem...and now that I've tried using the API through boto3...this feature seems _extremely_ buried (I read the docs and source code for about an hour until stumbling upon this issue)., to the point where it seems like spitting out a HTTP URL from a S3 object is not recommended practice?

[vlcinsky]

@dannguyen do not forget, that having public-read might not be always a constant feature of an object.

In past I was a bit surprised, when I have realized, that the temporary url can be generated even without having connectivity to related object.

To me the lesson are: when talking about temporary url:

• it is temporary url and it shall have an expiration regardless of current or future ACL configuration
• it is client's decision (which can be done without even looking to the object, could be about object which even does not exist at the moment...) how is such url created
• generating the url without touching relevant object is fine, in fact it allows nice efficiency (no network communication).

[cilia]

For S3 pre-signed URLs, I found this: https://boto3.readthedocs.org/en/latest/reference/services/s3.html#S3.Client.generate_presigned_url, and for CF pre-signed URLs, I found: https://boto3.readthedocs.org/en/latest/reference/services/cloudfront.html#CloudFront.Client.generate_presigned_url. When will they become available in a stable release of boto3?

[jamesls]

Both of those methods are available in the latest version of boto3. Let us know if there's any other questions.

[niallatbynder]

Hi there,

I've read the docs but I'm struggling to understand how to actually generate a pre-signed url to allow temporary access to download contents from a s3 bucket.

Any help? Thanks

[faerloev]

Mjello,

I'm in the same boat as @niallatbynder ; I can't seem to figure out how generate a signed url.

CloudFront.Client.generate_presigned_url() takes a ClientMethod argument, but fails to describe what this ClientMethod can be. From my trails I get the impression, that ClientMethod should be a method on CloudFront.Client, is that correctly understood? In that case, I can't find any methods on CloudFront.Client that fetches a file from a bucket, hence no methods to generate a signed url for.

An example of how to use this would be greatly appreciated.

Thanks in advance.

[radAragon]

Faerloev,

The ClientMethod param is not clearly described, but it just means a Client method like the "get_object" (http://boto3.readthedocs.org/en/latest/reference/services/s3.html?highlight=s3%20url#client). Here's my example:

        client = boto3.client('s3')
        bucket_name = 'api_reports'

        resp = client.put_object(
                                Bucket= bucket_name, 
                                Key= filename, 
                                Body= file,
                                Expires= datetime.utcnow() + timedelta(days=7), )
        print(resp)
        url = client.generate_presigned_url(
                                            'get_object', 
                                            Params = { 
                                                      'Bucket': bucket_name, 
                                                      'Key': filename, }, 
                                            ExpiresIn = 86400, )
        print(url)

[cilia]

@radhomero, thanks for pointing it out. I look through the methods listed under "These are the available methods:" under Client section, and I wonder what is the difference between download_file and get_object, and similarly, upload_file and put_object?

[radAragon]

@cilia See http://boto3.readthedocs.org/en/latest/reference/customizations/s3.html for the differences. Upload and Download_file are abstractions in high level to actually download or upload files from local folders.

[faerloev]

@radhomero My apologies for not being more specific. I'm talking about the CloudFront client. Not the S3 client.

As far I can tell, the CloudFront client is missing methods for downloading data (eg. get_object).

For now, I have overcome this by writing my own signing method based on an old Stackoverflow answer.

[xuchen81]

@faerloev how did you generate_presigned_url for cloudfront, has the missing get_object been fixed?

Thanks

[camilonova]

@radhomero Thank you :beers:

[faerloev]

@cx44yale My sincere apologies for not responding sooner.

I hope you managed to sign your urls, but if not (and for the sake of others ending up here) here's a hint:

I based my solution on the accepted answer from this question on Stackoverflow which pretty much works out of the box. I exchanged M2Crypto for rsa to get a pure Python implmentation.

[swarajban]

@radhomero +1, exactly what I needed. I think this should be documented in the boto3.S3 documentation