Aws-sdk-java-v2: Presigned URL Generators

[sorenbs]

I can't find a way to generate a presigned url for s3. Is this feature missing or am I missing something?

millems@ edit 10/28/19: This is now supported for S3's GetObject. See here.
millems@ edit 11/7/19: This is also now supported for S3's PutObject. Example here.

[dagnir]

@sorenbs You're correct, we don't currently have support for this in 2.0.

[dagnir]

Related to #181

[aaronanderson]

While we wait for an official implementation here is a workaround I am successfully using. Basically I plugged in a custom ExecutionInterceptor to create a presigned URL right before the request goes over the wire and then I added a mock SdkHttpClient that doesn't actually execute the request.

[cesartl]

This is a significant hindrance to switching to version 2 of the API.
Is there any update on this?

[millems]

It's on our roadmap, but we're unfortunately not able to comment on when it might be done. Bringing 2.x into parity with 1.11.x is a very high priority for our team.

[miere]

Hello maintainers,

It has been quite a long time since this issue have been opened. Do you have any thoughts on this? Are you willing to accept a PR for this feature? I mean, let me know if you have any plans to do so in the next couple of weeks, otherwise, I'd be glad to submit a PR.

I think this is a quite important feature and I'm afraid that, just as me, other developers would agree the S3 Java SDK 2.0 can't be considered finished without it.

[congtq]

It is almost 2 years since this issue was created. Do AWS really want developers to use 2.0?
To sad that now I have to switch back to 1.x after wasting days on implementation for AWS 2.0.

[cesartl]

I have to agree this is very strange indeed. Seems like a feature abandoned halfway through.

[debora-ito]

Hi everyone, just a quick update letting you know that we haven't forgotten this, it is in our V2 backlog along with other 1.11.x-parity features.

In the meantime, you can use the feature in SDK 1.x side by side with 2.x, here's an example on how to configure the pom.xml:
https://docs.aws.amazon.com/sdk-for-java/v2/migration-guide/side-by-side.html

[miere]

Hello @debora-ito. Thanks for you feedback and for your suggestion.

Unfortunately, I couldn't disagree more. Although I'm glad to finally hear something from AWS team regarding this issue, it's completely unacceptable to introduce even more overhead and bigger footprint to our deployment artifact by putting the both versions of the SDK at the same time.

I'm still open to make a PR and patch that for you, although the last time I've sent AWS a PR it took months to receive the first feedback on it.

If you ask me, I think such behavior is utterly inexcusable as we're a paying users willing to use more AWS IaaS/PaaS. Being completely honest with you, my report to my customers regarding AWS have changed quite a lot as other public cloud platforms have started to offer similar solutions and they have better customer support.

[albogdano]

I was just switching some of my code over to v2 and was shocked to find out this features is missing. This should be top priority for everyone involved in the development of SDK v2. I mean, come on guys, S3 is one of the core services! This is the second missing feature I encountered today. Earlier I was pretty disappointed to learn that the Waiter classes are missing as well. :-1:

[acroz]

I strongly agree with other commenters. I have wasted a considerable amount of development time updating code to use the new SDK only to discover that core functionality is missing.

There is no indication anywhere in the documentation that the client is incomplete; in fact this page specifically says the contrary. There should at least be some strong warning that the S3 client is incomplete so that developers do not waste their time and money upgrading.

[millems]

Sorry about the delay in responding, we wanted to come up with a response that we hope is more satisfying than “we promise we’re working on it”.

We agree with the sentiment from @acroz that we could do a better job communicating that this feature was missing. The documentation he linked says “all the service APIs are available today” which is very easy to interpret as “all methods on the 1.11.x client are available today” instead of our intended “all model-generated APIs which make remote service calls are available today”.

We’ve been working on bringing back the non-generated 1.11.x features to 2.x since GA, but we could have done a better job making it clear what our priorities are in that process.

What we’ll be doing based on this feedback:

1. We’re going to move the pre-signers up in our backlog. It’s clear that depending on 1.11.x for just this feature is a bigger blocker than we thought.
2. We’ll provide a code sample for you to use in the meantime, until the pre-signer task is completed. Unfortunately this isn’t something we can take a pull request on, because there are a number of services that need pre-signed URL support and we want our official SDK solution to support them all. (edit: link)
3. We’re going to publicly publish our prioritized list of 1.11.x features that we are going to bring to 2.x. This will make it clear what we’re currently working on and what we are going to be working on next in this area. (edit: link)
4. We’re going to link each feature in the backlog to an issue, and we will adjust the backlog’s order over time based on github feedback (upvotes, comments) and other similar channels.
5. We’ll update the documentation @acroz linked to make it clear that presigners aren’t in 2.x of the SDK yet.

[cesartl]

Thanks @millems , what counts as upvote for this issue? Upvotes on the original comment?

[albogdano]

@cesartl Most likely it's the number of reactions to the original issue post. This is in fact _the number 1 issue_ based on the number of reactions:

[millems]

@cesartl @albogdano correct!

[millems]

Update 11/7/19: This workaround is no longer necessary. See here for an officially supported solution.

Out-of-Date Content:
Below is a workaround you can use to generate a presigned S3 URL until the official SDK support is added. You will need to wait until tomorrow's release if you want to specify a custom expiration time... There's a bug we fixed today with presigned URLs to get it to work: https://github.com/aws/aws-sdk-java-v2/pull/1310

URI uri = presign(PresignUrlRequest.builder()
                                   .region(Region.US_WEST_2)
                                   .bucket("millems-203")
                                   .key("test")
                                   .build());

public URI presign(PresignUrlRequest request) {
    String encodedBucket, encodedKey;
    try {
        encodedBucket = URLEncoder.encode(request.bucket(), "UTF-8");
        encodedKey = URLEncoder.encode(request.key(), "UTF-8");
    } catch (UnsupportedEncodingException e) {
        throw new UncheckedIOException(e);
    }

    SdkHttpFullRequest httpRequest =
            SdkHttpFullRequest.builder()
                              .method(request.httpMethod())
                              .protocol("https")
                              .host("s3-" + request.region().id() + ".amazonaws.com")
                              .encodedPath(encodedBucket + "/" + encodedKey)
                              .build();

    Instant expirationTime = request.signatureDuration() == null ? null : Instant.now().plus(request.signatureDuration());
    Aws4PresignerParams presignRequest =
            Aws4PresignerParams.builder()
                               .expirationTime(expirationTime)
                               .awsCredentials(request.credentialsProvider().resolveCredentials())
                               .signingName(S3Client.SERVICE_NAME)
                               .signingRegion(request.region())
                               .build();

    return AwsS3V4Signer.create().presign(httpRequest, presignRequest).getUri();
}

public static class PresignUrlRequest implements ToCopyableBuilder<PresignUrlRequest.Builder, PresignUrlRequest> {
    private final AwsCredentialsProvider credentialsProvider;
    private final SdkHttpMethod httpMethod;
    private final Region region;
    private final String bucket;
    private final String key;
    private final Duration signatureDuration;

    private PresignUrlRequest(Builder builder) {
        this.credentialsProvider = Validate.notNull(builder.credentialsProvider, "credentialsProvider");
        this.httpMethod = Validate.notNull(builder.httpMethod, "httpMethod");
        this.region = Validate.notNull(builder.region, "region");
        this.bucket = Validate.notNull(builder.bucket, "bucket");
        this.key = Validate.notNull(builder.key, "key");
        this.signatureDuration = builder.signatureDuration;
    }

    public static Builder builder() {
        return new Builder();
    }

    public AwsCredentialsProvider credentialsProvider() {
        return credentialsProvider;
    }

    public SdkHttpMethod httpMethod() {
        return httpMethod;
    }

    public Region region() {
        return region;
    }

    public String bucket() {
        return bucket;
    }

    public String key() {
        return key;
    }

    public Duration signatureDuration() {
        return signatureDuration;
    }

    @Override
    public Builder toBuilder() {
        return builder()
                .credentialsProvider(credentialsProvider)
                .region(region)
                .bucket(bucket)
                .key(key)
                .signatureDuration(signatureDuration);
    }

    public static class Builder implements CopyableBuilder<Builder, PresignUrlRequest> {
        private AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.create();
        private SdkHttpMethod httpMethod = SdkHttpMethod.GET;
        private Region region;
        private String bucket;
        private String key;
        private Duration signatureDuration;

        public Builder credentialsProvider(AwsCredentialsProvider credentialsProvider) {
            this.credentialsProvider = credentialsProvider;
            return this;
        }

        public Builder httpMethod(SdkHttpMethod httpMethod) {
            this.httpMethod = httpMethod;
            return this;
        }

        public Builder region(Region region) {
            this.region = region;
            return this;
        }

        public Builder bucket(String bucket) {
            this.bucket = bucket;
            return this;
        }

        public Builder key(String key) {
            this.key = key;
            return this;
        }

        public Builder signatureDuration(Duration signatureDuration) {
            this.signatureDuration = signatureDuration;
            return this;
        }

        @Override
        public PresignUrlRequest build() {
            return new PresignUrlRequest(this);
        }
    }
}

[millems]

We’re going to publicly publish our prioritized list of 1.11.x features that we are going to bring to 2.x. This will make it clear what we’re currently working on and what we are going to be working on next in this area.

We've published our backlog of new (public) features: https://github.com/aws/aws-sdk-java-v2/projects/1

Presigned URL generation is on top of the backlog at this time.

You can also keep an eye on what else we're working on and some of what we're planning to work on, minus any secret stuff that we can't talk about yet.

[jvinding]

                              .host("s3-" + request.region().id() + ".amazonaws.com")

There is no DNS entry for s3-us-east-1.amazonaws.com at the moment, so this code generating invalid URLs in that region.
is this a problem with the DNS, with the code, or are pre-signed URLs not supported in us-east-1?

[spfink]

@jvinding not sure why the dash pattern was included in the code sample provided above. Replace "s3-" with "s3." and that should work in all regions including us-east-1.

[jvinding]

@jvinding not sure why the dash pattern was included in the code sample provided above. Replace "s3-" with "s3." and that should work in all regions including us-east-1.

Thanks for the response, unfortunately, that also fails with "The request signature we calculated does not match the signature you provided. Check your key and signing method."

I believe that the signer is not using the url in the request, but rather calculates the url based on the service and region using a -.

So, if I use . in the signing request, it won't actually match the url that gets signed.

[jvinding]

so, it seems there were 2 issues with the provided code, one is that s3-us-east-1.amazonaws.com does not exist.

I solved this by creating the url as so:

        val region = US_EAST_1 == request.region() ? "" : ("-" + request.region().id());
        final val httpRequest = SdkHttpFullRequest.builder()
                                                  .method(request.httpMethod())
                                                  .protocol("https")
                                                  .host("s3" + region + ".amazonaws.com")
                                                  .encodedPath(encodedBucket + "/" + encodedKey)
                                                  .build();

The other being that / in keys should not be encoded. e.g. a key of bob/joe/steve.png should remain untouched, but the example code turned it to bob%2fjoe%2fsteve.png.
I've fixed it by altering the encoding section of presign function as follows:

encodedBucket = URLEncoder.encode(request.bucket(), "UTF-8"); encodedKey = Arrays.stream(request.key().split("/")) .map(s -> { try { return URLEncoder.encode(s, "UTF-8"); } catch (UnsupportedEncodingException e) { throw new UncheckedIOException(e); } }) .collect(Collectors.joining("/"));

[sigpwned]

I noticed that the sample code does not contain some important features for generating presigned URLs from the v1 library, namely the ability to provide custom Content-Type and Content-Disposition headers on the response. I've added these features to the code sample and integrated the feedback from others in the thread. I can confirm this code works for me in some unit tests I've written:

public class PresignUrlRequest
        implements ToCopyableBuilder<
            PresignUrlRequest.Builder,
            PresignUrlRequest> {
    public static URI presign(PresignUrlRequest request) {
        String encodedBucket, encodedKey;
        try {
            encodedBucket = URLEncoder.encode(request.bucket(), "UTF-8");

            encodedKey = Arrays.stream(request.key().split("/"))
                .map(s -> {
                    try {
                        return URLEncoder.encode(s, "UTF-8");
                    }
                    catch (UnsupportedEncodingException e) {
                        throw new UncheckedIOException(e);
                    }
                })
                .collect(joining("/"));
        }
        catch (UnsupportedEncodingException e) {
            throw new UncheckedIOException(e);
        }

        String regionPart=request.region() == Region.US_EAST_1
            ? ""
            : ("-" + request.region().id());
        SdkHttpFullRequest.Builder httpRequestBuilder =
            SdkHttpFullRequest.builder()
                .method(request.httpMethod())
                .protocol("https")
                .host("s3" + regionPart + ".amazonaws.com")
                .encodedPath(encodedBucket + "/" + encodedKey);

        if(request.responseContentDisposition() != null)
            httpRequestBuilder.appendRawQueryParameter(
                "response-content-disposition",
                request.responseContentDisposition());

        if(request.responseContentType() != null)
            httpRequestBuilder.appendRawQueryParameter(
                "response-content-type",
                request.responseContentType());

        SdkHttpFullRequest httpRequest=httpRequestBuilder.build();

        Instant expirationTime = request.signatureDuration() == null
            ? null
            : Instant.now().plus(request.signatureDuration());

        Aws4PresignerParams presignRequest =
            Aws4PresignerParams.builder()
                .expirationTime(expirationTime)
                .awsCredentials(request
                    .credentialsProvider()
                    .resolveCredentials())
                .signingName(S3Client.SERVICE_NAME)
                .signingRegion(request.region())
                .build();

        return AwsS3V4Signer.create()
            .presign(httpRequest, presignRequest)
            .getUri();
    }

    private final AwsCredentialsProvider credentialsProvider;
    private final SdkHttpMethod httpMethod;
    private final Region region;
    private final String bucket;
    private final String key;
    private final Duration signatureDuration;
    private final String responseContentType;
    private final String responseContentDisposition;

    private PresignUrlRequest(Builder builder) {
        this.credentialsProvider = Validate.notNull(builder.credentialsProvider, "credentialsProvider");
        this.httpMethod = Validate.notNull(builder.httpMethod, "httpMethod");
        this.region = Validate.notNull(builder.region, "region");
        this.bucket = Validate.notNull(builder.bucket, "bucket");
        this.key = Validate.notNull(builder.key, "key");
        this.signatureDuration = builder.signatureDuration;
        this.responseContentType = builder.responseContentType;
        this.responseContentDisposition = builder.responseContentDisposition;
    }

    public static Builder builder() {
        return new Builder();
    }

    public AwsCredentialsProvider credentialsProvider() {
        return credentialsProvider;
    }

    public SdkHttpMethod httpMethod() {
        return httpMethod;
    }

    public Region region() {
        return region;
    }

    public String bucket() {
        return bucket;
    }

    public String key() {
        return key;
    }

    public Duration signatureDuration() {
        return signatureDuration;
    }

    public String responseContentType() {
        return responseContentType;
    }

    public String responseContentDisposition() {
        return responseContentDisposition;
    }

    @Override
    public Builder toBuilder() {
        return builder()
            .credentialsProvider(credentialsProvider)
            .region(region)
            .bucket(bucket)
            .key(key)
            .signatureDuration(signatureDuration);
    }

    public static class Builder implements CopyableBuilder<Builder, PresignUrlRequest> {
        private AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.create();
        private SdkHttpMethod httpMethod = SdkHttpMethod.GET;
        private Region region;
        private String bucket;
        private String key;
        private Duration signatureDuration;
        private String responseContentType;
        private String responseContentDisposition;

        public Builder credentialsProvider(AwsCredentialsProvider credentialsProvider) {
            this.credentialsProvider = credentialsProvider;
            return this;
        }

        public Builder httpMethod(SdkHttpMethod httpMethod) {
            this.httpMethod = httpMethod;
            return this;
        }

        public Builder region(Region region) {
            this.region = region;
            return this;
        }

        public Builder bucket(String bucket) {
            this.bucket = bucket;
            return this;
        }

        public Builder key(String key) {
            this.key = key;
            return this;
        }

        public Builder signatureDuration(Duration signatureDuration) {
            this.signatureDuration = signatureDuration;
            return this;
        }

        public Builder responseContentType(String responseContentType) {
            this.responseContentType = responseContentType;
            return this;
        }

        public Builder responseContentDisposition(String responseContentDisposition) {
            this.responseContentDisposition = responseContentDisposition;
            return this;
        }

        @Override
        public PresignUrlRequest build() {
            return new PresignUrlRequest(this);
        }
    }
}

All credit to @millems for the original sample code, and @jvinding for pointing out a couple of errors in it.

[mblund]

I had some problems with keys that contained spaces. My solution was to change the way the key was encoded and also set that the signer should not double encoded the url.

These are my changes if it helps someone else:

                encodedKey = Arrays.stream(request.key().split("/"))
                    .map(s -> {
                        try {
                            return URLEncoder.encode(s, "UTF-8")
                                    .replaceAll("\\+", "%20")
                                    .replaceAll("\\%21", "!")
                                    .replaceAll("\\%27", "'")
                                    .replaceAll("\\%28", "(")
                                    .replaceAll("\\%29", ")")
                                    .replaceAll("\\%7E", "~");
                        }
                        catch (UnsupportedEncodingException e) {
                            throw new UncheckedIOException(e);
                        }
                    })
                    .collect(joining("/"));

        Aws4PresignerParams params =
                Aws4PresignerParams.builder()
                        .expirationTime(expirationTime)
                        .awsCredentials(request
                                .credentialsProvider()
                                .resolveCredentials())
                        .signingName(SERVICE_NAME)
                        .signingRegion(request.region())
                        .doubleUrlEncode(false)
                        .build();

[Crysis21]

Wow, still no support for this feature. Why bother publish the version 2 of the SDK years ago, if you can't maintain it? Going back to 1.xx

[bendem]

I'd like to provided a simpler updated version (without builder and with APIs that don't require handling exceptions that never happen, thanks java 11). I've not tested it extensively with special characters but https://github.com/aws/aws-sdk-java-v2/issues/203#issuecomment-530463127 shows a way to deal with that.

public URI generateUrl(AwsCredentialsProvider credentialsProvider, Region region, String bucket, String key) {
    var encodedBucket = URLEncoder.encode(bucket, StandardCharsets.UTF_8);
    var encodedKey = Stream.of(key.split("/"))
        .map(part -> URLEncoder.encode(part, StandardCharsets.UTF_8))
        .collect(Collectors.joining("/"));
    var host = "s3." + region.id() + ".amazonaws.com";

    var httpRequest = SdkHttpFullRequest.builder()
        .method(SdkHttpMethod.PUT)
        .protocol("https")
        .host(host)
        .encodedPath(encodedBucket + "/" + encodedKey)
        .build();

    var presignRequest = Aws4PresignerParams.builder()
        .awsCredentials(credentialsProvider.resolveCredentials())
        .signingName(S3Client.SERVICE_NAME)
        .signingRegion(region)
        .expirationTime(Instant.now().plus(5, ChronoUnit.MINUTES))
        .doubleUrlEncode(false)
        .build();

    return AwsS3V4Signer.create()
        .presign(httpRequest, presignRequest)
        .getUri();
}

[adityasrini]

I was using @bendem's (concise and readable!) code to programatically generate a presigned URL for Heroku restores and for The request signature we calculated does not match the signature you provided. Check your key and signing method.. For those that come across a similar error, use SdkHttpMethod.GET to generate those URLs.

[rachitpant]

Forget the half baked and missing feautures, I sincerely hate the API documentation for v2 sdk. You can only do a guess work and spend hours and play with it.

[millems]

@rachitpant Can you elaborate more on that in a new, separate github issue? We're aware that the lack of stack overflow history can be frustrating for a new product, and that the docs on usage of the individual services needs improvement across all of the SDKs, but specific examples of where you need to do guess work will help us prioritize fixes.

[brettevocalize]

The public project board doesn't really seem to have any progress made in the last several months. I could be reading that wrong though. I also don't understand why this is still not done. Is this project being put on maintenance?

[millems]

@brettevocalize This project is very much still under development (we're committing, I promise!). We just haven't been super active on the public board tasks, including this issue.

We've made some headway on this feature this week, though! You can read our initial design/plan here. Feel free to leave comments!

[millems]

Latest: We've finished our initial design/plan, and have started implementation.

If you'd like to, please let us know if you disagree with anything in the design/plan. Everything is mutable until we GA the feature.

[SuhanKoh]

@millems Thanks for the update! One thing to comment on, (maybe endpointOverride(..) is intended for this?). Maybe also support String type region, if AWS added a new region, and due to some reason we cannot update the SDK yet, this would provide a way to overwrite the existing Region object.

[millems]

@SuhanKoh In V2, we actually support getting a Region from a string with Region.of("some-additional-region"), even if the SDK isn't aware of that region.

[msanaulla]

@millems Its amazing to know that the plan is done and the feature is being implemented.

Do you have any timelines for getting the feature available for use?

[millems]

@msanaulla Unfortunately we can't share timelines for features being released, but we're planning on launching support for S3's GetObject, then PutObject, and then other operations.

We're currently working on GetObject. We'll let you know as soon as we have something for you to try out.

[danielcompton]

The design seems pretty good to me. One thing which I didn't see was any discussion of how URLs for buckets with . in the name will be handled, and whether you would default to using path-style addressing or virtual-host based addressing? Virtual-host based addressing seems to be strongly encouraged for the future: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/.

I've opened a similar issue for the v1 SDK at https://github.com/aws/aws-sdk-java/issues/2105.

[millems]

There are some issues with virtual-style addressing where '.' is in the bucket name. From that blog post:

Bucket Names with Dots – It is important to note that bucket names with “.” characters are perfectly valid for website hosting and other use cases. However, there are some known issues with TLS and with SSL certificates. We are hard at work on a plan to support virtual-host requests to these buckets, and will share the details well ahead of September 30, 2020.

The current plan is that the presigners will follow the same logic as the S3 client: favor virtual-style addressing, but fall back to path-style addressing when the bucket name isn't considered "DNS compatible". The rules for what that means is complicated, but it currently means that path-style addressing will be used for buckets with a '.' in the name.

[millems]

We have pushed a draft implementation for a presigned URL generator that supports GetObject requests to a public github branch. It would be awesome if some of you could give comments on the feature PR, or even tried it out try to see how the user experience feels.

Here's an example usage if you want the cliff-notes version of things.

We don't have an exact date for when the implementation will hit master, but we feel like we're getting close.

[millems]

The GetObject presigner has been merged and will go out with the next release. Sample code: https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/java/software/amazon/awssdk/services/s3/presigner/S3Presigner.java

[millems]

This task will remain open until we've reached presigner parity with 1.x. Next will be S3's PutObject. Let us know if there's a specific presigner you'd like prioritized after that.

[kimxogus]

As initial design, is it right that generating S3's PresignedPOST url will be included in M6 milestone?

[millems]

@kimxogus POST to a /bucket/key location isn't something I'm super familiar with. Is this to upload an object using a presigned URL? If so, that should be supported by the M2 milestone.

[kimxogus]

@millems I mean presigned url for multipart post upload which exists in aws-js-sdk, specifying conditions is needed in our service.
Our logic is Client request to upload an object > Check pre-conditions for upload(like user authentication) in backend > generate presigned post upload url and fields with conditions(object key, content-length and content type) in aws lambda in js and return it to browser client > client execute multipart upload with the presigned url with conditions.

[millems]

@kimxogus Thanks for the clarification. POST policies as they are in the amplify SDK aren't in the scope of this feature, and I'm not even sure they're supported in V1 of the SDK.

A subset of the same functionality will be available via pre-signed PUTs, but not all POST policy conditions will be supported.

Could you create a separate feature request for that piece of functionality?

[kimxogus]

V1 sdk doesn't support this feature also(https://github.com/aws/aws-sdk-java/issues/834).

I'll create a separate feature request.

[kimxogus]

Created a feature request #1493

[driptaroop-n26]

I'd like to provided a simpler updated version (without builder and with APIs that don't require handling exceptions that never happen, thanks java 11). I've not tested it extensively with special characters but #203 (comment) shows a way to deal with that.

public URI generateUrl(AwsCredentialsProvider credentialsProvider, Region region, String bucket, String key) {
    var encodedBucket = URLEncoder.encode(bucket, StandardCharsets.UTF_8);
    var encodedKey = Stream.of(key.split("/"))
        .map(part -> URLEncoder.encode(part, StandardCharsets.UTF_8))
        .collect(Collectors.joining("/"));
    var host = "s3." + region.id() + ".amazonaws.com";

    var httpRequest = SdkHttpFullRequest.builder()
        .method(SdkHttpMethod.PUT)
        .protocol("https")
        .host(host)
        .encodedPath(encodedBucket + "/" + encodedKey)
        .build();

    var presignRequest = Aws4PresignerParams.builder()
        .awsCredentials(credentialsProvider.resolveCredentials())
        .signingName(S3Client.SERVICE_NAME)
        .signingRegion(region)
        .expirationTime(Instant.now().plus(5, ChronoUnit.MINUTES))
        .doubleUrlEncode(false)
        .build();

    return AwsS3V4Signer.create()
        .presign(httpRequest, presignRequest)
        .getUri();
}

This works for me. 🎉 Although I'm not sure how to enable server side encryption for the pre-signed url in this approach. anyone have any idea?

[millems]

@driptaroop-n26 We have an official PutObject version in a pull request right now, and it should go out in a few days: https://github.com/aws/aws-sdk-java-v2/pull/1502

It should support all fields in PutObject, including server-side encryption. I know that doesn't unblock you immediately.

[millems]

FYI all, the PutObject presigner has been merged and released in today's SDK release: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/presigner/S3Presigner.html#presignPutObject-software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest-

The PutObject presigner can be used to generate a presigned URL that another system can use to upload an object to S3:

S3Presigner presigner = S3Presigner.create();
PresignedPutObjectRequest presignedRequest =
    presigner.presignPutObject(r -> r.signatureDuration(Duration.ofMinutes(5))
                                        .putObjectRequest(por -> por.bucket(testBucket).key(objectKey)));

System.out.println("Pre-signed URL to upload a file to: " + 
                   presignedRequest.url());
System.out.println("Which HTTP method needs to be used when uploading a file: " + 
                   presignedRequest.httpRequest().method());
System.out.println("Which headers need to be sent with the upload: " + 
                   presignedRequest.signedHeaders())

Here's an example of uploading to S3 using a PresignedPutObjectRequest:

PresignedPutObjectRequest presignedRequest = ...;
SdkHttpClient httpClient = ApacheHttpClient.builder().build();

ContentStreamProvider contentToUpload = () -> new StringInputStream("Hello, World!");
HttpExecuteRequest uploadRequest = HttpExecuteRequest.builder()
                                                     .request(presignedRequest.httpRequest())
                                                     .contentStreamProvider(contentToUpload)
                                                     .build();

HttpExecuteResponse response = httpClient.prepareRequest(uploadRequest).call();
Validate.isTrue(response.httpResponse().isSuccessful());

[driptaroop-n26]

Finally 😅. Played with it a little. Everything seems to be in order. Thank you.

[millems]

This is now supported for Polly's SynthesizeSpeech API: https://github.com/aws/aws-sdk-java-v2/blob/master/services/polly/src/main/java/software/amazon/awssdk/services/polly/presigner/PollyPresigner.java

Does anyone else have any specific presigners they're missing that were in 1.11.x? We haven't backfilled all of them, yet, but would like some feedback on what people are blocked on.

[millems]

Resolving this issue for now, then. Please open separate issues for tracking specific request presigners you're missing from 1.11.x. We'll backfill those eventually, but we believe we've backfilled the most used ones at this time.

[jokoso]

Thank you very much for this! We're using both the GetObject and PutObject variants.

[acroz]

@millems Thanks very much for prioritising these presigners - this goes a long way towards allowing us to drop our dependency on the old SDK.

The one thing we are still missing for our application is presigning of an UploadPartRequest - do you have any plans or are there any existing requests/issues to support that?

---

In case I'm missing something:

On the old SDK we presigned multipart uploads like so (Scala):

    val request =
      new GeneratePresignedUrlRequest(bucket, path.path, HttpMethod.PUT)
        .withExpiration(Date.from(Instant.now.plus(presignedUrlExpirationDuration.toJava)))
    request.addRequestParameter("uploadId", uploadId)
    request.addRequestParameter("partNumber", partNumber.toString)
    s3ClientV1.generatePresignedUrl(request)

I looked at the PutObject presigning functionality can couldn't find a similar way to manually add such parameters - if there's a way to do something similar with the new SDK I would be glad to see an example.

[millems]

@acroz We should be able to support UploadPart very easily. It's just a cut+paste of the changes in https://github.com/aws/aws-sdk-java-v2/pull/1502/files, replacing PutObject with UploadPart and writing the appropriate tests.

Do you currently use a normal initiate-multipart-upload request using the client, then use a presigned request for upload-part, and use the client to complete the multipart upload? Or do you use a presigned initiate/complete?

[gagangoel1994]

@millems by when can we expect support for UploadPart in the sdk? I use a normal initiate-multipart-upload request using the client, then use a presigned request for upload-part, and use the client to complete the multipart upload.

[millems]

@gagangoel1994 We will add it to our task list, but we can't provide dates. A pull request would get it through quicker, if it's urgent.

[millems]

Opened https://github.com/aws/aws-sdk-java-v2/pull/1646 to add support for pre-signing uploadPart requests (and a few other part-related operations).

[millems]

@gagangoel1994 We've merged support for this.

[acroz]

🎉 Great news!

[gncj]

Hello,

I'm having an error when using the "PresignedPutObjectRequest" with objectKey like this:
'private/eu-west-1:2b4aa120-2c1b-4208-94fa-620e581c686f/report20200310.pdf'. The error obtained is: 403 - Forbidden.

If I remove the ':' there's no problem. But I can't remove the ':' from the objectKey.
Because objectKey is composed of:
'private/${cognito-identity.amazonaws.com:sub}/FILENAME.pdf'
The value 'cognito-identity.amazonaws.com:sub' is basically the Cognito identity id of an authorized user that contains ':'.

I already tried with "S3Client" with method "putObject" and have also an error:
software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: S3, Status Code: 403, Request ID: DF0CAEFC74A5DA32)

Can anyone help me?

Thanks in advance

[angelddios999]

@gagangoel1994 We've merged support for this.

Hi @millems. I think this implementation is what I'm looking for (use the sdk to upload a multi-part request using a pre-signed URL), however, I'm not clear on how to use this PresignedCreateMultipartUploadRequest. I have searched in the AWS documentation with no luck. Is there a place where I can find an example of the usage? Thank you!

[millems]

@angelddios999

These are our only official samples right now, which aren't multi-part: https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-s3-presign.html

As far as unofficial stuff, you could derive some usage examples out of our integration tests for the feature: https://github.com/aws/aws-sdk-java-v2/blob/b9379466c0925f0edf9c53773358674b39203333/services/s3/src/it/java/software/amazon/awssdk/services/s3/S3PresignerIntegrationTest.java#L238-L352

If you need any help on what parameters to specify in the create-multipart-upload request you can use the official create-multipart-upload documentation. It applies for presigned requests as well: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html

[angelddios999]

@angelddios999

These are our only official samples right now, which aren't multi-part: https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-s3-presign.html

As far as unofficial stuff, you could derive some usage examples out of our integration tests for the feature:

https://github.com/aws/aws-sdk-java-v2/blob/b9379466c0925f0edf9c53773358674b39203333/services/s3/src/it/java/software/amazon/awssdk/services/s3/S3PresignerIntegrationTest.java#L238-L352

If you need any help on what parameters to specify in the create-multipart-upload request you can use the official create-multipart-upload documentation. It applies for presigned requests as well: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html

Hi, @millems. Thanks for the quick reply. I checked at these resources, but they look as if the signer and the uploader live in the same place. What I'm trying to do is to create two components: One signer (maybe a lambda function) with S3 access that generates and provides a single URL to One client without privileges (maybe an EC2 instance).
I have checked at this kind of implementation but it seems like for MultiPart upload the client needs to retrieve one URL for each part of the file. Does this PresignedCompleteMultipartUploadRequest addresses this problem?

Following some of the examples, I tried the following:

In the URL provider side:
`

    // Presigner presigner = ...
    CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
            .bucket(bucket).key(fileKey).build();

    CreateMultipartUploadPresignRequest createMultipartUploadPresignRequest = CreateMultipartUploadPresignRequest.builder()
            .signatureDuration(Duration.ofMinutes(10)).createMultipartUploadRequest(createMultipartUploadRequest).build();

    PresignedCreateMultipartUploadRequest presignedCreateMultipartUploadRequest =
            presigner.presignCreateMultipartUpload(createMultipartUploadPresignRequest);

    return presignedCreateMultipartUploadRequest.url();

`

In the consumer side:
`

        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setDoOutput(true);
        connection.setRequestProperty("Content-Type", "text/plain");
        connection.setRequestMethod("POST"); // I tried with PUT also
        OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
        out.write("This text uploaded as an object via presigned URL.");
        out.close();

`

But this is returning HTTP 403.

Is this the correct approach when the consumer only receives the URL from the Signer? Do you have an idea of why it's returning HTTP 403? Thanks in advance.

[millems]

@angelddios999

The PresignedCreateMultipartUploadRequest only creates a multipart upload on the S3 side. You have to use upload-part requests to fulfill that multipart upload (and then make another call to complete it). If you just want to do an upload via a presigned URL, you can do it single-part via a PresignedPutObjectRequest.

[angelddios999]

@angelddios999

The PresignedCreateMultipartUploadRequest only creates a multipart upload on the S3 side. You have to use upload-part requests to fulfill that multipart upload (and then make another call to complete it). If you just want to do an upload via a presigned URL, you can do it single-part via a PresignedPutObjectRequest.

Thank you, @millems. I don't mean to spam this thread, but I just would like to confirm...

I would like to use the multipart upload from a client, where this client only has access to pre-signed URLs, and there is a trusted provider returning these URLs to the client on demand. Is my only option to upload the file in a single part from the client using one URL? Or is there a way to achieve multipart upload from the client using only these URLs provided by my URL service?

Thanks!

[millems]

@angelddios999 You can do a multipart upload using presigned requests, but it's complicated because you will have to presign multiple upload-part requests for the client to use.

E.g. If the client will upload in 4 parts, your service will have to presign 4 distinct upload-part requests (in addition to the create-multipart-upload and complete-multipart-upload requests).

[shambhand]

Can someone please update the https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md
if this feature is available now. Looks like documentation is still saying "Not Supported"

[c4lm]

Can someone please update the https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md
if this feature is available now. Looks like documentation is still saying "Not Supported"

As far as I can tell, while the feature is available, it is not supported in the client (S3Client/S3AsyncClient), at least for the version 2.15.35.