Aws-sdk-java-v2: HTTP client with AWS signing logic?

[amihaiemil]

First off, v2 looks much better than v1, thank you for implementing it!

But, would there be a possibility to have just an HTTP Client module which has the request signing logic implemented?

Some applications just need to call a few endpoints and for that they shouldn't need to have the whole SDK as a dependency. Besides, writing Java wrappers for APIs is very easy, maybe it would be easier for someone to write their own objects, over an HTTP Client, rather than learning how to properly use the SDK, But in AWS' case, the complicated part is signing the requests, so it would be nice to offer this functionality in a separate module.

[millems]

Our signers convert an unsigned HttpRequest into a signed HttpRequest, where HttpRequest is specific to the SDK.

Would you be fine building an HttpRequest object, having it signed by the SDK, and then needing to adapt it to an Apache request, netty request, etc.? Or would you want bindings for various HTTP request libraries?

[amihaiemil]

@millems I would just like to perform a aws-signed HTTP request to an aws HTTP endpoint, what is complex in that? Why or how is an HTTP request tied to the SDK? All I want to do is the following:

Request req = new HttpRequest("https://aws.amazon.com/elasticsearch/_search)
    .method("GET")
    .header("...", "...")
    .header("...", "...")
    .sign() //here!

Response resp = req.fetch();

So, I have interface Request which allows me to build on http request (adding a host, method, headers, body etc) and which has method sign().

then needing to adapt it to an Apache request, netty request?

Who said that I should do that? Apache, netty or maybe a bare-bones UrlConnection is what you guys would encapsulate and use inside this HTTP client library. That would be on you. Personally, I wouldn't bother with Apache (which I really believe is a bad library).

[millems]

Thanks for the clarification! If you don't care about the HTTP implementation itself, that makes it easier. We already expose our HTTP clients in a separate module as a public API, so we would just need to put our signers into a separate one and design them as public APIs.

Example of using our HTTP client:

ApacheSdkHttpClientFactory httpClientFactory = ApacheSdkHttpClientFactory.builder().build();
SdkHttpClient httpClient = httpClientFactory.createHttpClient();

SdkHttpFullRequest request = SdkHttpFullRequest.builder()
                                               .method(SdkHttpMethod.GET)
                                               .protocol("https")
                                               .host("localhost")
                                               .port(8080)
                                               .header("Host", "localhost:8080")
                                               .header("Accept", "*/*")
                                               .content(new ByteArrayInputStream(new byte[] {}))
                                               .build();

SdkRequestContext context = SdkRequestContext.builder().build();

SdkHttpFullResponse response = httpClient.prepareRequest(request, context)
                                         .call();

Right now, it's really hard for a customer to use the signer, and the HTTP client itself could also use some work (eg. context should be optional) . We could definitely make the signers a public API as part of the signer refactor story we have: https://github.com/aws/aws-sdk-java-v2/issues/28

We already intend to iterate on the HTTP client API before we freeze for backwards compatibility.

[amihaiemil]

@millems

so we would just need to put our signers into a separate one and design them as public APIs.

Nice :)
I wanted to add something about the snipped above, but I see I've already commented on #28 back in April, haha.

Yes, as much encapsulation as possbile, and extensibility through interfaces (e.g. I may want to decorate some objects or reimplement stuff) would be the ok.

[varunnvs92]

This feature has been implemented in https://github.com/aws/aws-sdk-java-v2/pull/496 and merged to master branch. It will released to maven during our next release (sometime next week). Please try out the changes and provide your feedback. See AwsS3V4SignerIntegrationTest and SignersIntegrationTest that has examples of using the signers directly without the use of SDK clients.

[amihaiemil]

@varunnvs92 thank you so much, I will have a look asap :)

[volnei]

There are any documentation about using signed http request instead of services?

What dependencies is required?
Any example code?

Thanks

[amihaiemil]

@volnei See the test classes linked in the comment above, by @varunnvs92, I think they should help

[amihaiemil]

@varunnvs92 @millems Well, looks ok (definetely better than v1):

    AwsS3V4Signer signer = AwsS3V4Signer.create();
    SdkHttpFullRequest httpFullRequest = generateBasicGetRequest();

    // sign the request
    SdkHttpFullRequest signedRequest = signer.sign(httpFullRequest, constructSignerParams());

    SdkHttpClient httpClient = ApacheHttpClient.builder().build();

    SdkHttpFullResponse response = httpClient.prepareRequest(
        signedRequest, 
        SdkRequestContext.builder().build()
     ) .call();

I see you have wrapped the Apache HttpClient in your own builder, nice. Personally, I would have implemented the org.apache.http.client.HttpClient interface and maybe go with the decorator pattern, but that's another story.

What I don't understand is that SdkRequestContext -- what is it supposed to do/represent, and could SdkHttpClient have an overloaded version of prepareRequest that wouldn't require that parameter?

[volnei]

@amihaiemil So, I supose that you need to have whole S3 dependencies to do a simple put, right?

I really can't understand why it's is not a vanilla rest with a decorator. IMHO, it could be simple as...

Request request = RequestFactory.create(...) // implementation independent 
request.sign()
Response response = request.call()

And services only consume this for any action. AWS APIs since v1 is too verbose. I just wanna a signer for any http to do my calls directly and jar size/count matters.

[amihaiemil]

@volnei I am not sure, look how it is distributed. In V1 of the SDK, you could pull in just the core module, which had just the http client classes (see here). Maybe V2 has a similar distribution and you can have just the httpclient part as a dependency.

[volnei]

@amihaiemil, thanks!

But in examples above I need AwsS3V4Signer witch is included in s3 package, right? Then even if I just call rest requests I need S3 package in my app.

[amihaiemil]

@volnei I think AwsS3V4Signer offers some specific signing logic only for S3, but in SignersIntegrationTest I see Aws4Signer which seems to be the generic one (the same one as in V1) -- try that, it should be in a common, core package, I would say.

[varunnvs92]

@amihaiemil
We were using SdkRequestContext previously to pass additional parameters that were used in metrics etc. We have a task to investigate its use and remove if its not necessary.

@volnei
We kept AwsS3V4Signer class in s3 module as its specific to s3. But it makes sense that customers might want to use it without depending on s3 module. We can move it to auth module where other signers currently reside.

[volnei]

@amihaiemil Thanks

@varunnvs92

We kept AwsS3V4Signer class in s3 module as its specific to s3. But it makes sense that customers might want to use it without depending on s3 module. We can move it to auth module where other signers currently reside.

Great :)

[amihaiemil]

@varunnvs92 I see. I close this issue, then :) thanks!