Aws-sdk-java: Improve S3 async support

[rocketraman]

TransferManager is useful for a subset of operations (upload and download) but doesn't support other operations like getting metadata, deleting objects, checking the existence of an object, and so forth. Since these are all remote operations, blocking is an issue.

Please add async support for other S3 operations in addition to upload/download.

Implementation note: I'd like to have an ability to get a Future in the implementations of this feature [1]. With TransferManager Upload and Download, I must create my own Future as a ProgressListener, which is messy given that internally Upload and Download can already get Future from their internal monitors. This will make it easier to adapt the S3 API to RxJava and other async libs.

(Followup on issue #140)

[1] While returning Future would be a good lowest common denominator implementation, alternatively, consider exposing a reactive-streams interface for client code. If you do this, the API will be trivially future-compatible with the JDK9 flow apis, and compatible with all sorts of existing java async libraries your users might be using, including RxJava (see for example: https://github.com/ReactiveX/RxJava/wiki/Reactive-Streams) but also many others like Akka, Quasar, Vert.X and others (see http://www.reactive-streams.org/announce-1.0.0). In addition, it has a nice TCK to verify your implementation against the spec (https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.0/tck/README.md). Also note that page above on the RxJava wiki has some good concrete recommendations for library authors.

[varunnvs92]

@rocketraman
Sorry for the delayed response. Thank you for bringing up this one. We discussed about having async support for all clients but we don't have bandwidth for this right now. We added this feature request to our backlog. Thank you.

[eswenson1]

+1. Our service logic uses only async APIs for everything else (AWS and otherwise). The only blocking calls we have now are S3 calls.

[zeldigas]

@eswenson1 please don't treat async aws sdk calls as non blocking. They actually blocks, but that is moved to separate threads, managed by dedicated thread pool.

To be real non blocking internal http client should be changed to non blocking operations (like apache async http client or ning's one), but it seems that it'd be hard to do with current interfaces. My experiments at least show that it's not 5 minute task.

[eswenson1]

Thanks, @zeldigas. I have updated my code to treat all calls to AWS async APIs as blocking.

[zenogantner]

Hello, not sure whether this fits into the scope of this ticket -- if I understand the current TransferManager correctly, it supports async S3 downloads that are stored into files. It would also be nice to provide an API call that takes a callback on an InputStream.

[lukestephenson]

@zenogantner I don't think it should provide an API that takes a callback on an InputStream. The InputStream is not reactive. You ask it for data with a call to .read or some other variant. The point of an async client is not to introduce blocking calls. (Unless you are implying the InputStream has been fully read and will never block, but defeats the purpose of that interface)

[ghost]

👍 for this, would love to have an async S3 Java client. . .

[zeldigas]

Hey folks, just a heads up. Some experiments shows that it's possible with a number of limitations.
In 2 weeks i'm going to publish my non-blocking sdk experiment here: https://github.com/zeldigas/aws-sdk-non-blocking
Right now i'm mostly focused on dynamodb as field for tests, but suppose it could be spread onto other parts of aws sdk.

[ghost]

@zeldigas sounds great!!! Thanks.

[varunnvs92]

Hey guys, we are working on new major version of the SDK which will go into public dev preview at end of month (tentative). The new SDK will have support for non-blocking IO. But it won't include high level utilities like S3 TransferManager, DynamoDb Mapper. Please watch out for the announcement. We really appreciate if you guys try out the new SDK when its released and provide feedback on it.

[ghost]

@varunnvs92 sounds excellent, thanks for the heads up!

[zeldigas]

@mcummingscb code published: https://github.com/zeldigas/aws-sdk-non-blocking

[ghost]

Awesome, thanks @zeldigas I should be able to try that out soon.

[shorea]

We just launched a Developer Preview of AWS SDK for Java 2.0 that adds support for this feature request. Try it out and let us know what you think!

https://github.com/aws/aws-sdk-java-v2
http://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/welcome.html
https://aws.amazon.com/blogs/developer/aws-sdk-for-java-2-0-developer-preview/