Druid: Move to AWS S3 SDK

[niketh]

I plan to move to using AWS S3 SDK instead of using JetS3T. I have noticed the following problems with JetS3T:

1. AWS SDK has better support for auth (reading credentials files etc)
2. It is far behind AWS S3 SDK in terms of development-release/support.

What do you think @gianm @leventov @cheddar ?

[gianm]

+1, aws-java-sdk seems better than jets3t these days. Related: #4116 which would add support for s3a (aws-java-sdk backed S3) to Hadoop indexing.

[drcrallen]

There are some in the community who use jets3t.properties, so such a change would either need a migration plan or a major version update

[gianm]

My feeling is it's not worth supporting both at the same time, and instead we should do it in 0.11.0 and tell people that jets3t properties will no longer be respected (but whatever new aws-java-sdk configurator thing will).

[drcrallen]

@gianm agreed. also it wasn't clear in my prior message, but I think moving to the AWS sdk is a great idea.

[leventov]

Duplicate of #3886

[leventov]

Labelling Incompatible and Release Notes because present users may have jets3t.properties file to change some s3 configurations, that will stop working silently when code is changed to use official SDK.

[drcrallen]

@leventov is the solution for this one going to have to have a config that creates a https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/ClientConfiguration.html , or do you know of a way to have a jets3t.properties kind of file on the classpath?

[leventov]

@drcrallen I don't know, but creating a config via "Druid native" properties sounds like a good solution, for unification.

[leventov]

Related #4382

[jihoonson]

I'm working on this.

[sjtoik]

Cool! The endpoint configuration should be trivial with the new AWSClient-builder thingy. That just needs to be propagated trough-out the stack.