Aws-sdk-java-v2: How to use ACCESS_KEY and SECRET_KEY to initialize an S3AsyncClient

[zhuoliu]

This V2 non-blocking S3 client looks cool to me. I am glad to try it. Question: in the test example, it creates a client by "S3AsyncClient client = S3AsyncClient.create();". But that does not include either the endpoint or the access key and private key for your S3 server account. Could you provide an example for how to use these info to actually create a runnable V2 client? Thanks a lot.
similar to what we have for AmazonS3Client V1:

AWSCredentials creds = new BasicAWSCredentials(ACCESS_KEY, SECRET_KEY);
ClientConfiguration config = new ClientConfiguration();
amazonS3Client = new AmazonS3Client(creds, config);
amazonS3Client.setEndpoint(ENDPOINT);

[Wangab]

I have the same confusion.It increases the complexity of application deployment.

[tyehle]

Does this work?

AwsCredentialsProvider creds = StaticCredentialsProvider.create(AwsCredentials.create(accessKeyId, secretAccessKey));
S3AsyncClient s3Client = S3AsyncClient.builder().credentialsProvider(creds).build();

[shorea]

You should also be able to use a Lambda expression.

S3AsyncClient s3Client = S3AsyncClient.builder().credentialsProvider(() -> creds).build();

[zhuoliu]

@tyehle @shorea thanks for the reply.

I could write some initialization code as below:

AwsCredentialsProvider creds = new StaticCredentialsProvider(new AwsCredentials(ACCESS_KEY, SECRET_KEY));
S3AsyncClient s3Client = S3AsyncClient.builder().credentialsProvider(creds).build();

However, we also need to specify the ENDPOINT and REGION, then I replace the 2nd line above as:

 S3AsyncClient s3Client = S3AsyncClient.builder().credentialsProvider(creds)
    .region(Region.of(REGION))
    .endpointOverride(new URI(ENDPOINT))
    .build();

But I got the following exception. Any idea on how to set the endpoint and region?
` Exception in thread "main" java.lang.IllegalStateException: The signing region could not be determined. at software.amazon.awssdk.client.builder.DefaultClientBuilder.lambda$signingRegion$0(DefaultClientBuilder.java:156) at java.util.Optional.orElseThrow(Optional.java:290) at software.amazon.awssdk.client.builder.DefaultClientBuilder.signingRegion(DefaultClientBuilder.java:156) at software.amazon.awssdk.services.s3.DefaultS3BaseClientBuilder.defaultSignerProvider(DefaultS3BaseClientBuilder.java:50) at software.amazon.awssdk.config.defaults.ClientConfigurationDefaults.applyDefault(ClientConfigurationDefaults.java:112) at software.amazon.awssdk.config.defaults.ServiceBuilderConfigurationDefaults.applyOverrideDefaults(ServiceBuilderConfigurationDefaults.java:53) at software.amazon.awssdk.config.defaults.ClientConfigurationDefaults.applyDefaultValues(ClientConfigurationDefaults.java:127) at software.amazon.awssdk.config.defaults.ClientConfigurationDefaults.applyDefaultValues(ClientConfigurationDefaults.java:71) at software.amazon.awssdk.config.defaults.ClientConfigurationDefaults.applyAsyncDefaults(ClientConfigurationDefaults.java:61) at software.amazon.awssdk.client.builder.DefaultClientBuilder.asyncClientConfiguration(DefaultClientBuilder.java:213) at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:28) at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:22) at software.amazon.awssdk.client.builder.DefaultClientBuilder.build(DefaultClientBuilder.java:110) at software.amazon.awssdk.services.s3.S3AsyncClient.create(S3AsyncClient.java:183)

[siddharths067]

The S3AsyncClient throws null pointer exception for me, I have tried all endpoints, can someone check on this

this.s3Client = S3AsyncClient.builder()
          .credentialsProvider(this.credentials)
          .region(Region.AP_SOUTHEAST_1)
          .endpointOverride(new URI("s3-ap-southeast-1.amazonaws.com"))
          .build(); // Throws Null pointer Exception

At Stackoverflow

https://stackoverflow.com/questions/51966354/s3asyncclient-throws-java-lang-nullpointerexception-with-every-region-endpoint

java.lang.NullPointerException at software.amazon.awssdk.config.ImmutableClientConfiguration.lambda$schemeToProtocol$3(ImmutableClientConfiguration.java:141) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464) at software.amazon.awssdk.config.ImmutableClientConfiguration.schemeToProtocol(ImmutableClientConfiguration.java:141) at software.amazon.awssdk.config.ImmutableClientConfiguration.initializeLegacyConfiguration(ImmutableClientConfiguration.java:110) at software.amazon.awssdk.config.ImmutableClientConfiguration.<init>(ImmutableClientConfiguration.java:48) at software.amazon.awssdk.config.ImmutableAsyncClientConfiguration.<init>(ImmutableAsyncClientConfiguration.java:42) at software.amazon.awssdk.client.builder.DefaultClientBuilder.asyncClientConfiguration(DefaultClientBuilder.java:216) at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:28) at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:22) at software.amazon.awssdk.client.builder.DefaultClientBuilder.build(DefaultClientBuilder.java:110) at services.storage.Implementation.ObjectUploaderInterfaceImpl.<init>(ObjectUploaderInterfaceImpl.java:41) at services.storage.ObjectUploaderInterfaceTest.<init>(ObjectUploaderInterfaceTest.java:12) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) null at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [error] Test services.storage.ObjectUploaderInterfaceTest.uploadTest failed: java.lang.NullPointerException: null, took 0.008 sec at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

[zoewangg]

@zhuoliu Could you try with the latest version? Feel free to re-open if you continue to see the error after version upgrade.

[zhuoliu]

@zoewangg yes, I updated to the latest version:

 compile "software.amazon.awssdk:ec2:2.4.2"
 compile "software.amazon.awssdk:s3:2.4.2"

But still got the following error:
```
Exception in thread "main" java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to marshall request to JSON: protocol must not be null.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at com.apple.amp.simpleblob.s3v2.TestS3V2.main(TestS3V2.java:41)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to marshall request to JSON: protocol must not be null.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
at software.amazon.awssdk.services.s3.transform.GetObjectRequestMarshaller.marshall(GetObjectRequestMarshaller.java:53)
at software.amazon.awssdk.services.s3.transform.GetObjectRequestMarshaller.marshall(GetObjectRequestMarshaller.java:31)
at software.amazon.awssdk.core.client.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:53)
at software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:94)
at software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:81)
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:58)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClient.getObject(DefaultS3AsyncClient.java:1884)
at com.apple.amp.simpleblob.s3v2.TestS3V2.main(TestS3V2.java:34)
Caused by: java.lang.NullPointerException: protocol must not be null.
at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:117)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.standardizeProtocol(DefaultSdkHttpFullRequest.java:63)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.(DefaultSdkHttpFullRequest.java:52)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.(DefaultSdkHttpFullRequest.java:41)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:331)
at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:170)
at software.amazon.awssdk.protocols.xml.internal.marshall.XmlProtocolMarshaller.finishMarshalling(XmlProtocolMarshaller.java:122)
at software.amazon.awssdk.protocols.xml.internal.marshall.XmlProtocolMarshaller.marshall(XmlProtocolMarshaller.java:86)
at software.amazon.awssdk.protocols.xml.internal.marshall.XmlProtocolMarshaller.marshall(XmlProtocolMarshaller.java:49)
at software.amazon.awssdk.services.s3.transform.GetObjectRequestMarshaller.marshall(GetObjectRequestMarshaller.java:51)
... 7 more

[zhuoliu]

My code as below:
```
AwsCredentialsProvider creds = StaticCredentialsProvider.create(AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY));

    S3AsyncClient s3Client = S3AsyncClient.builder().credentialsProvider(creds)
            .region(Region.of(REGION))
            .endpointOverride(new URI(ENDPOINT))
            .build();

    CompletableFuture<GetObjectResponse> futureGet = s3Client.getObject(
            GetObjectRequest.builder()
                    .bucket(BUCKET)
                    .key("joe/666")
                    .build(),
            AsyncResponseTransformer.toFile(Paths.get("/tmp/myfile.out")));

    futureGet.get();

[zhuoliu]

@zoewangg could you please reopen this one?

[zoewangg]

@zhuoliu The error message is protocol must not be null, so please make sure the endpoint you passed has https or http

[zhuoliu]

@zoewangg that works, thank you 👍

[zoewangg]

1067 was merged to provide better error message if URI scheme is null.

[nagsenlkamble]

@zhuoliu Could you please share your imports for your working code?

[nagsenlkamble]

@zoewangg I have below code and it is throwing error software.amazon.awssdk.core.exception.SdkClientException: Credentials must not be null. Could you please help me here? Though I am providing credentials, I don't know what it is not getting them.

`import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Paths;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;

public class S3AsyncStreamOps {

public static void main(String[] args) {

    AwsCredentialsProvider creds = StaticCredentialsProvider.create(AwsBasicCredentials.create("my_access_key", "my_secret_key"));
    S3AsyncClient s3Client;
    try {
        s3Client = S3AsyncClient.builder().credentialsProvider(creds)
                .region(Region.US_WEST_1)
                .endpointOverride(new URI("https://abc.xyz.com:9021"))
                .build();
        CompletableFuture<GetObjectResponse> futureGet = s3Client.getObject(
                GetObjectRequest.builder()
                        .bucket("my_bucket_name")
                        .key("/somepath/anotherpath/myData.pdf")
                        .build(),
                AsyncResponseTransformer.toFile(Paths.get("/osfile/myfile.pdf")));
        futureGet.get();
    } catch (URISyntaxException e1) {
        e1.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (ExecutionException e) {
        e.printStackTrace();
    }
}

}`

Exception stack trace is:
ExecutionException Occurred
java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Credentials must not be null.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at com.aexp.ecp.vertx.spring.verticles.S3AsyncStreamOps.main(S3AsyncStreamOps.java:54)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Credentials must not be null.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)
at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:263)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:205)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:134)
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:58)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClient.getObject(DefaultS3AsyncClient.java:3949)
at com.aexp.ecp.vertx.spring.verticles.S3AsyncStreamOps.main(S3AsyncStreamOps.java:32)
Caused by: java.lang.NullPointerException: Credentials must not be null.
at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:117)
at software.amazon.awssdk.auth.signer.params.Aws4SignerParams.(Aws4SignerParams.java:42)
at software.amazon.awssdk.auth.signer.params.AwsS3V4SignerParams.(AwsS3V4SignerParams.java:27)
at software.amazon.awssdk.auth.signer.params.AwsS3V4SignerParams.(AwsS3V4SignerParams.java:21)
at software.amazon.awssdk.auth.signer.params.AwsS3V4SignerParams$BuilderImpl.build(AwsS3V4SignerParams.java:121)
at software.amazon.awssdk.auth.signer.AwsS3V4Signer.constructAwsS3SignerParams(AwsS3V4Signer.java:93)
at software.amazon.awssdk.auth.signer.AwsS3V4Signer.sign(AwsS3V4Signer.java:59)
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.signRequest(SigningStage.java:63)
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:49)
at software.amazon.awssdk.core.internal.http.pipeline.stages.SigningStage.execute(SigningStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.doExecute(AsyncRetryableStage.java:208)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.execute(AsyncRetryableStage.java:107)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.execute(AsyncRetryableStage.java:102)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:78)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:51)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:29)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:191)
... 6 more

[zoewangg]

@nagsenlkamble What SDK version are you using? Can you create a new issue?

[nagsenlkamble]

@zoewangg Will create new issue. btw I am using below dependencies.
<dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>aws-sdk-java</artifactId> <version>2.10.56</version> </dependency>

[nagsenlkamble]

@zoewangg I have opened new issue https://github.com/aws/aws-sdk-java-v2/issues/1633. Could you please check?

[nagsenlkamble]

@zhuoliu I am trying to do in exact same way of yours but I am getting java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Credentials must not be null. Could you please share your working code structure along with which aws-sdk-java and jdk versions your are using?