Aws-sdk-java-v2: "Host is null" error when using S3Client

[jsvede]

When using the S3Client as described in the documentation, https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/examples-s3-objects.html#list-object, it is not specified that Region is a required value when building the S3Client object.

Failing to specify a valid region will cause the client to error out with an error message that doesn't describe the actual issue.

This seems related to this issue: https://github.com/aws/aws-sdk-java-v2/issues/453 . However it is not identical.

Expected Behavior

When creating the S3Client, it is not clear that the Region is required field when connecting to S3. At runtime, if the code fails to specific valid region, there should be a descriptive error message saying that region is required.

Current Behavior

Currently if you build a client as follows:

        String accessKeyId = "accessKey"; //actual key removed
        String secretAccessKey = "secret"; //actual secret removed
        Region region = software.amazon.awssdk.regions.Region.of("AP_EAST_1"); //note the bad String value
        AwsBasicCredentials awsCredentials = AwsBasicCredentials.create(accessKeyId, secretAccessKey);

        S3Client s3Client = S3Client.builder()
                .credentialsProvider(StaticCredentialsProvider.create(awsCredentials))
                .region(region)
                .build();

Note that the Region String value is incorrect ("the correct region would be "ap-east-1"). The Javadoc doesn't provide any visibility into the String value: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html .

And then go on to access a bucket (assuming all else is correct), you will get the following error:

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to marshall request to JSON: host must not be null.
    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
    at software.amazon.awssdk.services.s3.transform.PutObjectRequestMarshaller.marshall(PutObjectRequestMarshaller.java:53)
    at software.amazon.awssdk.services.s3.transform.PutObjectRequestMarshaller.marshall(PutObjectRequestMarshaller.java:31)
    at software.amazon.awssdk.core.runtime.transform.StreamingRequestMarshaller.marshall(StreamingRequestMarshaller.java:48)
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:69)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:132)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:102)
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:7376)
    at com.charter.scp.s3.v2.test.S3TestSdkV2.main(S3TestSdkV2.java:56)
Caused by: java.lang.NullPointerException: host must not be null.
    at software.amazon.awssdk.utils.Validate.paramNotNull(Validate.java:117)
    at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.<init>(DefaultSdkHttpFullRequest.java:53)
    at software.amazon.awssdk.http.DefaultSdkHttpFullRequest.<init>(DefaultSdkHttpFullRequest.java:41)
    at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:331)
    at software.amazon.awssdk.http.DefaultSdkHttpFullRequest$Builder.build(DefaultSdkHttpFullRequest.java:170)
    at software.amazon.awssdk.protocols.xml.internal.marshall.XmlProtocolMarshaller.finishMarshalling(XmlProtocolMarshaller.java:122)
    at software.amazon.awssdk.protocols.xml.internal.marshall.XmlProtocolMarshaller.marshall(XmlProtocolMarshaller.java:86)
    at software.amazon.awssdk.protocols.xml.internal.marshall.XmlProtocolMarshaller.marshall(XmlProtocolMarshaller.java:49)
    at software.amazon.awssdk.services.s3.transform.PutObjectRequestMarshaller.marshall(PutObjectRequestMarshaller.java:51)

The main issue here is that the top level error message does not provide the developer very much to go on:

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to marshall request to JSON: host must not be null.

The nested exception is also not helpful:

Caused by: java.lang.NullPointerException: host must not be null.

Please provide a clearer error message and update the documentation to make it clear that Region is required.

Possible Solution

This issue is resolved by specifying a correct region.

Steps to Reproduce (for bugs)

1. Build a simple S3 client application to list a bucket or write a file.
2. Specify an incorrect region or do not specify one at all
3. Run the code, you should see the exception.

Sample Code:

package aws.s3.test.case;

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.S3Object;

import java.io.File;
import java.io.FileInputStream;
import java.util.List;
import java.util.ListIterator;


public class S3TestSdkV2 {

    public static void main(String... args) throws Exception {

        String accessKeyId = "";
        String secretAccessKey = "";
        Region region = software.amazon.awssdk.regions.Region.of(""); //Specify an incorrect region string

        AwsBasicCredentials awsCredentials = AwsBasicCredentials.create(accessKeyId, secretAccessKey);

        S3Client s3Client = S3Client.builder()
                .credentialsProvider(
                        StaticCredentialsProvider.create(awsCredentials))
                .region(region)
                .build();

        // provide your own bucket and fq local file name
        String bucketName = "";
        String localFileName = "";
        String bucketPath = "";

        File file = new File(localFileName);

        FileInputStream fis = new FileInputStream(file);

        String objectKey = bucketPath + "/" + file.getName();

        PutObjectRequest putObjectRequest = PutObjectRequest.builder().bucket(bucketName).key(objectKey).build();

        RequestBody requestBody = RequestBody.fromBytes(fis.readAllBytes());

        PutObjectResponse response = s3Client.putObject(putObjectRequest, requestBody);

        System.out.println(String.format("wrote file %s to bucket %s with response code %s", objectKey, bucketName, response.sdkHttpResponse().statusCode()));

        GetObjectRequest getObjectRequest = GetObjectRequest.builder().bucket(bucketName).key(bucketPath).build();

        ListObjectsRequest listObjects = ListObjectsRequest
                .builder()
                .bucket(bucketName)
                .prefix(bucketPath)
                .build();

        ListObjectsResponse res = s3Client.listObjects(listObjects);
        List<S3Object> objects = res.contents();

        for (ListIterator iterVals = objects.listIterator(); iterVals.hasNext(); ) {
            S3Object myValue = (S3Object) iterVals.next();
            System.out.print("\n The name of the key is " + myValue.key());
            System.out.print("\n The object is " + calKb(myValue.size()) + " KBs");
            System.out.print("\n The owner is " + myValue.owner());
        }

    }

    private static long calKb(Long val) {
        return val/1024;

    }
}

Context

For our project we need to specify the bucket information in configuration details and cannot hardcode the region meaning we cannot use Region.US_EAST_1 in code.

Your Environment

• AWS Java SDK version used:

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>2.10.56</version>
        </dependency>

• JDK version used: 11
• Operating System and version: MacOS 10.14.6

[debora-ito]

Thank you for the detailed report @jsvede. We'll work in making the error message more clear in this case.

As for the documentation, the ListObjects example in the link you provided shows a region being set in the client, and we also have one section in the AWS SDK for Java Developer Guide dedicated to region selection - https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/java-dg-region-selection.html

Where else do you recommend we add this information in the documentation?

[jsvede]

@debora-ito - I think just adding a clearer error message would solve the problem. The message "host must not be null" is confusing (at least to me) since I didn't set a proper hostname, just a bucket name.

Documenting it is great, but devs as a group aren't always known for reading everything. Also, while the example does show using a region it's not clear that it's required. The builder.build() method seems like to should throw an error when region isn't specified.