Aws-sdk-java-v2: S3AsyncClient fails with EOFException when executing HeadObjectRequest for gzip encoded object

[richardtitze]

I am trying to execute a HeadObjectRequest for objects stored in S3. All objects are compressed and do have Content-Encoding: gzip.

Expected Behavior

Just head info's are returned - and no exception is thrown.

Current Behavior

This stacktrace is logged:
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:816) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:797) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at com.example.demos3reactive.DemoS3ReactiveApplication.main(DemoS3ReactiveApplication.java:13) [classes/:na]
Caused by: java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_202]
at com.example.demos3reactive.DemoS3ReactiveApplication.run(DemoS3ReactiveApplication.java:25) [classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:813) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
... 5 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkClientException: null
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:123) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:105) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_202]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$1(MakeAsyncHttpRequestStage.java:137) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_202]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_202]
Caused by: java.io.UncheckedIOException: java.io.EOFException
at software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException(FunctionalUtils.java:180) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:110) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.decompressing(Crc32Validation.java:85) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.process(Crc32Validation.java:62) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.validate(Crc32Validation.java:44) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:52) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_202]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:180) ~[netty-nio-client-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:67) ~[netty-nio-client-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:298) ~[netty-nio-client-2.5.28.jar:na]
at com.typesafe.netty.HandlerPublisher.publishMessage(HandlerPublisher.java:362) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.flushBuffer(HandlerPublisher.java:304) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.receivedDemand(HandlerPublisher.java:258) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.access$200(HandlerPublisher.java:41) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:452) ~[netty-reactive-streams-2.0.0.jar:na]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
... 1 common frames omitted
Caused by: java.io.EOFException: null
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) ~[na:1.8.0_202]
at software.amazon.awssdk.core.http.Crc32Validation.lambda$decompressing$2(Crc32Validation.java:85) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108) ~[utils-2.5.28.jar:na]
... 24 common frames omitted

Steps to Reproduce

package com.example.demos3reactive;

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;

@SpringBootApplication
public class DemoS3ReactiveApplication implements CommandLineRunner {

public static void main(String[] args) {
    SpringApplication.run(DemoS3ReactiveApplication.class, args);
}

@Override
public void run(String... args) throws Exception {
    S3AsyncClient client = S3AsyncClient.builder()
            .serviceConfiguration(builder -> builder.checksumValidationEnabled(false))
            .build();
    HeadObjectRequest hor = HeadObjectRequest.builder()
            .bucket("a_valid_bucket")
            .key("a_key") // object stored has Content-Encoding: gzip
            .build();
    client.headObject(hor).get();
}

}

Context

I'm trying to create a reactive spring microservice in front of S3.

Your Environment

SDK used: aws-sdk-java-v2 2.5.28
Java: Amazon Corretto 8
OS: MacOS

[varunnvs92]

I could not repo the issue. I used the exact same code except I didn't use spring boot.

1) Is there anything special about the object? I tried with empty and non-empty objects but can't reproduce.
2) Did you try with the latest version of the SDK?
3) Can you try with plain Java code and check if you still see the issue

[richardtitze]

I have created a self-contained example of what i'm trying to achieve. You can find it here: https://github.com/richardtitze/demos/tree/master/demo-s3-reactive

AsyncClientTest starts up a minio server, creates a bucket, puts a gzip encoded object in the bucket, and then executes a head request. If you run "mvn test" you'll see that the one test fails with the same error i reported before. I also upgraded the v2 sdk dependency to 2.5.30

[alexklibisz]

@richardtitze I'm getting the same thing. We're basically just calling headObject on a gzipped json file, and that yields a chain of CompletionException -> SdkException -> EofException. I don't quite have the code in a state where I can offer a reproducible example.

[jkeillor]

I work with @alexklibisz and have looked into this issue some as well.

This issue only happens when the object metadata indicates content-encoding: gzip. CR32Validation attempts to unzip the content body to do the validation which fails because for http HEAD the response content has been set to a zero-length byte array.

One solution would be to update AsyncResponseHandler to only set the content on SdkHttpFullResonse when a non-empty response body exists.

[oripwk]

Hitting this issue too. I have to idea what to do.

Is there a fix in the pipeline?

I also do not understand to workaround, if you could provide a code example that would be fantastic

[jkeillor]

@oripwk we were using HEAD to check whether an object exists and were able to work around this issue by using the listObjects operation for that purpose instead.

Here's a gist with a diff showing the change to AsyncResponseHandler that I mentioned in my earlier comment: https://gist.github.com/jkeillor/9b41a4b36ba0ff1613421ccf5c0f0566

[oripwk]

@jkeillor So I guess I cannot use list-objects since I use head-object to inspect metadata.

Do you know of any way of changing the AsyncResponseHandler without forking?

[debora-ito]

Apologies for the long delay in response in this issue, thank you @oripwk for calling our attention to this.
I can reproduce the issue with the latest version of the SDK.

@jkeillor would you like to submit a PR?

[oripwk]

Hi guys,

FWIW, I've put together a workaround which is quite effective.

The solution is to use getObject with a custom AsyncResponseTransformer. Then, you have access to all the fields as in headObject:

object Main {
  def main(args: Array[String]): Unit = {
    …
    val metadata = s3.getObject(GetObjectRequest.builder()
      .bucket("bucket")
      .key("key")
      .build(), new WorkaroundAsyncTransformer
    ).get().metadata()
    …
  }
}

class WorkaroundAsyncTransformer extends AsyncResponseTransformer[GetObjectResponse, GetObjectResponse] {
  @volatile private var future: CompletableFuture[GetObjectResponse] = _
    override def prepare(): CompletableFuture[GetObjectResponse] = {
      future = new CompletableFuture[GetObjectResponse]()
      future
    }
    override def onResponse(response: GetObjectResponse): Unit = future.complete(response)
    override def onStream(publisher: SdkPublisher[ByteBuffer]): Unit = publisher.subscribe(new DrainingSubscriber[ByteBuffer])
    override def exceptionOccurred(error: Throwable): Unit = future.completeExceptionally(error)
}

[productivityindustries]

sync client is still affected by this issue

[debora-ito]

Fix for the S3AsyncClient was released in SDK 2.15.46 2.15.51.
Since the original issue was addressed I'll go ahead and close this.

@productivityindustries: please open a new issue describing the problems you're seeing.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[oripwk]

Fix for the S3AsyncClient was released in SDK 2.15.46.
Since the original issue was addressed I'll go ahead and close this.

@productivityindustries: please open a new issue describing the problems you're seeing.

@debora-ito For me it works only for versions starting from 2.15.51

[debora-ito]

@oripwk you're right, I corrected my previous comment, thank you!