Aws-sdk-java: AWS S3 Client throws Unable to execute HTTP request: The target server failed to respond

Created on 28 Jun 2017 · 18Comments · Source: aws/aws-sdk-java

We recently changed our AWS S3 Client instantiation to use aws-java-sdk version 1.1.150 in order to use the AWS recommended Default Region Provider Chain, following details in http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html

The S3 client is instantiated as follows:
AwsS3ClientBuilder.standard().withForceGlobalBucketAccessEnabled(true).build();

We are seeing a large number errors as below when getting objects from S3 using this client.

Exception stack trace:

com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1378)

We did some further analysis on our side and were able to conclude the following:

(1) Using aws-java-sdk version 1.10.54, and creating S3 client using the constructor as follows does NOT exhibit any exceptions/errors when calling getObject() on the client. This was the version that we were using before upgrading. Code used for creating client and get object:

AmazonS3 s3Client = new AmazonS3Client();
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));

(2) Using aws-java-sdk version 1.11.150, and using the code snippet same as above (creating S3 client using the constructor), we see the exception.

(3) Using aws-java-sdk version 1.11.150, and creating S3 client using the Client Builder (as recommended in your docs), we see the exception. Code used for creating client and get object:

AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withForceGlobalBucketAccessEnabled(true).build();
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, key));

Please review and advise.

guidance response-requested

Source

sballa12

Most helpful comment

Was this issue ever resolved?

gavinlreynolds on 2 Nov 2018

👍4

All 18 comments

@sballa12 Can you you please provide wire logs so we can see the request the client is making? Please ensure you remove any sensitive information.

dagnir on 28 Jun 2017

Also is it just GET object that fails? Do other operations like list buckets work?

dagnir on 28 Jun 2017

Adding wire log of a failed request to get object here:

2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "GET kkk HTTP/1.1[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "Host: bbb.s3.amazonaws.com[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "Authorization: AWS4-HMAC-SHA256 Credential=ccc/20170628/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=sss[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "X-Amz-Date: 20170628T174112Z[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "User-Agent: aws-sdk-java/1.11.150 Windows_7/6.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.40-b25/1.8.0_40[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "amz-sdk-invocation-id: yyy[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "Content-Type: application/octet-stream[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "Connection: Keep-Alive[\r][\n]"
2017-06-28 13:41:12,325 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 >> "[\r][\n]"
2017-06-28 13:41:12,326 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-57 << "end of stream"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "GET kkk HTTP/1.1[\r][\n]"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "Host: bbb.s3.amazonaws.com[\r][\n]"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "Authorization: AWS4-HMAC-SHA256 Credential=ccc/20170628/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=sss[\r][\n]"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "X-Amz-Date: 20170628T174112Z[\r][\n]"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "User-Agent: aws-sdk-java/1.11.150 Windows_7/6.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.40-b25/1.8.0_40[\r][\n]"
2017-06-28 13:41:12,371 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "amz-sdk-invocation-id: yyy[\r][\n]"
2017-06-28 13:41:12,372 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "amz-sdk-retry: 1/43/495[\r][\n]"
2017-06-28 13:41:12,372 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "Content-Type: application/octet-stream[\r][\n]"
2017-06-28 13:41:12,372 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "Connection: Keep-Alive[\r][\n]"
2017-06-28 13:41:12,372 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 >> "[\r][\n]"
2017-06-28 13:41:12,375 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-59 << "end of stream"
2017-06-28 13:41:12,448 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "GET kkk HTTP/1.1[\r][\n]"
2017-06-28 13:41:12,448 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "Host: bbb.s3.amazonaws.com[\r][\n]"
2017-06-28 13:41:12,448 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
2017-06-28 13:41:12,448 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "Authorization: AWS4-HMAC-SHA256 Credential=ccc/20170628/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=sss[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "X-Amz-Date: 20170628T174112Z[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "User-Agent: aws-sdk-java/1.11.150 Windows_7/6.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.40-b25/1.8.0_40[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "amz-sdk-invocation-id: yyy[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "amz-sdk-retry: 2/72/490[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "Content-Type: application/octet-stream[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "Connection: Keep-Alive[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 >> "[\r][\n]"
2017-06-28 13:41:12,449 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-53 << "end of stream"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "GET kkk HTTP/1.1[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "Host: bbb.s3.amazonaws.com[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "Authorization: AWS4-HMAC-SHA256 Credential=ccc/20170628/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=sss[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "X-Amz-Date: 20170628T174112Z[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "User-Agent: aws-sdk-java/1.11.150 Windows_7/6.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.40-b25/1.8.0_40[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "amz-sdk-invocation-id: yyy[\r][\n]"
2017-06-28 13:41:12,708 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "amz-sdk-retry: 3/256/485[\r][\n]"
2017-06-28 13:41:12,709 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "Content-Type: application/octet-stream[\r][\n]"
2017-06-28 13:41:12,709 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "Connection: Keep-Alive[\r][\n]"
2017-06-28 13:41:12,709 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 >> "[\r][\n]"
2017-06-28 13:41:12,711 [http-bio-8080-exec-5] DEBUG o.a.h.wire http-outgoing-38 << "end of stream"
com.amazonaws.SdkClientException: Unable to execute HTTP request: The target server failed to respond
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1378)

sballa12 on 28 Jun 2017

@sballa12 I could not reproduce this locally. Do you see this for every get object request only at certain times?

Some Googling lead me to this https://issues.apache.org/jira/browse/HTTPCLIENT-1610. Do you by any chance have Apache HttpClient 4.4 as a dependency? You can check to make sure by using mvn d ependency:tree

dagnir on 29 Jun 2017

@dagnir Thank you for pointing us in the right direction! We were using Apache HttpClient 4.4. After upgrading to Apache HttpClient 4.5.2 and Apache HttpCore 4.4.4, this problem went away. Thanks again.

sballa12 on 30 Jun 2017

Glad to help! Closing this issue.

dagnir on 30 Jun 2017

I am using aws-sdk 1.11.233 with HttpCLient 4.5.3 and HttpCore 4.4.8. And Still facing the same issue any idea?

deepakagrawal121 on 15 Dec 2017

We are also facing the exact same issue. Any update on this. Or should we update the httpClient library ?

ImranBijapuri on 3 Jul 2018

I am also facing the same issue.

yifanguo247 on 16 Jul 2018

Likewise. These started happening for us very sporadically.

jfernandez on 16 Jul 2018

We are facing the same issue.

Ev3rlasting on 19 Jul 2018

We are facing the same issue.

abbccdda on 25 Jul 2018

@abbccdda @jfernandez @Ev3rlasting @yifanguo247 @ImranBijapuri
Could you provide the following information?

SDK version
Java Version
Apache http client version
When did you start to see this issue and does it happen sporadically or all the time?
Have you made any code changes or SDK upgrades before seeing this error?

zoewangg on 26 Jul 2018

Was this issue ever resolved?

gavinlreynolds on 2 Nov 2018

👍4

This is happening for us too. It happens sporadically when downloading a large file - around 1.2GB (i.e. lots of parts being downloaded by TransferManager).
SDK version 1.11.501
Java version 1.8.0_202
Apache http client 4.5.6

No code changes or SDK upgrades.

akhaku on 9 Apr 2019

We're still running in to this pretty often, even with the latest versions of Apache HTTP Client

notenbrood on 17 Dec 2019

Could this be related to:
https://stackoverflow.com/questions/23912407/frequent-nohttpresponseexception-with-amazons3-getobjectrequest-getobjectconte?

In particular:

Fortunately, the AWS SDK for Java has built-in facilities for retrying requests. Unfortunately, they do not cover the case of SocketExceptions [...]

Some things worth trying (untested, please be sure to understand these if applying them in production):

AmazonDynamoDBClientBuilder.standard()
                    .withClientConfiguration(
                            new ClientConfiguration()
                                    .withValidateAfterInactivityMillis(200))

```java
AmazonDynamoDBClientBuilder.standard()
.withClientConfiguration(
new ClientConfiguration()
.withConnectionTTL(0))

```java
AmazonDynamoDBClientBuilder.standard()
                    .withClientConfiguration(
                            new ClientConfiguration()
                                    .withMaxErrorRetry(10)
                                    .withConnectionTimeout(10_1000)
                                    .withSocketTimeout(10_000)
                                    .withTcpKeepAlive(true))

bobtiernay-okta on 30 Apr 2020

Issue still found in below version, and seems d) is better than before.

a)
Java Version. -- 1.8 Amazon Corretto 8
AWS SDK version - Java 1.11.398
HTTP Client version - 4.5.9
HTTP core 4.4.11

b)
Java Version. -- 1.8 Amazon Corretto 8
AWS SDK version - Java 1.11.839
HTTP Client version - 4.5.12
HTTP core 4.4.13

c)
Java Version. -- 1.8 Amazon Corretto 8
AWS SDK version - Java 1.11.839
HTTP Client version - 4.5.10
HTTP core 4.4.11

d)
Java Version. -- 1.8 Amazon Corretto 8
AWS SDK version - Java 1.11.839
HTTP Client version - 4.5.2
HTTP core 4.4.4

Log:
org.apache.http.NoHttpResponseException: The target server failed to respond
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5105)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5051)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
at com.refinitiv.etlcommon.aws.utils.S3Util.initContent$1(S3Util.scala:31)
at com.refinitiv.etlcommon.aws.utils.S3Util.$anonfun$getObject$2(S3Util.scala:43)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at com.refinitiv.etlcommon.aws.utils.S3Util.$anonfun$getObject$1(S3Util.scala:40)
at scala.util.control.Breaks.breakable(Breaks.scala:42)
at com.refinitiv.etlcommon.aws.utils.S3Util.getObject(S3Util.scala:40)
at com.refinitiv.etlcommon.ObjectServicesUtil$.extractblob(ObjectServicesUtil.scala:157)
at com.refinitiv.etlcommon.ObjectServicesUtil$.attributeMap2Obj(ObjectServicesUtil.scala:337)
at com.refinitiv.etlcommon.ObjectServicesImp.$anonfun$getChangedObjectscs$1(ObjectServicesImp.scala:207)
at scala.collection.immutable.List.map(List.scala:286)
at com.refinitiv.etlcommon.aws.utils.DynamoDBUtil.queryHashKeyWithKeyFilter(DynamoDBUtil.scala:219)
at com.refinitiv.etlcommon.ObjectServicesUtil$.$anonfun$paraQueryObjByIdInCurrentView$2(ObjectServicesUtil.scala:699)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedThreadPool$TimingRunnable.run(ExecutorInstrumentation.scala:437)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)