Aws-sdk-java: Useless error message: "Unable to execute HTTP request: connect timed out"

Created on 21 Feb 2017 · 11Comments · Source: aws/aws-sdk-java

If a "connect timed out" error occurs while using the S3 API, the resulting error message does not include the method name or the URL that the API was using.

As a result, debugging code that uses this API is made significantly more difficult than it should be.

Please fix the error to contain both the method and the URL being used in the error message.

INFO [2017-02-21 12:13:19,608] [360] com.amazonaws.http.AmazonHttpClient: Unable to execute HTTP request: connect timed out ! java.net.SocketTimeoutException: connect timed out ! at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_121] ! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_121] ! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_121] ! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_121] ! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_121] ! at java.net.Socket.connect(Socket.java:576) ~[na:1.7.0_121] ! at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:635) ~[na:1.7.0_121] ! at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:524) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:403) ~[httpclient-4.3.6.jar:4.3.6] ! at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:128) ~[aws-java-sdk-core-1.10.20.jar:na] ! at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.3.6.jar:4.3.6] ! at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) ~[httpclient-4.3.6.jar:4.3.6] ! at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728) [aws-java-sdk-core-1.10.20.jar:na]

response-requested

Source

minfrin

Most helpful comment

Looking at the code from 1.11.94, it looks like the error message is still useless in it's current form:

Exception in thread "main" com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to 10.255.255.1:443 [/10.255.255.1] failed: connect timed out

As I pointed out, we need the method and the URL. Given we are a client request we have both of these bits of information at hand.

Given that v1.11 also doesn't contain the method or URL using this version of the library is not a solution for us.

Please fix the error handling.

minfrin on 23 May 2017

👍3

All 11 comments

Hi @minfrin sorry to see you're having trouble with the SDK. You can also enable wire logging to get additional information, which include information about the request, including the URL, request method, parameters, etc. Will this work for you?

dagnir on 21 Feb 2017

Oops answered too quickly. The request information wouldn't be available if you don't have a connection...

What version of the SDK are you using? 1.11 of the SDK uses 4.5.2 of the Apache Http client, which has slightly better error messages during timeout which includes the URL. As far the method, are you talking about the method on the client? Were not able to find it in the stacktrace? It should be somewhere below the com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728)

dagnir on 21 Feb 2017

This is a commercial product we're using, this isn't our code. The version is aws-java-sdk-s3-1.10.20.

To set the expectation, what we want is the error message to tell the operator clearly and simply what the error was, so that we can quickly determine where the problem lies.

If we're forced to crank out wireshark/tcpdump to debug this, it means a very expensive resource has to be applied to make up for the lack of detail in the error message.

minfrin on 23 Feb 2017

Thanks for the context. It's not clear from your response, but does that mean using 1.11 is not an option? As I mentioned, there's improved error reporting with the new HTTP client that provides more information in the exception message (both the host and the IP it resolved to).

As a comparison, here's the output of a connect timeout for listBuckets using __1.10.20__:

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to execute HTTP request: connect timed out
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:500)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3604)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3557)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:697)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:703)

and here is the same code using __1.11.94__:

Exception in thread "main" com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to 10.255.255.1:443 [/10.255.255.1] failed: connect timed out
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1038)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4141)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4135)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:927)
    at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:933)

dagnir on 23 Feb 2017

Closing this issue due to lack of activity. Please feel free to reopen this if you still require further assistance.

dagnir on 8 May 2017

My experience has been that this is related to https://github.com/aws/aws-sdk-java/issues/833

And the fix proposed here works: https://github.com/aws/aws-sdk-java/issues/833#issuecomment-248528234

$ sudo /var/lib/dpkg/info/ca-certificates-java.postinst configure

christopherdcunha on 23 May 2017

Looking at the code from 1.11.94, it looks like the error message is still useless in it's current form:

Exception in thread "main" com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to 10.255.255.1:443 [/10.255.255.1] failed: connect timed out

As I pointed out, we need the method and the URL. Given we are a client request we have both of these bits of information at hand.

Given that v1.11 also doesn't contain the method or URL using this version of the library is not a solution for us.

Please fix the error handling.

minfrin on 23 May 2017

👍3

Hi @minfrin, can you clarify what you mean by "method"? Are you referring to the HTTP method of the request itself (GET, POST, etc), or the method on the client? If it's the latter, it should be available in the stacktrace for you.

As far as the example error I gave above, it probably wasn't the best one because I just chose an unmapped IP. If you got the error trying to connect to a real host with a name like myhost.com with IP 1.2.3.4, the error message would look like

Exception in thread "main" com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to 1.2.3.4:443 [myhost.com/1.2.3.4] failed: connect timed out

dagnir on 23 May 2017

The HTTP method.

To explain the problem in more detail, in a busy system whose job it is to transfer many files, when something fails it needs to be clear which file failed, and at what point in the process the failure has occurred. This will prevent a whole lot of trouble trying to reproduce a problem to get the additional missing information, but will have enough info to diagnose from the get-go.

minfrin on 5 Aug 2017

Hello Team I am getting error message. Please help me. Thanks

Exception :- com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to s3.amazonaws.com:443 [s3.amazonaws.com/55.516.542.132] failed: connect timed out
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1163)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1109)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1807)