Aws-sdk-java: SQS: Unable to execute HTTP request: Remote host closed connection during handshake: com.amazonaws.SdkClientException

[desmond27]

I wrote a Lambda function to put messages into a SQS queue. This works sometime and other times I see the following exception in the cloudwatch logs:

Unable to execute HTTP request: Remote host closed connection during handshake: com.amazonaws.SdkClientException
com.amazonaws.SdkClientException: Unable to execute HTTP request: Remote host closed connection during handshake
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
    at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2207)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2174)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2163)
    at com.amazonaws.services.sqs.AmazonSQSClient.executeSendMessage(AmazonSQSClient.java:1762)
    at com.amazonaws.services.sqs.AmazonSQSClient.sendMessage(AmazonSQSClient.java:1734)
    at com.desmond.processing.DocumentExtraction.handleRequest(DocumentExtraction.java:310)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:994)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
    at com.amazonaws.http.conn.$Proxy8.connect(Unknown Source)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
    ... 17 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at sun.security.ssl.InputRecord.read(InputRecord.java:505)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
    ... 40 more

This exception occurs when the last line in this snippet executes:

System.out.println("\nPutting DB write job message on queue: "+dbInsertJson.toJSONString());
AmazonSQS sqsClient = AmazonSQSClient.builder().withRegion(Regions.US_EAST_1).build();
SendMessageRequest request = new SendMessageRequest();
request.setQueueUrl(sqsQueueUrl);
request.setMessageBody(dbInsertJson.toJSONString());
request.setMessageGroupId(context.getAwsRequestId()+"_"+UUID.randomUUID());
sqsClient.sendMessage(request);

To the best of my knowledge, I don't think I have missed any further configuration required. Also, since this is executing in a AWS Lambda function, this issue should not occur at all. But the main issue that confounds me is that some messages are getting put into the queue while some are not.

Could it be that the issue is arising from multiple concurrent instances of the lambda function running simultaneously?

[debora-ito]

@desmond27 a quick search on StackOverflow (here and here) shows this can be caused when a security protocol is not accepted by the host.

How are you calling the Lambda Function?

[desmond27]

It's triggered by a SNS notification.

[desmond27]

So, is there any advise on this? I had to rewrite my code to use SNS instead of SQS due to the sporadic nature of this error. Perhaps there is some workaround where SQS can be used reliably?

[desmond27]

Update: I am now getting this same error in my web app running on EC2 when trying to access S3.

[zoewangg]

The error indicates that the connection was closed on the service side before the SDK was able to perform handshake. It should be retryable.

How often did you see the error? Could you add -Djavax.net.debug=ssl property to see if there are any more details in the ssl logs? Could you also check the TLS versions supported on your machine? You can add the following to check:

System.out.println(Arrays.toString(SSLContext.getDefault().getSupportedSSLParameters().getProtocols()))

[desmond27]

Ok, I will try this out and report. However, I still don't understand why I'd get this error in Lambda? Lambda's runtime environment is provided by AWS, it's not something I could set up.

It should be retryable.

If I get such an error in production, I cannot tell users to retry. My best bet is to ensure that this exception never occurs.

How often did you see the error?

Rarely, but sporadically. There is not fixed pattern, and I cannot predict when it might occur again. This is part of a data processing pipeline that could process upwards of 100k files when it goes to production.

Could you add -Djavax.net.debug=ssl property to see if there are any more details in the ssl logs?

I can do this on my local machine, but not on Lambda, is there any way to check this in the Lambda environment?

Could you also check the TLS versions supported on your machine? You can add the following to check:

The output is as follows:

[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]

This is on my local machine though, I have not checked on Lambda.

[desmond27]

Today I received this same exception when publishing a SNS notification from my Lambda function as well. Therefore, currently my Lambda function is unable to work at all.

[desmond27]

Ok, I have observed something that works and might help diagnose this issue: If I publish a message directly on the topic that triggers my Lambda function, the Lambda function will then publish the notification without any issues and the next lambda gets triggered. But when the same Lambda function is triggered programmatically using an SDK, then I get the above exception.

[shubhsharma186]

Hi I am getting this error in my application:
com.amazonaws.SdkClientException: Unable to execute HTTP request: Received close_notify during handshake at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1136) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1082) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[aws-java-sdk-core-1.11.415.jar!/:?] at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2147) ~[aws-java-sdk-sqs-1.11.415.jar!/:?] at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2116) ~[aws-java-sdk-sqs-1.11.415.jar!/:?] at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2105) ~[aws-java-sdk-sqs-1.11.415.jar!/:?] at com.amazonaws.services.sqs.AmazonSQSClient.executeGetQueueUrl(AmazonSQSClient.java:1138) ~[aws-java-sdk-sqs-1.11.415.jar!/:?] at com.amazonaws.services.sqs.AmazonSQSClient.getQueueUrl(AmazonSQSClient.java:1110) ~[aws-java-sdk-sqs-1.11.415.jar!/:?] at com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:260) ~[aws-java-sdk-sqs-1.11.415.jar!/:?] at com.amazonaws.services.sqs.buffered.AmazonSQSBufferedAsyncClient.getQueueUrl(AmazonSQSBufferedAsyncClient.java:750) ~[aws-java-sdk-sqs-1.11.415.jar!/:?]
I am facing same problem as mentioned by @desmond27 this works sometimes and sometime I get the above error.

any updates on this @debora-ito @zoewangg

[aglumova]

In our application we also catch the same problem when we are trying to update visibility timeout in SQS

"message": "com.amazonaws.SdkClientException: Unable to execute HTTP request: Remote host terminated the handshake",
        "stacktrace": [
            "com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)",
            "com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)",
            "com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)",
            "com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)",
            "com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)",
            "com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)",
            "com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)",
            "com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)",
            "com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)",
            "com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2202)",
            "com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2169)",
            "com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2158)",
            "com.amazonaws.services.sqs.AmazonSQSClient.executeChangeMessageVisibility(AmazonSQSClient.java:597)",
            "com.amazonaws.services.sqs.AmazonSQSClient.changeMessageVisibility(AmazonSQSClient.java:568)",

[shubhsharma186]

I added retries with backoff delay and this solves my problem.