Spring-cloud-aws: AWS SQS cannot reconnect to Queue after network connection lost

[sanjaybhatol]

Hi Guys,
Giving a breif view, I am using SQS message listener to process the messages. On application start up I can connect to the queue and process the messages as expected.

Now, if network connection to SQS is lost, even after the connection is re-established the listener is not grabbing the messages from the Queue.

Noticed that org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer.java spins up a new thread to poll the messages on queue using the below code:

private class AsynchronousMessageListener implements Runnable {

    private final QueueAttributes queueAttributes;
    private final String logicalQueueName;

    private AsynchronousMessageListener(String logicalQueueName, QueueAttributes queueAttributes) {
        this.logicalQueueName = logicalQueueName;
        this.queueAttributes = queueAttributes;
    }

    @Override
    public void run() {
        while (isRunning()) {
            ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
            CountDownLatch messageBatchLatch = new CountDownLatch(receiveMessageResult.getMessages().size());
            for (Message message : receiveMessageResult.getMessages()) {
                if (isRunning()) {
                    MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
                    getTaskExecutor().execute(new SignalExecutingRunnable(messageBatchLatch, messageExecutor));
                } else {
                    break;
                }
            }
            try {
                messageBatchLatch.await();
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        }
    }
}

When network connection is lost, below code throws UnknownHostException, which is not handled in the code and causes the thread termination. getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest())

Also once the connection to SQS is lost, we cannot stop the server if we try to shutdown. We have to manually kill the process.

Will be great if you guys could add a catch for that in the code so that even if connection is lost the thread won't terminate itself. Instead when the network connection is back it could recover itself.

Thanks

[aemruli]

Thanks @sanjaybhatol for spotting. We will fix this issue soon with the next spring-cloud version 1.1

[alainsahli]

I introduced a back off time when an exception occurs in a polling thread. The default back off time is set to 10000 milliseconds and can be configured with XML or Java:

<aws-messaging:annotation-driven-queue-listener back-off-time="5000" />

@Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory( {
    SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
    factory.setBackOffTime(5000);
}

[sanjaybhatol]

Thanks for that @alainsahli. I will check it out once released.

[adrianboimvaser]

I've ran into a similar issue:

2015-09-15 04:12:54  INFO - pool-1-thread-5 : Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
    at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:118)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:706)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
    at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2422)
    at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1130)
    at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1678)
    at com.amazonaws.services.sqs.AmazonSQSAsyncClient$23.call(AmazonSQSAsyncClient.java:1676)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Is it the same?

[ghost]

Has this been fixed? I am seeing the same behavior in 1.1.0-RC1.

[senthilaru]

I m getting this issue in - spring-cloud-starter-aws » 1.0.6.RELEASE

[StealthyDev]

I'm facing this error in 1.1.1.BUILD-SNAPSHOT