Spring-cloud-aws: @MessageMapping -> after 1 day -> 413 Request Entity Too Large

[mrandi]

I used in my project the very usefull annotation @MessageMapping("${sqs.url}").
But after a day I get tons of exceptions, the first exception I received is:

com.amazonaws.AmazonServiceException: Unable to unmarshall error response (HTTP content length exceeded 1662976 bytes.) (Service: AmazonSQS; Status Code: 413; Error Code: 413 Request Entity Too Large; Request ID: null)

than only this:

Service: AmazonSQS; Status Code: 413; Error Code: 413 Request Entity Too Large; Request ID: null)

I tried to debug in order to find a solution for this error. But I couldn't reproduce it locally.

So I tried different times to deploy this application but after different hour (on average ~1 day ) I get this exceptions. Is very interesting that I always get the same exception, also the 1662976 bytes are always the same.

I have no idea how can I fix my problem. Right now I implemented a sqs poller by myself and the application works very well, but would be nice to use the spring one.

[alainsahli]

Hi @mrandi, the payload you are trying to send is around 1.5MB. SQS supports payloads up to 256KB, so I suppose that this error happens because the payload you're trying to send is too big.

If you want to send bigger payloads, you could use the Amazon SQS Extended Client Library but it is not supported by Spring Cloud AWS so far and you'll have to configure the client yourself.

[mrandi]

Hi @alainsahli
I also think the same, but I cannot understand how the request look like when I using the annotation @MessageMapping

I write myself only the poller and still used the client from spring. Seems to be a problem of the spring poller when build the request.

---

INFO: I forgot to mention that all my messages are S3 notification. Average 1.9 KB.

[micjaesc]

Hi, any progress with magic status code 413. We have exactly same error message and case as you described @mrandi . Do you still use own puller ?

[mrandi]

@micjaesc yes, still using an own poller. No idea if is now fixed.

[nsmolenskii]

I got same problem during integration with app-dynamic-agent, that add request/response handler with asking for additional header.

It was happened because of many factors:

• usage header names as list instead of set by aws-java-sdk-sqs ReceiveMessageRequest#withMessageAttributeNames
• usage recieve request as constant by spring-cloud-aws-messaging AsynchronousMessageListener#run
• adding required header name by app-dynamic-agent without checking for existing one (sources are not available)

As result aws-sqs-client combine header for _receive request_ with _header names_ from previous request plus new from _request/response handler_ again again and again.

A possible solutions could be:

• fix spring-cloud-aws-messaging logic of QueueAttributes#getReceiveMessageRequest from getter to factory method
• fix aws-java-sdk-sqs logic of ReceiveMessageRequest.#withMessageAttributeNames and use set insead
• fix external request/response handlers logic

[nsmolenskii]

A most easiest work-around could be aspect, that cover:

• AmazonSQS#receiveMessage(ReceiveMessageRequest)
• AmazonSQSAsync#receiveMessageAsync(ReceiveMessageRequest)

import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.springframework.stereotype.Component;

import java.util.LinkedHashSet;

@Aspect
@Component
public class ReceiveMessageRequestAspect {
    @Before("execution(* com.amazonaws.services.sqs.AmazonSQS*.receiveMessage*(com.amazonaws.services.sqs.model.ReceiveMessageRequest)) && args(request)")
    public void onReceiveMessage(ReceiveMessageRequest request) throws Throwable {
        request.setAttributeNames(new LinkedHashSet<>(request.getAttributeNames()));
        request.setMessageAttributeNames(new LinkedHashSet<>(request.getMessageAttributeNames()));
    }
}

[alainsahli]

Hi,

I would like to reproduce this issue before fixing it, but I cannot reproduce it...

I used the advice above to log the size of both lists (attributes and attribute names) but the size is always 1 even when I send hundreds of SQS messages. You can check my test code here: https://github.com/spring-cloud-samples/aws-refapp/commit/93ff4ad8bc2b32d1c4a39e1cf12ccc3c0321f644. If you want to test it, you have to checkout the branch "issue-111".

[nsmolenskii]

@alainsahli, my apologises for possible misunderstanding. As I said, the problem is related to 3rd party library/java-agent/whatever (are not able to be changed, but have to be used) that could make add addition header name to request (in meaning of one-time-usage request) and make a damage to whole application (in meaning of reusable request).

You can add some code like that, to emulate 3rd party side effect, on 20k+ headers it will make a result :

@Aspect
@Component
public class ReceiveMessageRequest3rdPartyExampleAspect {
    @Before("execution(* com.amazonaws.services.sqs.AmazonSQS*.receiveMessage*(com.amazonaws.services.sqs.model.ReceiveMessageRequest)) && args(request)")
    public void onReceiveMessage(ReceiveMessageRequest request) throws Throwable {
        request.withMessageAttributeNames("3rd-party-example-header");
        request.withAttributeNames("3rd-party-example-header");
    }
}

[alainsahli]

Hi @mrandi, @nsmolenskii,

I changed the implementation of the container by always returning a new instance of ReceiveMessageRequest to avoid potential side effects produced by third party code (369d931f986105276e9cfbec6ee2a9a6133702bb). It would be great if you could try it out and reopen the issue in case it still doesn't solve the issue.

[nsmolenskii]

@alainsahli I was check it from '1.1.1.BUILD-SNAPSHOT' version and it works perfect (at least for me).
Thanks.

[rpbrehm]

@mrandi, I have run into the same issue with message size > 256mb and Spring AWS Cloud not supporting AWS extended SQS client library. I am developing the same logic but could make use of the poller that you developed. Did you publish your poller code in a public repo anywhere?

Error explanation from production log: Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.BatchRequestTooLong; Request ID: _); nested exception is com.amazonaws.services.sqs.model.BatchRequestTooLongException: Batch requests cannot be longer than 262144 bytes. You have sent 262203 bytes.

[mrandi]

Hi @rpbrehm,
I needed some time to extract it, here is the repo: https://github.com/mrandi/sqs-poller
And here can where you can handle the message: https://github.com/mrandi/sqs-poller/blob/master/src/main/java/io/github/mrandi/handler/SqsMessageHandlerImpl.java#L11

I did not have an sqs for testing it, but after configuring the region and sqs url here: https://github.com/mrandi/sqs-poller/blob/master/src/main/resources/config/application.yml should work.

Let me know if was ok and even if is working ;)