Spring-cloud-aws: SQS Message Attributes - Number types

[sql4bucks]

When a message attribute includes a Number type but does not specify the _optional_ custom type, the method getNumberValue(MessageAttributesValue) in QueueMessageUtils (line 93) causes a StringIndexOutOfBoundsException when trying to parse the "numberTypeClass" out of the DataType field of the attribute.

Since this is an optional part of the DataType field in the MessageAttribute, the code should gracefully handle when it is not present instead of blindly trying to parse it out. If it is not present, you could default to parsing the value as a long or whatever numeric type seems to be appropriate as a default.

This is a blocker for me at the moment, since the library that is putting the message in the queue is not in my control and is not specifying the optional piece as part of the DataType of the message attribute. You could potentially argue he should be adding that number class type, but I would think that Spring AWS Cloud should be more resilient than just failing to handle the message.

[alainsahli]

Hi @sql4bucks

Thanks for reporting this issue. We are going to fix this by using a default number type to parse the value.

[sql4bucks]

Thank you very much, great news. I appreciate your efforts and consideration.

[mnimer]

It looks like Number headers in SQS messages only work with the format Number., such as "Number.java.lang.Integer". While you're fixing this can you add a simple mapping lookup for the different Number subclasses, so Number.Integer works.

Or at least so the examples from Amazon's documentation work
" types similar to the following: Number.byte, Number.short, Number.int, and Number.float. "

One more thing, the first line in getNumberValue() should be in the try/catch too, so a proper MessagingException is thrown when the header value is wrong.
"String numberType = value.getDataType().substring(MessageAttributeDataTypes.NUMBER.length() + 1);"

[cthulu]

It'd be great to have the ability to disable the smartness of the Number parsing completely. A library is sending Number.long as attribute and that is failing to get properly cast it to anything in Java so that the whole message processing fails.

[bplies-ATX]

Encountered this issue today when using this library for sending SQS messages. A timestamp MessageAttribute of type long was being written out as

"timestamp": {
   "StringValue": "1508258085338",
   "DataType": "Number.java.lang.Long"
} 

For platform/implementation-agnostic microservices we cannot assume that a Consumer can make sense of Java types. As a workaround ended up using AOP to intercept and change this DataType before the underlying AmazonSQSClient tries to send it:

    @Aspect
    @Component
    protected static class AspectConfiguration {
        @Before(value = "execution(* com.amazonaws.services.sqs.AmazonSQS*Client.sendMessage*(com.amazonaws.services.sqs.model.SendMessageRequest,..)) && args(request,..)",
                argNames = "request")
        public void before(SendMessageRequest request) {
            Map<String, MessageAttributeValue> messageAttributes = request.getMessageAttributes();
            if (messageAttributes == null) return;
            MessageAttributeValue timestamp = messageAttributes.get("timestamp");
            if (timestamp == null) return;

            timestamp.setDataType("Number.long");
        }
    }

[igor-sokolov]

Hi guys,
is there any news about the fix for this issue? I agree with bplies-IAS, it is not best practice to make such assumption that a client will be Java application or will recognize Java specific custom data type. We basically bumped out that for example, Tibco plugin recognizes only three primitive types: String, Number and Binary. I would say it is better to make all header produced by Spring Cloud AWS SQS only of those three basic types and that should be only based on an agreement between sender-receiver to use any custom types.

[volkert-fastned]

Until this issue is fixed, can you please at least log an explanatory warning to the console?

Perhaps something like this:
Ignoring message with attribute "timestamp" having unsupported DataType "Number.java.lang.Long"

That way, developers immediately know what's going on.

We spent a lot of time trying to figure out why our messages weren't being consumed by the @JmsListener, because we didn't find any stacktraces, errors or warnings anywhere in the log. Also, googling for this didn't yield any immediate obvious explanations.

Thank you for considering this.

[volkert-fastned]

@bplies-IAS Thank you for your suggested workaround. Note however that although it indeed fixes it for listeners that are annotated with @JmsListener, it unfortunately breaks listeners that are annotated with @SqsListener:

Exception in thread "simpleMessageListenerContainer-28" org.springframework.messaging.MessagingException: Message attribute with value '1559303963060' and data type 'Number.long' could not be converted into a Number because target class was not found.; nested exception is java.lang.ClassNotFoundException: long
    at org.springframework.cloud.aws.messaging.core.QueueMessageUtils.getNumberValue(QueueMessageUtils.java:98)
    at org.springframework.cloud.aws.messaging.core.QueueMessageUtils.getMessageAttributesAsMessageHeaders(QueueMessageUtils.java:80)
    at org.springframework.cloud.aws.messaging.core.QueueMessageUtils.createMessage(QueueMessageUtils.java:56)
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$MessageExecutor.getMessageForExecution(SimpleMessageListenerContainer.java:380)
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$MessageExecutor.run(SimpleMessageListenerContainer.java:340)
    at org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer$SignalExecutingRunnable.run(SimpleMessageListenerContainer.java:397)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:825)
Caused by: java.lang.ClassNotFoundException: long
    at java.base/java.lang.Class.forNameImpl(Native Method)
    at java.base/java.lang.Class.forName(Class.java:345)
    at org.springframework.cloud.aws.messaging.core.QueueMessageUtils.getNumberValue(QueueMessageUtils.java:95)

So @JmsListener supports the data type Number.long, but not Number.java.lang.Long, whereas in the case of @SqsListener, it's exactly the other way around. I guess it's not typical to use both listener mechanisms within the same project or organization, but you might want to take this into account when you're in the progress of migrating from the one to the other. I guess right now the only way to fix or work around this for both standards is to avoid using long fields in whatever you end up serializing to an SQS message. Is there perhaps a way to configure spring-cloud-aws to send timestamps in ISO 8601 string format instead of (milli)seconds since epoch?

See also this pull request by @bvanloocke that hopefully fixes this problem: https://github.com/spring-cloud/spring-cloud-aws/pull/297

[msarun]

@alainsahli any ballpark dates when the fix will be available? don't see the fix in 2.2.5.RELEASE

[maciejwalkowiak]

We are releasing 2.3 RC tomorrow. Was supposed to be today, but we faced some obstacles.

[msarun]

thanks @maciejwalkowiak looking forward to the fix in 2.3 RC. Please let us know once it is available

[fblanke]

@maciejwalkowiak Any news regarding the 2.3 RC? Me and my team are eagerly awaiting the release :) Thanks for a great lib which really simplifies a lot of things for us

[fblanke]

For anyone else ending up here, please note that the project have moved to a new repo which is https://github.com/awspring/spring-cloud-aws. I am pretty sure that I made a reading mistake before, when asking on ETA for 2.3 release.

[maciejwalkowiak]

Apologies @fblanke. 2.3 has been released, recently we even published 2.3.1 with a bugfix.