Boto3: SQS returns uncapitalized dataType, stringValue, but Boto3 requires capitalization

[Tylermarques]

When using SQS as a Lambda trigger, the MessageAttributes return values with "dataType" and "stringValue", starting with lower case letters. When trying to send to an SQS queue, Boto3 requires capitalized "DataType" and "StringValue" to be part of the messageAttributes.

This is confusing and I think should be altered, as it looks odd to capitalize differently depending on whether you are sending or receiving a message.

Example:
Sent the following to SQS queue

'messageAttributes': {
        'startTime': {
          'StringValue': '20190929T000000',
          'DataType': 'String'
        },
        'endTime': {
          'StringValue': '20191006T000000',
          'DataType': 'String'
        },

Received the following in the Lambda event

'messageAttributes': {
        'startTime': {
          'stringValue': '20190929T000000',
          'stringListValues': [],
          'binaryListValues': [],
          'dataType': 'String'
        },
        'endTime': {
          'stringValue': '20191006T000000',
          'stringListValues': [],
          'binaryListValues': [],
          'dataType': 'String'
        },

[vfilimonov]

Furthermore: the capitalization is different across the whole structure:

Structure which is received by lambda function is not capitalized:

  "Records": [
    {
      "messageId": "b9ebc457-...",
      "receiptHandle": "AQEBs...==",
      "md5OfBody": "605e79544a68819ce664c088aba92658",
      "body": "Test 2",
...

But structure which is received from boto3 receive_message is capitalized:

  "Messages": [
    {
      "MessageId": "2dfe7646-...",
      "ReceiptHandle": "AQE...==",
      "MD5OfBody": "f4cd65fff38a3111863a6030fefdfc77",
      "Body": "Test 13f2c",
...

I.e. if using both ways of processing message (e.g. receive_message for testing purposes) one would need to write 2 different functions to handle the situation.

Further (I don't know if it is an expected behavior), as it is seen above: the key for messages is different between boto3 (Messages) and lambda (Records), though the rest of the keys are the same (up to capitalization issue)

[swetashre]

Thank you for your post. When i ran the receive_message api call in lambda i am getting this result 'DataType': 'String'. Even when i ran code in local environment i am getting capitalized data type.

Can someone please provide me the full debug logs for receive_message call where you are getting uncapitalized data type ? You can enable log by adding boto3.set_stream_logger('') to your code.

[vfilimonov]

Hello @swetashre

The example is actually trivial. Put it in a newly created lambda function, add SQS permissions and connect to some SQS feed.

import json

def lambda_handler(event, context):
    print(json.dumps(event, indent=2).replace('\n', '\t'))
    return dict(event)

Upon adding the message in the SQS - check the CloudWatch logs. You'll see the event as it is received by lambda - non-capitalized:

{
    "Records": [
        {
            "messageId": "...",
            "receiptHandle": "...",
            "body": "hello there",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1578907121082",
                "SequenceNumber": "...",
                "MessageGroupId": "lambdas",
                "SenderId": "...",
                "MessageDeduplicationId": "...",
                "ApproximateFirstReceiveTimestamp": "1578907121082"
            },
            "messageAttributes": {
                "Name": {
                    "stringValue": "SQS-test",
                    "stringListValues": [],
                    "binaryListValues": [],
                    "dataType": "String"
                }
            },
            "md5OfMessageAttributes": "",
            "md5OfBody": "...",
            "eventSource": "aws:sqs",
            "eventSourceARN": "...",
            "awsRegion": ""
        }
    ]
}

Note:

1. Records -> messageAttributes -> Name -> stringValue (and as you've seen, Boto3 function "receive_message" would return StringValue)
2. Records -> messageId (and Boto3 would return MessageId)
3. The structure is called Records (and Boto3 would return Messages)

All of these inconsitencies make it quite problematic to develop code which works with both boto3-responses and the direct responses from AWS.

[swetashre]

@vfilimonov - Whatever response you are getting with boto3 is directly coming from the service. When i ran this code in a lambda function i am getting this in the response:

def lambda_handler(event, context):
    client = boto3.client('sqs')
    response = client.receive_message(QueueUrl='https://sqs.us-west-2.amazonaws.com/123456789/my_queue',MessageAttributeNames=['All'])
    print(response)

'MessageAttributes': {'test': {'StringValue': 'testing', 'DataType': 'String'}}

This is the response of cloudwatch logs. Here you can see that it is returning StringValue(capitalized). After checking the debug logs i can confirm that this is the exact response coming from service.

I am not able to reproduce the issue . Can you please provide me the api you are using or sample code so that i can reproduce the issue ?

[Tylermarques]

@swetashre - If you set a Lambda function to be triggered by an SQS queue, and use the below code, you can reproduce.

```
def lambda_handler(event, context):
message = event['Records'][0]
body = json.loads(message.get('body'))
attributes = message.get('messageAttributes')
print(attributes)
````

'messageAttributes': {'test': {'stringValue': 'testing', 'dataType': 'String'}}

Lambda triggered by SQS is given the lowercase variable names, where as when you use boto3.client.receive_message you get uppercase.

[swetashre]

@Tylermarques - Thank you for providing me the steps to reproduce the issue. Now i am able to reproduce the issue. This is because lambda process the event in different way than the service SQS. I have internally escalated this issue to the service team. I will post here when i get any update.

[swetashre]

The message attributes are transformed to camel casing because of lambda's transformation logic in the invoke payload creation part. Fixing this could break customers who adapted to this change in casing when they started to use Lambda to process their queues. The lambda team is tracking this issue and they will discuss to find a fix that wouldn't affect the existing customers.

I am closing this issue as we can't do anything on our side to fix this issue. I will post here when i get any update from service team. Please reopen if you have any more concerns.

[Tylermarques]

What about a change to how boto returns SQS queue to match the current
Lambda? Would be a breaking change, but could be implemented in the next
version of Boto3.

On Mon, Feb 3, 2020 at 4:06 PM swetashre [e-mail] wrote:

Closed #2249 https://github.com/boto/boto3/issues/2249.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/boto/boto3/issues/2249?email_source=notifications&email_token=AEXWJRTQ3J5XK26XUGPVA5LRBCBLTA5CNFSM4KCRQTG2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWMFFIRI#event-3003798597,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEXWJRXCE7FXL2APWZGLSMTRBCBLTANCNFSM4KCRQTGQ
.

[swetashre]

@Tylermarques - They way boto3 returns the message attribute is same as the SQS service returns. Boto3 is only returning attribute in exact way it is getting from service. So i think here lambda has to be defined attribute same way as the service is returning.