Boto3: SQS Queue.receive_messages returns one message when set MaxNumberOfMessages to 10.

[tuxknight]

Hi Guys,
I can see two messages in my SQS queue in AWS web console.
But method receive_messages in sqs.Queue only return one message each time.
I am a little confused about this.

Here is my python code below:

sqs = boto3.resource('sqs', region_name="cn-north-1", aws_access_key_id=access_id, aws_secret_access_key=access_secret)
queue = sqs.Queue(sqs_url)
messages = queue.receive_messages(MaxNumberOfMessages=10,WaitTimeSeconds=10)
for message in messages:
    print message

[JordonPhillips]

Thanks for reporting. I confirmed that it only receives one message at a time. We'll take a look.

[JordonPhillips]

I've been investigating this issue, and it seems to be SQS's specified behavior. You are not guaranteed to receive 10 messages if there are 10 or more messages (especially with a small number of messages). All that long polling (enabled by setting WaitTimeSeconds) guarantees is that all the machines will be searched, and that it won't return empty until the time is up. When you have a large number of messages in the queue you are much more likely to receive 10 messages in a call.

Let me know if you have any more questions.

[tuxknight]

@JordonPhillips Yes, a distributed message queue cannot guarantee that 10 messages will be received each time even there are more than 10 messages. But no matter how many messages(I tested less than 10 messages, sorry), I always received one message each time.

I will test more and will be report if something wrong. Thanks.

----------- UPDATE ----------
When more than 10 messages in SQS queue. It will receive a bunch of messages one time using receive_messages, but less than ten.

$ python query_sqs.py 
Message: The IT Crowd.MP4 - a3128e53-6969-4e3c-b6aa-4e054031fdc4 - ObjectCreated:Put
Message: developing_modules.rst - 0e635aa9-acb0-48ad-b692-792afb9d1759 - ObjectCreated:Put
Message: developing_api.rst - 02481451-37d8-485b-9b26-14251757664c - ObjectCreated:Put
Message: intro_configuration.rst - c80d7f57-da6d-4d7f-98de-8df8b642f2cc - ObjectCreated:Put

$ python query_sqs.py
Message: intro_adhoc.rst - a613d28f-eb49-4c95-a98c-465ba698b1fc - ObjectCreated:Put
Message: intro_windows.rst - e67ceeee-b2c3-4669-b06c-753a19f800ff - ObjectCreated:Put

[JordonPhillips]

That is still a small number of messages. You can be more or less guaranteed to have 10 messages every time if you have several thousand messages in the queue. Here's a little script to put 5000 messages in the queue:

import boto3


sqs_url = QUEUE_URL
sqs = boto3.resource('sqs')
queue = sqs.Queue(sqs_url)


for i in range(500):
    messages = []
    for j in range(10):
        n = str(i * 10 + j)
        messages.append({
            "Id": n,
            "MessageBody": "Message %s" % n
        })
    queue.send_messages(Entries=messages)

After running that, I got 10 messages just about every time. You still won't get 10 if most of the messages are 'in flight'.

[arpithapradeep]

Hi Jordon,

Is this still expected behaviour, to only be guaranteed to receive 10 messages at a time when there are a several thousand messages in the queue? I have a queue with around 50 messages, would you say there is any other way of reading all the messages? Any other method or service I can use for small queues as such?

Thanks in advance!

[JordonPhillips]

You are never really guaranteed to get back 10 messages, but you can more or less count on it when you have a thousand or more messages that aren't in flight. If you want to read all of the messages on a small queue, you'll have to perform several requests.

[mllanes]

I'm seeing the same original issue; setting MaxNumberOfMessages 10 seems to come back always with 1 message

[cooncesean]

I take it, this is still an "issue" (not w/ the lib, more w/ SQS).

Context

I have a tiny queue (we'll only ever have ~8 messages in the queue at a time) that I'd like to expose over a Slack "slash" command. For example, typing /queue, would hit an API Gateway that would call a Lambda that would pull and display all messages in the queue (in order).

Temp Solution

I have some code that hacks around SQS's limitations (based on the feedback in this thread):

def lambda_handler(event, context):
    """
    Get messages from an SQS message queue and return them to the caller.
    """
    sqs = boto3.resource('sqs')
    queue = sqs.get_queue_by_name(QueueName='test-messages')

    messages = set()
    for i in range(0, 6):
        msg_list = queue.receive_messages(VisibilityTimeout=1, MaxNumberOfMessages=10, WaitTimeSeconds=5)
        for msg in msg_list:

            messages.add(msg.body)

    return list(messages)

Question

While this obviously isn't an optimal approach, it does work. That said, I was wondering if was possible to order these messages by the timestamp w/ which they were submitted to the queue?

Potential Approach

Set the timestamp as an attr on the message during message creation and order on that attr once I get my set....

[giladneural]

Don't count on the order of receiving the messages - put a timestamp in the message.

[Update]
I take it back - SQS does support FIFO - more here: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html

There are only two hard problems in distributed systems:

2 . Exactly-once delivery
1 . Guaranteed order of messages
2 . Exactly-once delivery...

[vaibhavrai46]

Adding a while True loop in your code will solve it
Ex -
while True:
for message in queue.receive_messages():
print(message.body)

This prints all the messages in the queue.

[Annant05]

I am still having the same issue/bug. Please resolve it or provide a workaround that does not include adding new messages and works with a small no of messages.

[joshuaherr]

Yeah I'm still having this issue now. I have a queue with 2 messages in it. When I call receive_message() through boto3 I only get 1 message back. When I use the Poll for Messages button through the AWS console I get back two messages (which is correct). This is incredibly discouraging and doesn't build much trust in boto3.