Rabbitmq-server: Message grouping

[petercuichen]

I have been using RabbitMQ for a really long time, but recently I met a problem that certain messages (with the same id or something) should be always consumed by the same consumer to be sure of processing in order, even if there are many consumers on a queue.

And I see an email in 2012 question-on-message-ordering-guarantees, it talked about the feature Message Grouping which ActiveMQ has, so I just wanna see is there some plan for you to implement this feature in near future?

Thanks a lot for your work.

[michaelklishin]

No immediate plans but we are not opposed to implementing it some day.

[michaelklishin]

@pycuichen I'm not sure how your original issue requires message grouping. Consumers can have priorities these days, you can use channel QoS to alter how consumers are picked.

[AVGP]

I have a different use case, but it would be great to have the feature, because I think message groups are what I'd need for it at the moment.

So, I'm having a task that usually sits on the queue for a quite some time (i.e. minutes or even hours) and while it's on the queue other users may request the exact same task.

Instead of adding a duplicate I'd like the duplicates to be handled as if one single job was created.

Currently I solve this with an additional storage that keeps track of things that have already been submitted... but I'd rather not have to do that.

[petercuichen]

@michaelklishin My original issue is like this, we producer many message like (id, status), which can be (134378578, 0) or (231394533, 1), to one exchange, and so on to one queue, but I have several consumers listening this one queue. Here comes the problem, I need to make sure the consumers consume the messages with the same 'id' in order.

For example, in queue, message A (134378578, 1) comes after message B (134378578, 0) , so I need message A to be processed also after message B, there is no problem with one consumer to one queue, but with several consumers, we can't be sure because of uncertain network delay, so what I am thinking is to make sure messages with the same 'id' be processed by the same consumer. So 'Message Grouping' may solve my problem.

I saw some solutions in exchange level, but in my case, I need something like sharding according to the message info in queue->consumer level.

I shall be in great grateful if you can give me some other solutions in RabbitMQ, if not Message Grouping.

Thanks again for your rapidly response.

[videlalvaro]

Have you considered using the the Consistent Hash Exchange and then hashing your messages IDs, so they are distributed to queues. In that way one consumer will manage its own queues, and process those messages in order.

[petercuichen]

@videlalvaro Good point, thanks for providing the idea. Yeah, I considered it before, but I have a few concern below:

• If I use one consumer for a queue, how could I do to make the message still be consumed when the one consumer down? I used multi-consumer for one queue before.
• May it lose some or even one message when binding a new queue to the exchange? (also unbinding). Cause in my application, the message is so important that we shouldn't even lose one.
• It may be a bit heavy for me cause when there is 20 consumers, I should create 20 queues to bind the exchange, and do it again every time I change the consumers (for the consumer in the application could be changed sometimes), and the OPS will kill me...

Thanks for your response.

[michaelklishin]

@pycuichen please post general questions to rabbitmq-users. I'm going to remove comments that have nothing to do with the issue.

[yairogen]

Any updates on this? I would also like to this feature.

[funwun]

Any updates on this?

[michaelklishin]

No update. It's not on the roadmap for 3.8.0 at the moment.

[ducdigital]

+1 for this. just looking around and it seem pretty important feature to have

[z0mb1ek]

+1

[kbrumer]

+1

[michaelklishin]

This feature is not on the roadmap for 3.7 and likely won't be on the roadmap for 3.8. We don't need any more +1s as they contribute very little to this "discussion" at this point. We will unlock this as there are specific details as to how message grouping might work and why.