Rabbitmq-server: Redelivery count for messages

[hairyhum]

From the 0-9-1 spec:

The server SHOULD track the number of times a message has been delivered to clients and
when a message is redelivered a certain number of times - e.g. 5 times -
without being acknowledged, the server SHOULD consider the message
to be unprocessable (possibly causing client applications to abort), and [dead letter] the message.

This should be made optional for compatibility.

More context is provided in #1020.
Related: https://github.com/rabbitmq/rabbitmq-stomp/issues/36.

Update:

• [x] Quorum queues in 3.8.0 track redelivery counts and expose the counter to clients via a header
• [x] #1889 introduces a redelivery limit (poison message handling) on top of that
• [ ] Implementation for classic queues is under consideration

[michaelklishin]

This is not really a requirement, and message properties don't have a field for redelivery count, only a boolean flag. Still worth having, of course.

[hairyhum]

First investigation results:
Required changes in backing_queue API to support integer redelivery count instead of flag in fetch/2 return value and functions working with fetch.
Also queue index format should be changed to support multiple deliver records, or delivery count. But because current implementation assumes exactly 1 deliver for each acknowledge there should be significant changes in index structure.
Any thoughts?

[vgkiziakis]

Also very interested in this feature. It's invaluable in scenarios where there is a "poisonous" message that for some reason is causing consumers to crash or behave erratically.

Are there any plans for adding this in a future release ?

[hairyhum]

It's in the roadmap for 3.7.x

[mikljohansson]

@hairyhum do you know if the delivery-count feature is still slated for inclusion in rabbitmq-server 3.7?

The reason I'm asking is because we see the poisonous-message problem every once in a while. Latest incident was last week and caused a system degradation in our own components, before the problem could be found and circumvented.

We'd very much like a robust broker-supported way of detecting messages that consistently trigger a severe problem in the client

[michaelklishin]

@mikljohansson we will do our best to fit it into 3.7.0 but no promises.

[plechev]

This is a much needed feature to enhance the robustness of message (re)delivery mechanism. We are dealing with the "poisonous messages" problem which is currently handled by the client, but it will be more elegant if we can use the broker for this.
Thumbs up!

[michaelklishin]

An update: this won't be in 3.7.0 because we had quite a few out-of-cycle and new features in 3.7.0 and we need to limit the scope at some point. But we are contemplating it for 3.8.0 (so that it doesn't have to wait for 4.0, that is).

[mattheworiordan]

An update: this won't be in 3.7.0 because we had quite a few out-of-cycle and new features in 3.7.0 and we need to limit the scope at some point. But we are contemplating it for 3.8.0

@michaelklishin out of interest, how do your minor release cycles work i.e. realistically if it made it into 3.8.0, when could we expect that release?

Separately, is there anything we could do to help if it was a priority for us?

[ghost]

I posted this issue here as a suggestion for the team to work on it. Your product is really great and I think it would become bigger if retry capabilities are present. Thank you!

[mattheworiordan]

Hi @michaelklishin, do you mind if I ask if any progress has been made on this? Running RabbitMQ on behalf of customers means RabbitMQ is quite susceptible to overload from one customer (unless we rate limit them which is not ideal) simply because of a faulty code loop.

[michaelklishin]

It is on the roadmap for 3.8.0.

3.7.0 introduces operator policies that can be used to force a "high
watermark" message TTL value.

On Thu, Feb 23, 2017 at 12:53 PM, mattheworiordan [e-mail]
wrote:

Hi @michaelklishin https://github.com/michaelklishin, do you mind if I
ask if any progress has been made on this? Running RabbitMQ on behalf of
customers means RabbitMQ is quite susceptible to overload from one customer
(unless we rate limit them which is not ideal) simply because of a faulty
code loop.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rabbitmq/rabbitmq-server/issues/502#issuecomment-281946707,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAEQh6FZat4Rqy05AA3vohry3BCJZOTks5rfVckgaJpZM4G2YHW
.

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[mattheworiordan]

It is on the roadmap for 3.8.0.

Great to hear, thanks Michael.

[rrmayer]

Do we know what the status is on this issue? We continue to see questions about this on Stack Overflow, so it appears that there is at least a reasonable demand for this feature.

[michaelklishin]

The status is that it has a chance of getting into 3.8.0 but we make no promises.

Update: #1889 has been merged and covers quorum queues.

[mattheworiordan]

The status is that it has a chance of getting into 3.8.0 but we make no promises.

🙏

[jlcd]

Also interested. This feature would be a life saver, as we won't have to create workarounds to stop requeueing messages that aren't processable.
Any updates whether it will be included into 3.8.0?

[Misiu]

I'm really new to RabbitMQTT but I also think this would be a lifesaver.
Normally when querying external API and getting an exception I'd like to try to make a request for same data couple of times. I'm aware I have Redelivered flag, but I would like to know how many times the same message was redelivered. This way I could reject it when I hit 5 tries. This could be even done on the server.

I have one additional question. Would it be possible to delay redelivery?
I want the same message to be redelivered after a specified time, for example, I want second try after 10 seconds, third after a minute.
Can this be done without publishing a new message?
In C# I can use BasicReject method, but this allows me to flag message to redelivery, but not to delay it.
Can https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/ be used somehow to accomplish this?

[Soarc]

@Misiu to achieve this, i ended up with using MassTransit over RabbitMQ. Something like this

            _bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
            {
                var host = sbc.Host(new Uri($@"rabbitmq://{_host}/"), h =>
                {
                    h.Username(_userName);
                    h.Password(_password);
                });
                sbc.ExchangeType = "direct";
                sbc.AutoDelete = false;
                sbc.Durable = true;
                sbc.Exclusive = false;

                sbc.ConfigurePublish(PublishCallBack);
                sbc.ReceiveEndpoint(host, _queueName, ep =>
                {
                    foreach (var suber in _handlerSubscribators)
                        suber(ep);                    
                });
                sbc.UseRetry(retryConfig => retryConfig.Immediate(_retryCount)); // <----- This line achives what you want
            });

[lukebakken]

@Misiu @Soarc - please use the mailing list for discussions.

[Avivsalem]

@mattheworiordan,
Did it make the cut for 3.8.0?

[mattheworiordan]

@avivsalem, I think you were meaning to ask @michaelklishin?

[Avivsalem]

Yup... My mistake. So @michaelklishin what do you say? Did it make it to 3.8.0?
Because in my production environment, if a message causes a consumer crash, it will be poison forever...

[michaelklishin]

A comment above asks all discussions and questions to be moved to the mailing list.

Quorum queues in master keep track of a redelivery count. We make no promises if classic queues will in 3.8. Currently, it's just a number consumers can use to make decisions, nothing else.

It's your responsibility to handle exceptions in your consumers. This is not going to change with any redelivery count implementation.

[michaelklishin]

1889 is in.

[michaelklishin]

Leaving this open until classic queues get a similar feature (or we really decide against that).

[michaelklishin]

It seems very unlikely that classic queues are going to get this feature in 3.8 (quorum queues do, though). Will close this a file a new issue specifically for classic queues.