Keda: RabbitMQ scale down is not reliable

[joskfg]

Description
Currently KEDA checks the amount of messages that are in ready statement as we can see here

https://github.com/kedacore/keda/blob/ce639ecc577d922c6b55f02494e0d973fc2722cb/pkg/scalers/rabbitmq_scaler.go#L132

Problem
When you have only one consumer that is consuming a single message and there are no more ready messages in the queue. In this situation KEDA is going to scale the deployment to the minimum.

The problem is that if you have 0 replicas as minimum and you don't have enougth time to process the message (taking into account the cooldownPeriod defined), the consumer will be terminated by kubernetes after the graceperiod. This produces the message to be requeued, so KEDA will detect it and scale it again to 1. So you can stay in that loop forever.

Possible solution
To prevent this I think KEDA should check the unacknowledge messages when it wants to scale to 0.

Workaround
To fix this an avoid the loop, you can increase the amount of cooldownPeriod to a higher time than the process takes to process the messages.

cooldownPeriod = (time to process a message * prefetch amount)

This is far from a good solution, but it will avoid the infinite loop.

[ahmelsayed]

Thanks for the detailed issue and workaround @joskfg

I think your proposed solution makes sense to me. I need to read about message acknowledgement in rabbitmq and see what we can do.

[sc-chad]

If return items.Messages, nil is returning Ready messages then I think this could be avoided by using Total instead. As that message count will stay at 1 until the message has been Acked.

[joskfg]

I totally agee. The best solution is to use the total messages (ready+ack) .

[sc-chad]

@ahmelsayed Is this something that might make it into 1.1? It's still an issue in 1.0. It's causing workers to scale down when there are messages in the ack queue.