Rabbitmq-server: A lazy queue process should not load messages into memory on RabbitMQ node startup

[Gsantomaggio]

Thread group

Step 1

require 'bunny'
b = Bunny.new.start
ch = b.create_channel
1500.times do |i|
  ch.queue("q#{i}-dl", durable: true, arguments: { "x-queue-mode" => "lazy" })

  q = ch.queue("q#{i}",
               durable: true,
               arguments: {

                 "x-message-ttl" => 60_000,
                 "x-dead-letter-exchange" => "",
                 "x-dead-letter-routing-key" => "q#{i}-dl",
                 "x-queue-mode" => "lazy"
               })
  puts q.name
  10_000.times do
    q.publish "aaaa"
  end
end

Memory is around 1.5 GB, and the messages are stored to the disk:

Step 2

Stop the server

Step 3

Start the server

Memory grows up ( ~ 20 GB ), messages are stored to the disk and also in memory:

Conclusion

When the broker starts it keeps all the messages in memory. (even if the queues are lazy queues).

[michaelklishin]

We have a decent understanding of what's going on. Lazy queues only become such after the policy application step and that happens after queues have recovered [most of] their durable data.

Broadly speaking, there are two different solutions:

• Tweak variable_queue (defaults or implementation or initial state) to load less data into RAM during initial state recovery
• Apply some or all policies before recovery

[dcorbacho]

Queues will load a segment on memory from the index on startup, and when delivering messages. With the current design this cannot be avoided, as reading single elements from the index causes an unacceptable performance degradation (throughput three orders of magnitude lower). If the memory usage is too high, users could consider if the messages they are embedding on the index are too big (see queue_index_embed_msgs_below configuration).

[carlhoerberg]

Do we have any ideas on how to go forward here? Btw, this happens both when the messages are in the index and when they aren't. We routinely have very long queues on some servers, which works great as long as the server is up and running. But if the server has to be restarted it can't be done without adding tons of RAM (that's not then needed once up and running, but costs a small fortune).

[lukebakken]

@carlhoerberg - the team spent a lot of time investigating solutions but couldn't find anything satisfactory. I will bring your concern to the team's attention. Today and tomorrow are US holidays. Thanks!

[michaelklishin]

The conclusion is roughly this. Messages embedded into queue index will be loaded all at once since indices are not "lazy loaded". On most workloads this doesn't matter but with a huge backlog of tiny messages embedded, it suddenly does. Disabling embedding avoids the problem.

We explored 2 or 3 different ways of avoiding this and are not happy with any of them. Embedding messages into queue index has other operational ramifications we now understand well and don't like, so chances are that feature will be going away in a future version.

[gerhard]

@carlhoerberg @Gsantomaggio can you please confirm if setting queue_index_embed_msgs_below to 0 solves your problem?

[gerhard]

We've confirmed that setting queue_index_embed_msgs_below to 0 does not fix the problem. Having talked to @Gsantomaggio this is what we've concluded:

It is unexpected and undesired for lazy queues to load messages in memory during RabbitMQ node startup. The primary reason for using lazy queues is to keep messages on disk as long as possible, which makes the current lazy queue behaviour on node startup unexpected. Furthermore, this results in RabbitMQ crashing if there are many lazy queues on a RabbitMQ node and the messages are below 4096 bytes in size even if queue_index_embed_msgs_below is set to 0. Currently, on node startup, lazy queues will load in memory 16384 messages. This number comes from SEGMENT_ENTRY_COUNT: https://github.com/rabbitmq/rabbitmq-server/blob/5d45fc999335451eeee3f9dd4c6f87b9fbfe4f3f/src/rabbit_queue_index.erl#L140 . In the current implementation, if the first message is 4096 bytes or lower it will be loaded into memory, as well as all other following messages, up to 16384, if they are 4096 bytes of lower. If the first message is 4096 bytes and the second messages is above 4096 bytes, only the first message will be loaded into memory.

Even if queue_index_embed_msgs_below is set to 0, therefore Queue Index embedding is disabled, messages will still be loaded into memory as described above.

The expected and desired lazy queues behaviour would be to not have messages loaded in memory on node startup.

Did we understand the problem correctly?

[michaelklishin]

@gerhard it was never the promise of lazy queues to not keep a reasonable number of messages (e.g. one index segment worth) in RAM. The promise is that messages will be moved to disk aggressively and regardless of whether they were published as persistent or transient.

I don't have the exact steps to try but I'm quite sure the root of the issue is still queue index embedding since otherwise QI entries are really small. Queue laziness is a red herring and the real issue is that each queue loads 16K messages on start, and they contain small yet non-zero payloads.

[michaelklishin]

Before anyone suggests that SEGMENT_ENTRY_COUNT should be lower or configurable: it you reduce it, it will break existing installations badly.

[carlhoerberg]

In our isolated tests we can start instances with more messages than RAM (even with a huge number of queues) with message index embedding disabled (but not with the default value).

We do however have real-world cases where it still hasn't been possible to start the server before it runs out of RAM, even with queue index embedding disabled, so more research has be to performed there on our side.

We've been deploying servers with queue index embedding disabled and lazy queue-mode enabled by default on all servers for the past year or so. In our world it's much preferred to have stability and reliability over some extra msgs/s throughput.

[gerhard]

@michaelklishin are we saying that it's a matter of documenting that SEGMENT_ENTRY_COUNT worth of messages will be loaded into memory on node startup, regardless of message size? This is what @Gsantomaggio observed & reported.

If this is the expected behaviour and nothing will be done about it until the queue index is replaced by something else, we should at least document it on https://www.rabbitmq.com/lazy-queues.html

Given our current team focus and the upcoming themes, I believe that a proper fix will need to wait until we start working on the message store level replication (a.k.a. _The Replicator_), which is at least a few months away. What do you think @michaelklishin ?

cc @dumbbell @kjnilsson

[gerhard]

@carlhoerberg There are some useful links in https://github.com/rabbitmq/workloads/tree/master/lqs-ha-confirm-multiack.

Every 24 hours, VMs get preempted, so if this is an issue, RabbitMQ nodes should fail to start. As you will notice, producers are consistently outpacing consumers, so once the queue limits will be hit, if everything works as it should, then the cluster should remain in equilibrium, with VMs coming and going.

[gerhard]

I can see that messages will be loaded into memory on node startup even if queue_index_embed_msgs_below is set to 0:

Notice how much memory message store index is using on the restarted node even though queue_index_embed_msgs_below is set to 0.

This is the state of the cluster after rmq0 gets restarted:

The memory alarm is getting triggered intermittently. No new messages are being published. While all channels are idle, all publisher connections are blocking. Most queues are idle, few queues are running.

The RabbitMQ cluster that I'm basing all the above observations on is public.

As a next step, we need to reach consensus within the team that the above observations are correct. Will let you know when we know more.

[gerhard]

All observations point to the fact that mirrored lazy queues are a bad idea:

Queues are sync-ing for a really long time, all queues keep messages on disk, nothing is loaded in memory:

rmq0 has all the messages & it's sync-ing very slowly with rmq1 & rmq2:

rmq1 has triggered the memory alarm, most memory is used by message store index & binaries:

In conclusion, mirrored lazy queues are pretty bad in practice. Will set up a single-node environment with many lazy queues to make public the actual behaviour.

[dcorbacho]

The message store index still requires 664-696 Bytes per message when queue_index_embed_msgs_below is set to 0, which is approximately 14Gb of RAM for those 21M messages on the previous example.

Thus, number of messages x 696 Bytes is the potential min memory requirement for the system to store that number of messages even using lazy queues and queue_index_embed_msgs_below = 0. Note that not all segments might be in memory at the same time, thus the footprint might be smaller.

| Mem used by msg index (Bytes) | Message payload (Bytes) |
| - | - |
| 664 | 0 |
| 672 | 1-8 |
| 680 | 9-16 |
| 688 | 17-24 |
| 696 | 25 and above |

[dcorbacho]

Note also that the lazy queue setting is known to be not particularly efficient whenmax-length is used, and both are configured in @gerhard deployment. Thus, we must be careful with conclusions drawn from those results.

[gerhard]

These are some excellent points @dcorbacho, I'm updating the lazy queues docs to capture these learnings.

Just to be clear, when we say that lazy queues are not efficient with max-length, do we mean:

1. use these 2 settings with care, performance of operations A, B, etc. will degrade, or
2. use these 2 settings with care, they may introduce cluster instability under certain conditions, e.g. A, B, etc. or
3. do not use these 2 settings together, they are not supported & you may experience unexpected problems

[gerhard]

I continued looking into this and can confirm that simple lazy queues do not load messages into memory on node startup if queue_index_embed_msgs_below is set to 0. This is on RabbitMQ 3.7.2 & Erlang 20.2.2. Things that don't matter:

1. lazy mode definition via policy / queue arg
2. Message body size: I tried with 100 byte & 1000 byte messages

Can you please confirm my observations?
RabbitMQ lazy queue memory & disk usage.xlsx

[gerhard]

We've shared our learnings in https://www.rabbitmq.com/lazy-queues.html?t=1#behaviour, please re-open this issue if you find the behaviour of Lazy Queues on node startup inconsistent with our latest learnings. This is in particular relevant:

To clarify, changing the queue_index_embed_msgs_below property after messages have already been embedded in the queue index will not prevent lazy queues from loading messages into memory on node startup.