Emqx: Subscribe takes too long and cpu spikes

[xlb0479]

Environment

• OS: Ubuntu16.04.4 LTS
• Erlang/OTP: 21
• EMQ: 3.0

Description

A description of the issue

Network: LAN
Subscribe takes 5~10 seconds, is this normal?
After subscribe send, CPU spikes immediately.

This is my dashboard UI

I use websocket to test

Subscribe causes CPU spikes

I retained 80k messages, but the retained message is not for the topic I subscribe, is this the reason?

Thanks.

[xlb0479]

The emqx_retainer.DAT file size only ~30MB.
But after I rm the data/mnesia directory and restart emqx, the subscribe takes only a few milliseconds.
I know little about emqx details, I guess every subscribe has to check all the retained message.
Considering the size of emqx_retainer.DAT and the message amount I retained(about 80k), I think it takes too long to do the check.

[HJianBo]

Hi, @xlb0479 The emqx_retainer will enumerate all of retained messages if a client subscribe a wildcard topic. It is the mainly reason to cause this problem.

I think we should improve the retained message implement for subscribing a wildcard topic in a numbers of data

[alorle]

Same problem here with more than 50k retained messages (~30kB each) only when subscribing to topics containing wildcards.
If a client subscribes to a topic without wildcard, connack is received "immediately".

[tradingtrace]

@xlb0479 @alorle

Hi, I mocked 100K retained messages under random topics.
I tried to subscribe testtopic/# (no retained messages for it), and it responded within 1 second.

Could you please provide some more details?

1. Some topic examples of the retained message.
2. How many nodes do you run?
3. Could some retained messages be expired when subscribing?

Thanks

[alorle]

1. Topics are something like channel/:room_type:/:room_id:/:measure: where :room_type:, :room_id: and :measure: can be combined with several values each. In my case, I want to subscribe, for example, to something like channel/living_room/1/+ to get all measures from a room, or, in other cases, something like channel/+/+/temperature to get the temperature from all rooms. The fact is that, when I have ~50k retained messages or more, the subscription becomes very heavy. I understand that this is because when subscribing with wildcards, the broker must search over the entire database of retained topics. Once the connack is received, everything goes as it should, the new messages arrive as expected. It is only problematic when you subscribe.
2. Only one node, since we are trying to evaluate the most suitable broker for our use case among those available in the _"market"_.
3. In our use caso no. At least not all, as some are events, so we use the functionality of retained to know what has been the last event occurred.

I have investigated a bit in the code, and although I do not know much about Erlang, I understand that the database of reatined messages is a key/value db. So that, the last message is stored for each topic. The topic is the key and the message the value.

Assuming this, it would not be a better solution (I repeat that I do not know Erlang and therefore I do not know its limits) to structure this database in the form of a tree in such a way that it would be more agile to search for topics by wildcards?