Redisearch: HIgh CPU Usage

[cit68]

Redis version 6.0.4,
Redisearch version 1.6,
Server 128GB RAM, 8 Core, Debian.

We have loaded 31G of data into Redis and we are experiencing 100% CPU usage on Redis instance even if we're not querying it.
We removed all data from Redis and the CPU usage went down. It seems that the load on Redis causes high CPU usage.
Is it normal behaviour?

[gkorland]

Are you updating docs while you see high CPU?
The immediate suspect here is the index cleanup GC, can you please check if you see a forked process causing the high CPU?
Notice in 1.8 the default GC triggers configurations were changed to cause less overhead in low rate updates (https://github.com/RediSearch/RediSearch/commit/9943207008f8a14a710d905e64e0f4d99d636380) .

You can changed the default on 1.6 too by increasing the fork_gc_clean_threshold and the fork_gc_run_intervall attributes.

[kkmuffme]

This issue is definintely the same one I encountered recently (see https://github.com/RediSearch/RediSearch/issues/978).
This behavior has been fixed (improved at least) in the 1.99.1 release.

You can however just fix it yourself in whatever version, by running redisearch with https://oss.redislabs.com/redisearch/Configuring.html
FORK_GC_RUN_INTERVAL 900 FORK_GC_RETRY_INTERVAL 60 FORK_GC_CLEAN_THRESHOLD 10000

[cit68]

@gkorland @kkmuffme

Are you updating docs while you see high CPU?

No, we are not.

check if you see a forked process causing the high CPU

No, we don't have any forked process.

It seems that including these parameters
FORK_GC_RUN_INTERVAL 900 FORK_GC_RETRY_INTERVAL 60 FORK_GC_CLEAN_THRESHOLD 10000
we solved the problem of High CPU Usage during periods of quiet where we don't perform any kind of operation on Redis (this was the main issue).

Now the high CPU problem appears in the following scenario:

I have a script that takes care of refreshing the cache. This script essentially inserts a large amount of data, and in particular:

• first removes records with "FT.DEL {indexname} {entryId} DD"
• and then adds new data with: "FT.ADD {indexName} {entryId} 1 REPLACE FIELDS {... fields} ".

At the end of the execution of this script the CPU remains normal (after the insertion of GC parameters) but if during the execution of the script we perform any kind of ft.search via nodejs (node-redis) the CPU rises up to 100% and it never goes down anymore.

Do you have any idea? Thanks

[kkmuffme]

Are you using RDB or AOF or both?
Can you try disabling RDB (in redis.conf save "" and comment all other save instructions in the file) and see if the issue persists?
In "top" command, does it show only this 1 redis instance or multiple "redis" things when this issue occurs?

[Gappa88]

@kkmuffme @gkorland

We are using only RDB. We already tried to disable it adding save "" either in the config file and via redis-cli but the problem persists.

using top command we can see only the redis instance ./redis-server ip:port, also right now while i'm writing this, CPU usage on redis (with redisSearch module) instance is 100% and there is no query or other actions active on that.

[kkmuffme]

Can you perhaps strace it and see what you get?

[Gappa88]

@kkmuffme
no, this is not possible

[kkmuffme]

Good luck then finding out what the issue is