Redisearch: How to maintain determinism with RediSearch 2.0?

[tw-bert]

_Preample:_
We've been using RediSearch extensively since 2017.
A lot of our RediSearch use cases involve transactional indexed data updates. We create/update/delete RediSearch index entries during updates on other keys/hashes/sorted sets inside Lua. This ensures that secondary processes, who utilize RediSearch for their recalculations (and often write results back into Redis) are guaranteed to have a consistent result.

_RediSearch 2.0_
Reading about the architectual changes in RediSearch 2.0 got us a bit worried. We read here that it follows a "strongly eventual consistent" policy. The word "eventual" conflicts with determinism...

How do we maintain determinism if we need to switch to RediSearch 2.0?

1. Immediately after starting a Redis server, the status will be "busy loading" for a few minutes (our instances are about 15GB), and after that, client processes can do their thing. Naturally, they will use FT commands, and expect the results to be correct. We've read that in RediSearch 2.0, the indexed data will not be part of the RDB, nor the RDB preample in the AOF. Is this correct? If so, how to tackle this?
2. If a Lua script or pipeline is executed, any parallelism during the redis command will work fine. But when the command has finished, and a next client can issue a command, the parallelism should have joined and ended. Is this the case?

Please enlighten us, TIA

[MeirShpilraien]

Hey @tw-bert,

The "strongly eventual consistent" is only related to CRDT support where strong eventual consistency is promised between sites. The index in each shard (and surly on oss Redis instance) index the data synchronously, i.e, you will not get the hset (or FT.ADD, which was deprecated) reply until the data was indexed. This means that once you get the reply the data was indexed and you will see it on queries (as it was on v1).

Regarding RDB, this is correct, the index is no longer saved on the RDB. Only the index definition is saved on the RDB on a special location in the RDB call AUX field. This allows us to reindex the data base on the index definition. When the RDB load finished we start to rescan the data and index it (this is when you see the "busy loading" status). But, if you used Redis 6.0.7 and above, it supports a special loaded event that allows us to index the data immediately (and synchronously) when Redis loads it. So if you use Redis >= 6.0.7, when the RDB loading finished you will have the index ready with all the data to query.

Hope this answers your questions, let me know if anything else is unclear.

[tw-bert]

Hi Meir,

Thanks a lot for your clear reply.
That all sounds fantastic, great design guys.

Switching to Redis 6.0.7 is no issue for us, and was on the roadmap anyway. We're on 6.0.6 atm. The new features in Redis' module API sound great as well, we'll certainly make use of that.

Closing this issue.

[MeirShpilraien]

@tw-bert please notice a small detail. Currently, there is no way for us to override the hset response and return an error in case of schema mismatch (which can only happened if you set TEXT data in a NUMERIC field). Such mismatch will cause silent index failure and you will not see the document in the results. The only way to know about such failure is on FT.INFO under the hash_indexing_failures field. There should be no issue if your data matches your schema but I still thought it is important to mention.
I hope that in future Redis versions we will be able to override the results and return an error in case of indexing failures.

[tw-bert]

@MeirShpilraien : Thank you for the heads-up. We'll add hash_indexing_failures to our metrics collector (InfluxDB/Grafana) to be automatically notified. I think we won't encounter this (due to the strict manner we populate RediSearch), but best be sure.