Redisearch: Keys in RediSearch not binary safe

[tw-bert]

Redis strings are binary safe. We use null characters in keys quite a lot (mainly when compound keys are involved, where we use the native lexicographical indexing of sorted set members).

RediSearch seems to handle this mostly okay, but fails at FT.SEARCH, trying to find the HSET with only the part of the name before the first null character.

Simple redis-cli example:

127.0.0.1:14223> FT.CREATE "ftFoo" STOPWORDS 0 SCHEMA MyText TEXT NOSTEM
OK
127.0.0.1:14223> FT.ADD "ftFoo" "compound\x00key\x0012345" 1.0 "FIELDS" "MyText" "the crazy fox jumps over the lazy dog"
OK
127.0.0.1:14223> FT.SEARCH "ftFoo" "crazy"
1) (integer) 1
2) "compound"
3) (empty list or set)
127.0.0.1:14223> keys "*compound*"
1) "compound\x00key\x0012345"
127.0.0.1:14223> HGETALL "compound\x00key\x0012345"
1) "MyText"
2) "the crazy fox jumps over the lazy dog"

[dvirsky]

I actually never committed to supporting binary keys, but it's not a big deal.

But why would you model compound keys on sorted sets if you have redisearch?

[tw-bert]

This is needed due to space requirements. We have sorted sets with several millions of members, and in each member, the values (only values, no field names) are stored in msgpack format. The accompanying meta data is stored in a separate key (only once). We will not use RediSearch for each and every field of each and every row of each and every table. There is no need. Still, RediSearch fulfills a big need for us in cases where the sorted set approach is too limiting.

[tw-bert]

On second thought, I shouldn't have mentioned sorted sets. Although it enables us to store a lot of data very efficiently, it is not relevant to the issue.

The following can, I hope, illustrate our need (or wish) for binary safe keys better.

Our architecture is designed for distributed persistence (read ánd write). In a lot of our tables, we have a compound key that identifies the row. We have constraints in all N-tier layers for that compound key.
So, if a compound key is for example { [string] "A", [double] 42, [bool] true } we need a compact representation for that. We also use numeric row identifiers, but these simple sequence-based identifiers are only used within one persistence domain (for example, on a huge backend RDBMS, or one Redis master on one far away layer, or some Mongo on a disconnected server). Similar to local-only integer identifiers and world-wide hash identifiers in git and mercurial. Distributed persistence is sequence-less by definition.

A hash of the compound key could be used as a RediSearch document ID. But, a hash is not reversible, could collide, and to us this simply doesn't follow KISS principles. Why invent a different ID if you already have one?

For simple "concatenation with delimiters", if a compound key contains two consecutive UTF-8 strings, there is no other character than the null character to use as a separator. Not without escaping the data at least, and that would add complexity (a small amount, but still... escaping, double escaping, triple escaping, it happens all around the world and always causes issues).

For redis and other persistence technologies, msgpack serialization has served us well. We did investigate other compact serialization formats like BSON, but msgpack has proven itself nicest and simplest to us. Therefore, next to simple delimited concatenation, I would consider using the msgpack representation of the compound key for the RediSearch document unique name. And... msgpack is binary, so no guarantee that it doesn't contain null characters.

I hope this has shed some light on our design.

I don't think we are using Redis in a way that's very special. In the end, it's just a bunch of bytes being stored in an efficient and organized way, with the excellent replication possibilities that Redis provides.

[dvirsky]

I completely understand. I've implemented the same thing in the past. Again as I said it's not a big deal as long as it's just the document keys.

[tw-bert]

Absolutely. Key and Payload are the only entities this applies to.

[dvirsky]

@mnunberg mind taking this one? Basically we need to make sure that document keys are always referenced in a binary safe way, most importantly in doc_table.

[mnunberg]

This should be resolved.