Redisearch: FT.ADD ... REPLACE without all fields leaves unindexed fields

[stockholmux]

If you create a schema with multiple fields then later use FT.ADD .. REPLACE but do not specify all the indexed fields, the document will still contain the field yet it won't be present in the index. Tested with 1.4.x and 1.6.x. Note age below:

> FT.CREATE myppl SCHEMA first TEXT last TEXT age NUMERIC
OK
> FT.ADD myppl instructor 1 FIELDS first Kyle last Davis age 39 country Canada
OK
> FT.SEARCH myppl "@age:[39 39]"
1) (integer) 1
2) "instructor"
3) 1) first
   2) "Kyle"
   3) last
   4) "Davis"
   5) age
   6) "39"
   7) country
   8) "Canada"
> FT.ADD myppl instructor 1 REPLACE FIELDS first Loris last Cro
OK
> FT.GET myppl instructor
1) "first"
2) "Loris"
3) "last"
4) "Cro"
5) "age"
6) "39"
7) "country"
8) "Canada"
> FT.SEARCH myppl "@age:[39 39]"
1) (integer) 0

It's unusual behaviour but it seems like it's doing basically a FT.DEL without DD, then doing an FT.ADD with the specified fields only (which is makes sense since the un-indexed field country is also persisting).

[MeirShpilraien]

FT.ADD replace only update the index and the hash but not delete it (or delete already existing fields), it's the same as if you do hset doc1 ..some fields.. and then ft.add doc1 ..different fields.., would you expect redisearch to delete the extra fields added by the hset? Notice that if you do the ft.search again you will not find it in the results because redisearch did not index the age field.

[ashtul]

@stockholmux @gkorland @MeirShpilraien @mnunberg @K-Jo
Should we add a flag for HARDREPLACE?

[stockholmux]

@MeirShpilraien What you are saying is that is In effect this is FT.DEL (without DD) followed by FT.ADD.

This _looks_ like a bug as you have a document with indexed fields that are not indexed. At very least it should be completely documented. Better might be a true document replacement option (remove the underlying hash, like FT.DEL...DD).

[timmixell]

I'm encountering what I believe to be the same issue. In my case, I need to remove an indexed field because it is no longer relevant to the document. Confirming that in this case, the current expectation would be to FT.DEL and do a subsequent FT.ADD

[MeirShpilraien]

@timmixell so you are saying that you would expect the field in the hash to be deleted?

[timmixell]

That is correct. We have an indexed field that has been emptied out or set to null. What is the best way to update the index with this information?

[timmixell]

Greetings, this one is still causing me some issues. For what it's worth, the command reference at https://oss.redislabs.com/redisearch/Commands.html#ftadd reads:

REPLACE: If set, we will do an UPSERT style insertion - and delete an older version of the document if it exists.

and further down the page:

REPLACE: On its own, sets the document to the new values, and reindexes it. Any fields not given will not be loaded from the current version of the document.

REPLACE PARTIAL: When both arguments are used, we can update just part of the document fields, and the rest will be loaded before reindexing. Not only that, but if only the score, payload and non-indexed fields (using NOINDEX) are updated, we will not actually reindex the document, just update its metadata internally, which is a lot faster and does not create index garbage.

Using the following example, it seems like it actually just always acts like REPLACE PARTIAL, or am I misinterpreting the docs/expected behavior of REPLACE and REPLACE PARTIAL

Running Redis 5.0.7, with RediSearch 1.6.6:

FT.CREATE gh1193 SCHEMA text1 TEXT text2 TEXT numeric1 NUMERIC

FT.ADD gh1193 doc1 1 FIELDS text1 "hello" text2 "world" numeric1 1
FT.ADD gh1193 doc1 1 REPLACE FIELDS text1 "update" numeric1 100
FT.GET gh1193 doc1

FT.ADD gh1193 doc2 1 FIELDS text1 "hello" text2 "world" numeric1 1
FT.ADD gh1193 doc2 1 REPLACE PARTIAL FIELDS text1 "update" numeric1 100
FT.GET gh1193 doc2

PARTIAL doesn't appear to have any discernible difference in behavior.

[stockholmux]

@timmixell Internally, it has to do with how indexing occurs. REPLACE is effectively 'remove old doc' and 'add new doc' - so indexing be re-done completely on that document. Really, this has zero impact on the functionality aside from speed and the above mentioned issue where the new FT.ADD doesn't match the fields of the previous.

REPLACE PARTIAL is only working on the specified fields. So it's like 'remove old field(s)' and 'add new field(s)'.

The other difference with partial is the doc score. You can alter it without having re-index anything. It's very useful.

For the most part 99.9999% of the time you want REPLACE PARTIAL.

[timmixell]

@stockholmux that aligns with the documentation, but I'm not seeing that difference in behavior (using the commands I cited above):

127.0.0.1:6379> FT.CREATE gh1193 SCHEMA text1 TEXT text2 TEXT numeric1 NUMERIC
OK
127.0.0.1:6379> FT.ADD gh1193 doc1 1 FIELDS text1 "hello" text2 "world" numeric1 1
OK
127.0.0.1:6379> FT.GET gh1193 doc1
1) "text1"
2) "hello"
3) "text2"
4) "world"
5) "numeric1"
6) "1"
127.0.0.1:6379> FT.ADD gh1193 doc1 1 REPLACE FIELDS text1 "update" numeric1 100
OK
127.0.0.1:6379> FT.GET gh1193 doc1
1) "text1"
2) "update"
3) "text2"
4) "world"
5) "numeric1"
6) "100"

Based on your description, shouldn't the post-REPLACE response be:

1) "text1"
2) "update"
3) "numeric1"
4) "100"

?

Apologies in advance if I'm confusing the issue.

[stockholmux]

No, not quite. It's about querying the index.

So, with your data, try this:

> FT.SEARCH gh1193 "world"
1) (integer) 0

While using FT.GET you still have:

> FT.GET gh1193 doc1
1) "text1"
2) "update"
3) "text2"
4) "world"
5) "numeric1"
6) "100"

So, effectively, you're deleting the doc from the index and leaving the underlying hash.

[timmixell]

I think I'm back with you. To confirm, the behavior you're requesting with the new HARDREPLACE or FT.ADD + DD would also yield the results I outlined above.

[MeirShpilraien]

@timmixell yes but it will not be called DD (stands for Delete Document). Maybe MD (stands for Match Document)? or HARDREPLACE instead of REPLACE?

[MeirShpilraien]

ohh I see HARDREPLACE already been suggested :)

[MeirShpilraien]

@timmixell, by the way, there is already an open PR for this issue https://github.com/RediSearch/RediSearch/pull/1197 (still need to add the DD/MD/HARDREPLACE as I see from @ashtul last comment)