Redisearch: [QUESTION] Index stats not updated when documents are deleted

[vruizext]

I had over 10k documents in my index. After deleting of all the documents, I thought that the index would be totally empty, but as from the stats in FT.INFO, it seems there is some data that remains in the index, e.g. num_records and num_terms. Not sure if this is the _expected_ behaviour.

Are the dictionaries and other internal structures of the index cleaned up after a document is deleted? Is just the stats that are not updated?

 7) num_docs
 8) "0"
 9) max_doc_id
10) "13641"
11) num_terms
12) "71882"
13) num_records
14) "448391"
15) inverted_sz_mb
16) "2.9528093338012695"
17) offset_vectors_sz_mb
18) "1.0222682952880859"
19) doc_table_size_mb
20) "1.0927619934082031"
21) key_table_size_mb
22) "1.239776611328125e-05"
23) records_per_doc_avg
24) "inf"
25) bytes_per_record_avg
26) "6.9052344940018866"
27) offsets_per_term_avg
28) "2.2153433052848963"
29) offset_bits_per_record_avg
30) "8.6329031348782888"
31) gc_stats
32) 1) current_hz
    2) "98.010002136230469"
    3) bytes_collected
    4) "1561478"
    5) effectiv_cycles_rate
    6) "0.12918853451756157"

I'm using RediSearch 1.0.4

[vruizext]

When I manually run SAVE on the redis server, I see this on STDOUT:

65300:M 14 Jan 20:06:50.215 * <ft> Trie: saving 71882 nodes.
65300:M 14 Jan 20:06:50.536 * DB saved on disk

This means, there is data that stays on the index even when there aren't any documents.

My concern is, what would happen in the long term when there are millions of deletions, there would be an increasing residual memory usage due to this?

[dvirsky]

1. The data is deleted by the garbage collector. if you run FT.INFO a few times you will see the numbers decreasing. When a document is deleted nothing happens, the garbage collector gradually deletes data without hurting performance.

2. perhaps a few stats are wrongly decremented, but rest assured that we test all this for memory leaks, and everything that needs to be deleted is deleted eventually. Try it with like 10 documents and it will happen almost instantly.

[vruizext]

I think I found the answer to my previous question while I was building a test case for redisearch-rb.Here is the log of the redisearch commands

1515975646.393592 [0 127.0.0.1:64409] "FT.ADD" "test_idx" "id_1" "1.0" "FIELDS" "title" "Lost in translation" "director" "Sofia Coppola" "year" "2004"
1515975646.394261 [0 127.0.0.1:64409] "FT.ADD" "test_idx" "id_2" "1.0" "FIELDS" "title" "Ex Machina" "director" "Alex Garland" "year" "2014"
1515975646.396808 [0 127.0.0.1:64409] "FT.DEL" "test_idx" "id_1"
1515975646.397590 [0 127.0.0.1:64409] "FT.SEARCH" "test_idx" "@title:lost"
1515975646.398219 [0 127.0.0.1:64409] "FT.GET" "test_idx" "id_1"
1515975646.398237 [0 ?:0] "HGETALL" "id_1"
1515975646.398913 [0 127.0.0.1:64409] "del" "id_1"

"FT.SEARCH" "test_idx" "@title:lost" doesn't return any match as expected

"FT.GET" "test_idx" "id_1" returns the document, i.e. the document is removed from the index, but it's still stored in redis.

We need to call "del" "id_1" to force the deletion, redisearch is leaving the doc in redis

[dvirsky]

That's on purpose. We support indexing existing docs and indexing a doc in
multiple indexes.

On Mon, Jan 15, 2018, 2:24 AM Victor Ruiz [e-mail] wrote:

I found the answer to my previous question while I was building a test
case for redisearch-rb.Here is the log of the redisearch commands

1515975646.393592 [0 127.0.0.1:64409] "FT.ADD" "test_idx" "id_1" "1.0" "FIELDS" "title" "Lost in translation" "director" "Sofia Coppola" "year" "2004"
1515975646.394261 [0 127.0.0.1:64409] "FT.ADD" "test_idx" "id_2" "1.0" "FIELDS" "title" "Ex Machina" "director" "Alex Garland" "year" "2014"
1515975646.396808 [0 127.0.0.1:64409] "FT.DEL" "test_idx" "id_1"
1515975646.397590 [0 127.0.0.1:64409] "FT.SEARCH" "test_idx" "@title:lost"
1515975646.398219 [0 127.0.0.1:64409] "FT.GET" "test_idx" "id_1"
1515975646.398237 [0 ?:0] "HGETALL" "id_1"
1515975646.398913 [0 127.0.0.1:64409] "del" "id_1"

"FT.SEARCH" "test_idx" "@title:lost" doesn't return any match as expected

"FT.GET" "test_idx" "id_1" returns the document, i.e. the document is
removed from the index, but it's still stored in redis.

We need to call "del" "id_1" to force the deletion

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/RedisLabsModules/RediSearch/issues/261#issuecomment-357555821,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAetup0jSdVCjd55w7YnqAXKiG5GBUAAks5tKpq1gaJpZM4Rdsht
.

[vruizext]

ok, thanks for clarifying

[vruizext]

I overlooked this on the documentation, my bad

After deletion, the document can be re-added to the index. It will get a different internal id and will be a new document from the index's POV.

NOTE: This does not actually delete the document from the index, just marks it as deleted. Thus, deleting and re-inserting the same document over and over will inflate the index size with each re-insertion.

[dvirsky]

Actually this is incorrect now that we have the garbage collector :) if you
want make a PR removing the statement about inflating the index.

On Mon, Jan 15, 2018, 6:15 PM Victor Ruiz [e-mail] wrote:

I overlooked this on the documentation, my bad

After deletion, the document can be re-added to the index. It will get a
different internal id and will be a new document from the index's POV.

NOTE: This does not actually delete the document from the index, just
marks it as deleted. Thus, deleting and re-inserting the same document over
and over will inflate the index size with each re-insertion.

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/RedisLabsModules/RediSearch/issues/261#issuecomment-357727432,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAetuhd2MKQjCr1bDL-C3GoSD2ODWf_uks5tK3mzgaJpZM4Rdsht
.