RediSearch produces corrupt rdb file - 'numericdx'

[mmasouras]

This is an example that causes the error - also see https://github.com/RedisLabsModules/RediSearch/issues/266
for a list of alternative paths to get to the same result.

````
127.0.0.1:6379> FT.CREATE atest SCHEMA textfield TEXT numfield NUMERIC
OK
127.0.0.1:6379> HSET foo textfield blah numfield 1
(integer) 2
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> hset foo numfield 2
(integer) 0
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK

$ redis-server&
.....
12739:M 20 Jun 12:52:22.680 # Error loading data from RDB (short read or EOF). Read performed by module 'ft' about type 'numericdx' after reading '0' bytes of a value.

````

[salt4pommes]

Thats not the same error as #266, because having an payload and then updating a document, causes a different type of corruption on rdb.

[mmasouras]

The error message looks identical.

I can't tell what kind of corruption happens under the hood.

[mnunberg]

The error messages will usually look the same. We'll need to get more detailed in our error reporting.

[mmasouras]

Some more info that might help home-in. The error seems to be caused when using REPLACE even if you haven't changed the value in the db, or if you delete the key from the index:

1. Same value

127.0.0.1:6379> FT.CREATE atest SCHEMA numfield NUMERIC
OK
127.0.0.1:6379> hset foo numfield 1
(integer) 1
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> SHUTDOWN
$(restart)
....
12999:M 20 Jun 15:03:05.838 # Error loading data from RDB (short read or EOF). Read performed by module 'ft' about type 'numericdx' after reading '0' bytes of a value.

1. Add, Delete, Add doesn't work either:

127.0.0.1:6379> FT.CREATE atest SCHEMA numfield NUMERIC
OK
127.0.0.1:6379> hset foo numfield 1
(integer) 1
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> FT.DEL atest foo
(integer) 1
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> SHUTDOWN

$(restart)
....
13015:M 20 Jun 15:04:49.024 # Error loading data from RDB (short read or EOF). Read performed by module 'ft' about type 'numericdx' after reading '0' bytes of a value.

[mmasouras]

Ouch - it also happens if you just delete a document - no replace necessary

127.0.0.1:6379> FT.CREATE atest SCHEMA numfield NUMERIC
OK
127.0.0.1:6379> hset foo numfield 2
(integer) 1
127.0.0.1:6379> FT.ADDHASH atest foo 1
OK
127.0.0.1:6379> FT.DEL atest foo
(integer) 1
127.0.0.1:6379> SHUTDOWN
....
13190:M 20 Jun 15:22:35.527 # Error loading data from RDB (short read or EOF). Read performed by module 'ft' about type 'numericdx' after reading '0' bytes of a value.

[mmasouras]

@mnunberg can you confirm if you can reproduce the error - and if you are trying to fix it/ETA?

As things are now, it's not possible to use redisearch since you are bound to hit one of these obstacles (and we have it running in production live-indexing things as we speak :(

[mmasouras]

I am checking out some older commits - so far it seems that this commit is OK (does not reproduce the error):

commit 57a8bbb4cae0d8011bbbab42618761e8e8a023de (HEAD)
Author: Dvir Volk [e-mail]
Date: Sun Jan 21 17:20:46 2018 +0200

Fixed corruption of rdb due to duplicate numeric entries.

[mmasouras]

@mnunberg - I managed to find a commit that causes this:

Jun 3 https://github.com/RedisLabsModules/RediSearch/commit/e5db15a46d8800c4b328cbd6cbd0538fdf98453d
--> does not work

May 29 https://github.com/RedisLabsModules/RediSearch/commit/dbac7c2ec67aac685dd013067fb6a165ea1ec963
---> works

Current head checkout and build also doesn't work.

[mnunberg]

Ok, I've managed to reproduce it. I think the fix is fairly simple, actually:

diff --git a/src/gc.c b/src/gc.c
index da8e6b14..cae2d97d 100644
--- a/src/gc.c
+++ b/src/gc.c
@@ -272,6 +272,7 @@ size_t gc_NumericIndex(RedisModuleCtx *ctx, GarbageCollectorCtx *gc, int *status
     blockNum = InvertedIndex_Repair(nextNode->range->entries, &sctx->spec->docs, blockNum,
                                     DOCS_TO_SCAN_EACH_ITERATION, &bytesCollected, &recordsRemoved);
     /// update the statistics with the the number of records deleted
+    numericGcCtx->rt->numEntries -= recordsRemoved;
     totalRemoved += recordsRemoved;
     gc_updateStats(sctx, gc, recordsRemoved, bytesCollected);
     // blockNum 0 means error or we've finished

[salt4pommes]

Still crashes when using payloads :( But with another error:

Commands:

FT.CREATE atest SCHEMA textfield TEXT numfield NUMERIC
FT.ADD atest anId 1 PAYLOAD '{"hello":"world"}' FIELDS textfield sometext numfield 1234
FT.ADD atest anId 1 PAYLOAD '{"hello":"world2"}' REPLACE PARTIAL FIELDS numfield 1111
shutdown

Error:

8720:M 22 Jun 09:00:15.005 # The RDB file contains module data for the module '�"' that is not terminated by the proper module value EOF marker

Using latest master from today and redis 4.0.9

[JefStat]

From some testing it looks like enabling AOF recovers the RDB persistence
ie add the following to redis.conf

appendonly yes
aof-use-rdb-preamble yes

My test setup

Redis 4.0.10

127.0.0.1:6379> module list
1) 1) "name"
   2) "ft"
   3) "ver"
   4) (integer) 10200

Corruption steps

FT.CREATE atest SCHEMA textfield TEXT numfield NUMERIC
FT.ADD atest anId 1 PAYLOAD '{"hello":"world"}' FIELDS textfield sometext numfield 1234
FT.ADD atest anId 1 PAYLOAD '{"hello":"world2"}' REPLACE PARTIAL FIELDS numfield 1111
shutdown

Corruption Error
57060:M 22 Jun 14:30:55.930 # Error loading data from RDB (short read or EOF). Read performed by module 'ft' about type 'ft_index0' after reading '0' bytes of a value.

[mnunberg]

RS should be warning or flat-out refusing to load if you don’t set that. What version of Redis?

On Jun 22, 2018, at 2:26 PM, Jef Statham [e-mail] wrote:

From some testing it looks like enabling AOF recovers the RDB persistence
ie add the following to redis.conf

appendonly yes
aof-use-rdb-preamble yes
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/RedisLabsModules/RediSearch/issues/366#issuecomment-399536339, or mute the thread https://github.com/notifications/unsubscribe-auth/AAanUcdbOLCF17CgW69rPBl2u9iPQmlXks5t_TbNgaJpZM4Uv5D3.

[JefStat]

Added more details to my comment. I run redis without AOF by default but I'm making the changes to enable it by default now.

[mnunberg]

Ahh, I meant that AOF without the preamble setting won't work. I'll look into the steps soon.

[mnunberg]

Ok, I've managed to reproduce!-

[salt4pommes]

@JefStat is right :) The database won't be corrupted if you set in config:

appendonly yes
aof-use-rdb-preamble yes

Only rdb seems to be corrupted.

[mnunberg]

Should work now! - and yes, it was related to RDB. - https://github.com/RedisLabsModules/RediSearch/commit/286c79457f025bcea53fb920304b19471ad37485

[mmasouras]

Does that fix the PAYLOADS issue, or also the REPLACE issue reported at the top?

[mnunberg]

There's a commit before that, which fixes the issue at the top

[mmasouras]

I now get a different error for the steps at the top:
44219:M 09 Jul 11:16:46.086 # The RDB file contains module data for the module '' that is not terminated by the proper module value EOF marker

I use redis 4.0.10 and built redisearch from head.

// start redis with no dump.rdb file (no data)

$ redis-server /Users/michaelmasouras/src/redis-4.0.10/redis.conf &

43502:M 09 Jul 11:02:50.626 * RediSearch version 1.2.0 (Git=v1.2.0-159-gacd8b6a)
43502:M 09 Jul 11:02:50.626 * concurrency: ON, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, search pool size: 20, index pool size: 8,
43502:M 09 Jul 11:02:50.627 * Initialized thread pool!
43502:M 09 Jul 11:02:50.627 * Module 'ft' loaded from /Users/michaelmasouras/src/redis-4.0.10/redisearch.so

// make an index and write some data

127.0.0.1:6379> FT.CREATE atest SCHEMA textfield TEXT numfield NUMERIC
OK
127.0.0.1:6379> HSET foo textfield blah numfield 1
(integer) 2
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> hset foo numfield 2
(integer) 0
127.0.0.1:6379> FT.ADDHASH atest foo 1 REPLACE
OK
127.0.0.1:6379> SHUTDOWN

sh43502:M 09 Jul 11:05:53.914 # User requested shutdown...
43502:M 09 Jul 11:05:53.914 * Saving the final RDB snapshot before exiting.
43502:M 09 Jul 11:05:53.914 * Trie: saving 1 nodes.
43502:M 09 Jul 11:05:53.915 * DB saved on disk
43502:M 09 Jul 11:05:53.915 * Removing the pid file.
43502:M 09 Jul 11:05:53.916 # Redis is now ready to exit, bye bye...

// start redis again

$ redis-server /Users/michaelmasouras/src/redis-4.0.10/redis.conf &

44219:M 09 Jul 11:16:46.085 # Server initialized
44219:M 09 Jul 11:16:46.085 * RediSearch version 1.2.0 (Git=v1.2.0-167-g407c761)
44219:M 09 Jul 11:16:46.085 * concurrency: ON, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, search pool size: 20, index pool size: 8,
44219:M 09 Jul 11:16:46.086 * Initialized thread pool!
44219:M 09 Jul 11:16:46.086 * Module 'ft' loaded from /Users/michaelmasouras/src/redis-4.0.10/redisearch.so
44219:M 09 Jul 11:16:46.086 # The RDB file contains module data for the module '' that is not terminated by the proper module value EOF marker

[mmasouras]

I just built from head and this now looks fixed, thanks.