Redisearch: Synonyms in RediSearch

[chuckyz]

Enjoying RediSearch so far, but a big stopper for me personally is the lack of synonyms, e.g.: "kg,kilograms" or "mg,milograms" so, e.g.: FT.SEARCH myindex @weight:"10 kilograms" would need to match a text weight field with "10 kg."

Are there any plans for this, thus far?

[dvirsky]

For the time being you can implement your own query expander that does synonyms. If you want to make a PR for a built in one, I'd be willing to accept it.

[chuckyz]

I'd be down with making a PR. I saw there's quite good documentation on the extensions API itself, but is there any documentation on what's done by default? or is it 'nothing.'

Also what I would personally want is something like a JSON file of

{
 "kg, kilograms",
 "mg, milograms",
 "lbs, pounds",
}

instead of writing a full-on extensions plugin via C.

Would that still fall under that API, or would that be a 'new' feature?

For reference, that's pretty much exactly how ElasticSearch does it; I'm totally open to doing it a more Redis-y way.

[mnunberg]

@dvirsky We could do this at the stemmer level, via implementing an overlay over an existing stemmer. It sounds like a good idea IMO. We already essentially do this for Chinese.

Of course, we don't know when or if we want to do this now.

[chuckyz]

@mnunberg @dvirsky I was discussing this with my co-worker, and we came up with the idea of using a set with a namespace might be better than loading a file, as that way you can keep everything in redis and update it dynamically.

@mnunberg could you point me to where you're doing this with the Chinese language support? I'd like to take a crack at it at some point.

[mnunberg]

@chuckyz reading from a Redis set would not be updated dynamically. It might also cause issues in our clustered/distributed version.

For Chinese support we have a JSON file that's compiled in as a blob (well, it's transformed via JSON etc.) but you can also override it. You can read the bits about the Chinese support on https://github.com/RedisLabsModules/RediSearch/blob/master/src/tokenize_cn.c (and just follow the functions; use grep etc.

Before actually embarking upon writing something, it's best to wait for @dvirsky to respond.

Anyway, the way our Chinese tokenizer works is using a library called friso. Because Chinese doesn't actually have word separators, each sequence of characters is checked against a known dictionary, and all the possible matches are returned and tokenized/indexed/etc.

Note that we use the extension expander API internally as well. See https://github.com/RedisLabsModules/RediSearch/blob/master/src/ext/default.c

[dvirsky]

I would keep it as a special update-able field in the index, much like the stopwords (although they aren't update-able). but the idea of keeping it in redis and not in a file is a good one.

[dvirsky]

Ok, writing a short design here:

1. Synonyms will be part of the index options, set manually. To avoid complicating the index schema creation command, we'll keep a separate command for managing synonyms. Suggested API:

# Add synonyms to an index
FT.SYNADD {index} {term} {synonym} [{term} {synonym} ... ]

# Remove synonyms - either all or a specific term/synonym pair, or all synonyms for a term
FT.SYNDEL {index} [*]|[{term} {synonym}|*]

# List all synonyms for an index - optionally for a specific term
FT.SYNGET {index} [term]

1. Upon indexing, we will look for terms with synonyms, and add a record for the synonym as well as the original term, much like we do with stemming. (Note: perhaps we can use a bloom filter to speed the lookup per term? needs to be tested whether this actually improves anything). Synonyms will be marked with + like we do for stems.

2. When searching we will use the same synonym map to expand the query. i.e. foo to foo|+bar - just like with stems. VERBATIM will make this expansion disabled of course.

[dvirsky]

added in 1.2.0

[kkmuffme]

@dvirsky when will syndel be added?