Redisearch: Support for Chinese tokenization

[dvirsky]

Following the recent refactor of the tokenizer API, it's now much easier to add additional tokenizers.

Before we'll add it to the extnesion API, let's first add a Chinese one.

A few notes on that:

1. The candidates I'm seeing as valid options are:

   a. https://github.com/yanyiwu/cjieba
   b. https://github.com/lionsoul2014/friso

Another option would be to take their dictionaries, and implement something based on our own triemap, with the advantage of scanning the dictionary letter by letter and not doing a full lookup for each term. That can be left to a later stage or if the libraries are not fast enough.

1. Currently we do not support external data files, and all the algorithms rely on a very big dictionary (a few megabytes). There are two options here: include it as static data in the module and make it very big, or rely on an external file that will need to be distributed alongside redisearch (not a problem in the docker image, more so in Enterprise Cluster).

If we're going with bundling it into the source, we might want to consider making the whole Chinese support a compile time option, as many people will not need it.

Let's start by making this work, and then discuss the packaging options. Another would be a script that downloads the dictionary and saves it into Redis itself :) This might actually a good distribution option if we're going with a non bundled version.

1. Please make sure that whatever we have works well with mixed Chinese / English documents.

2. We will not do automatic language detection, and for now we will not add anything to the API. FT.ADD already supports the LANGUAGE modifier, we can use LANGUAGE chinese to trigger the special tokenizer. So when creating the add document context, based on the language you select the tokenizer.

PS Make sure whatever we end up doing is thread safe as tokenization is done in the thread pool.

[dvirsky]

@mnunberg how is this coming along?

[mnunberg]

Still the weekend for me :) - but I have a better grasp of things now. I'll need help from a chinese-speaker to tell me what the different lexical categories are. I did integrate the library (friso, so far) into the build; and I'll see what we can do about dictionary loading as well.

[dvirsky]

I didn't mean for you to work on the weekend of course... I've asked the people mentioned in the previous Chinese support issue to try this out once you are ready.

[suze521]

When will it be finished?

[kilianc]

Just a heads up, SLOP doesn't appear to work properly with Chinese. Not sure if this is related to tokenization or not.

[mnunberg]

I'll be uploading a very very very basic thing soon :)

[mnunberg]

@suze521 You can start playing around with the cn branch. Currently we are using friso to do tokenization.

In order to make it work properly, you should specify LANGUAGE chinese when adding a document, e.g.

ft.add idx docId1 LANGUAGE chinese FIELDS fld1 ...

When searching, you don't have to specify the language, but if you want to use highlighting/summarization, it might be helpful:

ft.search idx 之旅 LANGUAGE chinese

Please tell me if the search results at all make sense, and any other issues!

I've verified there are no leaks and it works properly (as far as I can tell). I've identified several areas for improvement in performance, so if you find it slow currently, I can tell you that it will get faster.

Note that in order to use the chinese tokenization, you must point it to Friso's dictionary file. This is best done using the DICTFILE environment variable. The simplest way is to do something like:

git clone git://github.com/github.com/lionsoul2014/friso
export DICTFILE=$PWD/friso/friso.ini
redis-server --loadmodule /path/to/redisearch.so ...

[jackygurui]

Initial test shows it works:

127.0.0.1:6379> ft.create chinese_news schema content text
OK
127.0.0.1:6379> ft.add chinese_news test_doc1 1 language chinese fields content "简介Redisson - 是一个高级的分布式协调Redis客户端"
OK
127.0.0.1:6379> keys *
 1) "ft:chinese_news/\xe6\x98\xaf"
 2) "idx:chinese_news"
 3) "ft:chinese_news/\xe4\xb8\x80\xe4\xb8\xaa"
 4) "ft:chinese_news/\xe7\xae\x80\xe4\xbb\x8b"
 5) "test_doc1"
 6) "ft:chinese_news/\xe5\xae\xa2\xe6\x88\xb7\xe7\xab\xaf"
 7) "ft:chinese_news/redis"
 8) "ft:chinese_news/\xe5\x88\x86\xe5\xb8\x83\xe5\xbc\x8f"
 9) "ft:chinese_news/\xe9\xab\x98\xe7\xba\xa7\xe7\x9a\x84"
10) "ft:chinese_news/\xe5\x8d\x8f\xe8\xb0\x83"
11) "ft:chinese_news/redisson"
127.0.0.1:6379> ft.search chinese_news 简介 language chinese
1) (integer) 1
2) "test_doc1"
3) 1) "content"
   2) "\xe7\xae\x80\xe4\xbb\x8bRedisson - \xe6\x98\xaf\xe4\xb8\x80\xe4\xb8\xaa\xe9\xab\x98\xe7\xba\xa7\xe7\x9a\x84\xe5\x88\x86\xe5\xb8\x83\xe5\xbc\x8f\xe5\x8d\x8f\xe8\xb0\x83Redis\xe5\xae\xa2\xe6\x88\xb7\xe7\xab\xaf"
127.0.0.1:6379> ft.search chinese_news 简介
1) (integer) 1
2) "test_doc1"
3) 1) "content"
   2) "\xe7\xae\x80\xe4\xbb\x8bRedisson - \xe6\x98\xaf\xe4\xb8\x80\xe4\xb8\xaa\xe9\xab\x98\xe7\xba\xa7\xe7\x9a\x84\xe5\x88\x86\xe5\xb8\x83\xe5\xbc\x8f\xe5\x8d\x8f\xe8\xb0\x83Redis\xe5\xae\xa2\xe6\x88\xb7\xe7\xab\xaf"
127.0.0.1:6379> ft.search chinese_news (redis|客户端) language chinese
1) (integer) 1
2) "test_doc1"
3) 1) "content"
   2) "\xe7\xae\x80\xe4\xbb\x8bRedisson - \xe6\x98\xaf\xe4\xb8\x80\xe4\xb8\xaa\xe9\xab\x98\xe7\xba\xa7\xe7\x9a\x84\xe5\x88\x86\xe5\xb8\x83\xe5\xbc\x8f\xe5\x8d\x8f\xe8\xb0\x83Redis\xe5\xae\xa2\xe6\x88\xb7\xe7\xab\xaf"

[jackygurui]

It would be really good if the search terms are automatically tokenized too i.e. redis客户端 becomes (redis|客户端) when the language is set to chinese.

[dvirsky]

@jackygurui thanks!

@mnunberg we can do that in the query expansion stage - but I'll need to tweak the API a bit.

[jackygurui]

@dvirsky It would be really useful if the tokenizer engine is swappable as other tokenizers may provide better result under certain use cases.

As you realised all those engines are based heavily on dictionaries, they need to be feed into the engine to work. The real headache is some engine require those dictionaries to be updated regularly in order to "catch up" with the latest "trend" in people's everyday life. So there should be a mechanism of updating those dictionaries without restart the node/cluster, although I guess this also requires the engine to support it.

[mnunberg]

@jackygurui That's totally doable! - we're still trying to figure out how to package the dictionaries. How often does a dictionary need to be updated usually? Is this something done once a day, week, month, year, etc.

[dvirsky]

@jackygurui the thing is that most of the parser and tokenizer ATM are built around understanding the structure of a complex query, and less about splitting the actual words. But there is a second phase post tokenization, where each token is evaluated and can be expanded (or soon - split) based on whatever logic or context you want. This is the ideal place to split Chinese text IMO. Before that we simply split on spaces or on characters that have semantic meanings like ( ) [ ] : @ {} and separate structure from text tokens.

I can have the lexer do an extra friso step per token before sending it to the parser, but it might as well be in the expander, the results will be the same.

[jackygurui]

@mnunberg There isn't an "Official" dictionary nor an update schedule, I am afraid. Typically dictionaries are categorised into smaller subset of domains. i.e. IT, pharmaceutical, places, names of person, etc. It depends on how often a new term is introduced and how popular they become.

It also depends on what you are indexing too. If you are indexing a film or tv show related content, then you need to keep updated quite regularly with the name of the films or tv shows, if you want the search result to be more relevant.

[dvirsky]

@jackygurui another question: in your example you showed that redis客户端 becomes (redis|客户端). Why is it a union and not an intersection? Why not turn it into redis 客户端?

[jackygurui]

@jackygurui another question: in your example you showed that redis客户端 becomes (redis|客户端). Why is it a union and not an intersection? Why not turn it into redis 客户端?

It depends on how you work out the ranking/relevance. redis客户端 means redis clients. Without the space indicates I want to find out more about clients of redis, and less interested in other things about redis and certainly not interested in other db clients.

[mnunberg]

@jackygurui @dvirsky - would it be useful to allow the selection of a sub-dictionary while adding a document? i.e. if you know the "sub-domain" of a document.

Our initial plan was to have a default dictionary compiled into the module (as a compile-time option) with an option to override this. But it sounds like updates would be done potentially rather frequently, especially if we're talking about things like social media.

Regarding query expansion, I'm not sure if we can (or should) run it through friso. In light of the dictionary query above, we'd need to know the dictionaries the document was tokenized with in order to expand the query term correctly. Or is it just a matter of being able to distinguish between Chinese and non-Chinese terms

[jackygurui]

@jackygurui ... But there is a second phase post tokenization, where each token is evaluated and can be expanded (or soon - split) based on whatever logic or context you want. ...

@dvirsky some tokenizer with correct settings/dictionary would do that for you, which is why I am saying there should be a choice of tokenizers.

For example the text I used in my test "简介Redisson - 是一个高级的分布式协调Redis客户端"

With default settings, friso has done it as 简介 Redisson - 是 一个 高级的 分布式 协调 Redis 客户端

While another library would be able to generate an output like 简介 redisson 一个 一 个 高级 分布式 分布 布 式 协调 协 调 redis 客户端 客户 户 端

I am not saying one result is better than the other, it's just one maybe more suitable than another depending on the actual use case.

[dvirsky]

I invite you too look at lexer.rl and see why it's not trivial to use a dictionary while tokenizing the query. The idea is that during the second phase you use a dictionary to split the tokens further if needed.

[jackygurui]

Right, I understand what you mean now. I agree, this can lead to a more relevant result depending on the dictionary it is choose on the second phase.

[mnunberg]

For packaging we've decided for now that:

1. Bundle Friso's dictionaries in its upstream form
2. Write a conversion script to compile friso's dictionaries in compressed form; suitable for inclusion as a header
3. Optionally include this header depending on make target (e.g. make cn)
4. Decompress this with zlib (single-file zlib miniz)
5. Convert in-memory decompressed format to friso's format
6. Profit

Dictionaries can be overridden by just updating the in-source friso format. Currently requires recompilation of the module. Will add support for runtime updating later on, but the above should at least allow a "turnkey" solution for Chinese tokenization.

[jackygurui]

@mnunberg Great stuff, thank you!

@dvirsky @mnunberg 非常感谢！

[dvirsky]

@jackygurui 感谢您的帮助!

[mnunberg]

@jackygurui @kilianc - we've added the splitting of query tokens using friso when LANGUAGE is chinese in the query. However, I have no idea to tell if it actually has the desired effect (other than seeing that the terms are indeed split).

[jackygurui]

Going to give it a try tonight and see.

[jackygurui]

I can see it is working correctly.

127.0.0.1:6379> ft.create chinese_news schema content text
OK
127.0.0.1:6379> ft.add chinese_news test_doc1 1 language chinese fields content "简介Redisson - 是一个高级的分布式协调Redis客户端"
OK
127.0.0.1:6379> ft.add chinese_news test_doc2 1 language chinese fields content "Redisson是架设在Redis基础上的一个Java驻内存数据网格（In-Memory Data Grid）"
OK
127.0.0.1:6379> ft.search chinese_news redis客户端 language chinese
1) (integer) 1
2) "test_doc1"
3) 1) "content"
   2) "\xe7\xae\x80\xe4\xbb\x8bRedisson - \xe6\x98\xaf\xe4\xb8\x80\xe4\xb8\xaa\xe9\xab\x98\xe7\xba\xa7\xe7\x9a\x84\xe5\x88\x86\xe5\xb8\x83\xe5\xbc\x8f\xe5\x8d\x8f\xe8\xb0\x83Redis\xe5\xae\xa2\xe6\x88\xb7\xe7\xab\xaf"
127.0.0.1:6379> ft.search chinese_news redisson客户端 language chinese
1) (integer) 1
2) "test_doc1"
3) 1) "content"
   2) "\xe7\xae\x80\xe4\xbb\x8bRedisson - \xe6\x98\xaf\xe4\xb8\x80\xe4\xb8\xaa\xe9\xab\x98\xe7\xba\xa7\xe7\x9a\x84\xe5\x88\x86\xe5\xb8\x83\xe5\xbc\x8f\xe5\x8d\x8f\xe8\xb0\x83Redis\xe5\xae\xa2\xe6\x88\xb7\xe7\xab\xaf"
127.0.0.1:6379> ft.search chinese_news 内存网格 language chinese
1) (integer) 1
2) "test_doc2"
3) 1) "content"
   2) "Redisson\xe6\x98\xaf\xe6\x9e\xb6\xe8\xae\xbe\xe5\x9c\xa8Redis\xe5\x9f\xba\xe7\xa1\x80\xe4\xb8\x8a\xe7\x9a\x84\xe4\xb8\x80\xe4\xb8\xaaJava\xe9\xa9\xbb\xe5\x86\x85\xe5\xad\x98\xe6\x95\xb0\xe6\x8d\xae\xe7\xbd\x91\xe6\xa0\xbc\xef\xbc\x88In-Memory Data Grid\xef\xbc\x89\xab\xaf"

Basically, with two contents added to one key: 简介Redisson - 是一个高级的分布式协调Redis客户端 and Redisson是架设在Redis基础上的一个Java驻内存数据网格（In-Memory Data Grid）

Searching both Redis客户端 and Redisson客户端 is able to return me back the first result, while searching for 内存网格 has yielded the second result. Just as I expected.

@mnunberg You have done a great job. Thank you!

[dvirsky]

Great. Well, now to the hard part, packaging this thing so that installing it will not be a pain.

[mnunberg]

Packaging is almost done and works out of the box by default. It currently uses Friso's dictionary files current to today in its github repo.

There's also the option to use other dictionary files, by specifying the FRISOINI config option when loading the module.

I'm going to also provide an option to override the values at build-time, in case you want to update the default built-in dictionaries

[mnunberg]

So judging by our wikipedia benchmark, performance is comparable.. 10k in 27s for english, 10k in 34s for chinese. however according to FT.INFO, the english docs average about 1300 records per doc, whereas the chinese ones are only 500 per doc; this might be because english wikipedia articles tend to be more detailed. the CPUs are definitely more active here, but all in all performance seems to be comparable, or at least not slower by several orders of magnitude.

[dvirsky]

@mnunberg great, make a PR when ready and let's merge it.

[mnunberg]

PR is at https://github.com/RedisLabsModules/RediSearch/pull/219

This works well. Some outstanding issues:

• More tests (we could _always_ use more tests:))
• Handling of Latin tokens within Chinese text is currently done within Friso itself rather than using our existing Latin tokenizer:

Regarding the second item:

I would rather it use our normal latin codepath for these cases. The problem in implementing this however is that our stemmer is based of the document's language, which in our case is chinese. We need a way to specify alternate languages - either at the doc level or in the spec level. We might assume the second one is English, but it can also be French, Spanish, or even Hebrew(!, we don't support stemming for that yet, though).

Additionally, we'd need to use this approach when expanding input terms for queries.

All in all I don't believe there's any major difference between how Friso does tokenization and how we do it natively - except maybe things like stopwords and punctuations - and hopefully it has sane defaults.

@jackygurui @kilianc If possible, I'd be interested to see if you could load a bunch of Chinese documents in there, and see whether the search "in general" works as expected (i.e. ordering, etc. makes sense).

[mnunberg]

PR is merged now