Mattermost-server: Cannot search CJK contents

[evannook]

Hi,

I've downloaded docker image mattermost/platform:latest and ran it for testing purpose, and found that the current version of mattermost cannot search CJK contents.

[it33]

Please try adding * after your search term and use at least 3 characters? Full text search in your database setup is probably set to English, there would probably need a setting for CJK, if anyone on the community has guidance it would be highly appreciated

[evannook]

I've found related discussions here: https://forum.mattermost.org/t/supporting-full-text-for-japanese-and-other-idiographic-languages/779

I'm willing to help solving this issue.

[it33]

Fantastic! We're open to ideas, which approach do you feel might be best?

[evannook]

IMHO, it's better if we can handle this issue in database layer. According to http://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html, MySQL 5.7+ has built-in CJK full text search capability that can solve the issue. Keep to find out how to gracefully solve this issue with PostgreSQL...

[it33]

Hi @evannook, would you be open to testing the instructions and reporting back if they work?

If so, they could be added to Mattermost documentation to help other CJK users

[it33]

Sorry, hit the wrong button, didn't meant to close,

[artiya4u]

I have a workaround with MySql 5.7, using full text index with ngram parser.
DROP INDEX idx_posts_message_txt ON Posts;
CREATE FULLTEXT INDEX idx_posts_message_txt ON Posts (Message) WITH PARSER ngram;

For more information see MySql document:
ngram Full-Text Parser

[it33]

Thanks @artiya4u!

Really helpful, can anyone in the community verify this approach?

If so, would love to add it to the documentation--and if anyone knows the equivalent for Postgres, please share as well?

[seikichi]

I tried the ngram parser approach. It works fine! Thanks @artiya4u!

I uploaded the docker image to DockerHub, so you can try by the following command:

docker run --name mattermost-cjk -d --publish 8065:80 seikichi/mattermost

If you interested the Dockerfile see this commit.

[it33]

Thanks to everyone for helping solve this issue with CJK search. A note has been added to the database requirements documentation referencing ngram parsing and this thread.

Closing this thread for now. Please file a new issue if there's any further issue with CJK search.

[evannook]

Sorry for replying so late, the solution provided by @artiya4u works for me, thank you all. :D

[schemacs]

@evannook Is there any way to do search CJK on PostgreSQL? Or any help on migration PostgreSQL database to MySQL?

[schemacs]

Found the way to keep dancing on PostgreSQL.

[accordtsai]

I confirm the solution proposed by @artiya4u works on Mattermost 3.5 and MySQL 5.7 community for searching Tradiational Chinese characters.

Is it possible to include this as the default Mattermost behaviour (i.e. using the ngram parser if mysql 5.7+ is detected?)

[it33]

Hi @accordtsai, not sure, we're not really experts. Is this something you'd be interested in contributing to the community?

[rock0018]

@artiya4u ,search simple chinese phase with mysql.5.7.16 also work .thanks a lot