Parlai: Creating chatbot in other language

[Wickky]

Are we able to use this for building a chatbot for another language?

[stephenroller]

Sure, the tools are language agnostic. Unfortunately all of our current data sets are in English. We would greatly welcome dialogue datasets in other languages.

[Wickky]

Thank you for answering my question. For training for other language, is it fine to only change the dataset in other language? What else do we need?

[stephenroller]

It’s hard to say, the situation you’ve described is too vague. Ostensibly yes, but maybe not depending on your situation.

[jsedoc]

The main issue you will face is the tokenization. Otherwise, everything should work. Languages like Spanish work almost entirely out of the box whereas Chinese will take some work.

[stephenroller]

We just added c3, which is a Chinese reading comprehension dataset. It's not dialogue, but it's a start! #2665.

[Wickky]

@stephenroller @jsedoc
Thank you for your suggestions.
For the tokenization case, where can I change it?
Suppose I have a tokenization model for my own language, where can I update the code to work?

[stephenroller]

If you want to use a BPE tokenizer, you can follow HuggingFace's instructions to train your own. If you have a premade tokenizer, you can add say wickky_tokenize to this class:

https://github.com/facebookresearch/ParlAI/blob/d191b8063a8b8500737f73b658188d393cc96211/parlai/core/dict.py#L391-L402

def wickky_tokenize(self, text):
     return text.split("x")

And then use rest of parlai with the --dict-tokenizer wickky argument.

[Wickky]

@stephenroller Thank you so much! It works!

[Wickky]

@stephenroller I have one more question sir. When we predict say in interactive mode, does it use the tokenizer we passed while training for the user input message ?

[stephenroller]

Yes.

[Erdene-Ochir0417]

Hello @stephenroller
I have one question. How can I use sentence piece tokenizer?