Sentence-transformers: Any pointers for training other language other than english?

[ec1841]

For training other language other than english, can I train on top of the english pretrained model? How much corpus do I need roughly to get a good fidelity? How do I train for product listing like scenario (max 50 words)? How important is having the dataset with representative vocabularies (ie if we are doing semantic search can it recover well with missing vocab?)

[nreimers]

Hi @ec1841
You are asking difficult questions ;)

For other languages, I would use the Multilingual BERT as base. Then, you would need some training data that fits your later task well. How much training data is needed is hard to answer, as it depends on the complexity of the task + how much quality you need. More training data is usually better.

Your training data should be close to the desired task, BERT/Sentence-BERT does not perform that well when the domain shift is too large.

Getting this up and running with no/little training data is not soo simple with BERT for a different language. It will take some time and effort to get a system that is better than average word embeddings.

Especially in your case I could image that average word embeddings with embeddings trained on your corpus is a quite strong system.

[tphreera]

@nreimers Hi, I'm not sure that if I using other pre-trained like multilingual for Google or Thai pre-trained BERT instead of the model that you provide, is it possible?
I'm very interested in your model architecture design and I want to apply it to another language.
If it's possible to use other model instead, please advice some solution.

[nreimers]

Hi,
BERT, at least the English version, produces out of the box rather bad sentence embeddings. What you need is a fine-tuning step on certain, suitable data. This git provides some examples how to fine tune BERT in this way (see the training_....py examples).

If you have training data for e.g. Thai, you can load the model, tune it and then use it for producing sentence embeddings.

Best
Nils Reimers

[nreimers]

I just release a model for multiple languages:
https://github.com/UKPLab/sentence-transformers/blob/master/docs/pretrained-models/multilingual-models.md