Hub: Is it possible to fully fine-tune universal-sentence-encoder-multilingual-qa adding new entries to the existing vocab?

[adai183]

Sorry. I am aware that there are a couple of related issues. But the responses are a fuzzy and and also do not seem up to date.

I am building a semantic retrieval application at reply.ai. We have been getting some stunning results with universal-sentence-encoder-multilingual-qa but our corpus is very domain specific and named entities like product names are crucial. So we need to adjust the vocab to our corpus and fine-tune.

Here are the steps I want to implement:

• Add new tokens to the vocab (there should be a term frequency threshold to decide whether a token makes it into the updated vocab)
• Initialize the embedding vectors of the new vocab entries with the mean vector of the entire vocab
• Retrain and update the weights

Is this possible to integrate with universal-sentence-encoder-multilingual-qa ?
I can see there is a planned Cloud AI Workshop. I am hoping the workshop will address this.

Thanks in advance to everybody who can help clarify this.

[ghost]

Same here. Our corpus is very domain specific. We are getting good results with the pre-trained model, but we need to train it further to include words that are specific to our domain. Is there any way to do that or can you suggest a solution, if re-training it with our corpus is not an option?

[vbardiovskyg]

Hi, it is not currently unfortunately not possible to add (or change) terms to vocabulary.

[vitojph]

Hi,

Any news on the starting dates for the experiment on Semantic Similarity for Natural Language? Looking forward to fine tune the models. Thanks!

[gowthamkpr]

Closing this issue as it has been answered. Please add additional comments and we can open the issuer again. Thanks!

[ghost]

Hi, it is not currently unfortunately not possible to add (or change) terms to vocabulary.

@vbardiovskyg what happens to terms that are not in the vocabulary? How are they embedded?

[vbardiovskyg]

Please see: https://groups.google.com/a/tensorflow.org/forum/#!topic/hub/6ZH6UzKtwgA

[adai183]

Hey @vp1993pr !
I can see how it is difficult to to add (or change) terms to vocabulary because of the underlying bucket structure. But I still would like to fine tuned using domain-specific data without modifying the vocab. In the blogpost it says this is possible. Could you please shed some light on how a training pipeline like this would look like? Maybe there is some public code and I just couldn't find it.
Thank you!

[ArmandGiraud]

Hello,
I have the same issue here, I would like to fine-tune or retrain a model from scratch using this module. I could not find the code or data used for the initial training, only the graph or the paper.
Providing the original code and/or data could be very helpful to understand better the behaviour of the model and possibly design a script to fine-tune on another ranking task. Thank you !

[bhavsarpratik]

Hi guys, can we fine-tune the newly released Multilingual USE without adding any vocab as it uses a 128k sentencepiece tokeniser unlike the original which uses GloVe vocab of 400k?

https://arxiv.org/pdf/1907.04307.pdf
"SentencePiece tokenization (Kudo and Richardson, 2018) is used for all of the 16 languages supported by our models. A single 128k SentencePiece vocabulary is trained from 8 million sentences sampled from our training corpus and balanced across the 16 languages."