Sentence-transformers: How to handle OOV words

[harrypotter90]

Hi,
I am using one of the "Sentence similarity" models e.g. 'distilbert-base-nli-stsb-quora-ranking'
As in my domain, I am sure I have quite unique words in my use-case.
How can I handle OOV words when getting sentence embedding?

If there a way I can get a vocabulary list being used to train the model.

Thanks

[nreimers]

Have a look at the BERT paper and the section about word pieces.

BERT (and other transformer networks) don't use words. They have a fixed size vocabulary consisting of character ngrams. So OOV only happen if your words consists of characters that are not in that vocabulary. If you word consists of only characters e.g. a-z, the worst case what can happen is that this word is broken down into individually characters.

[harrypotter90]

Thanks for your quick reply. I understood, But I have realized i am using a different model based on 'Roberta'

I guess i have to read its documentation, how they have handled OOV.

"architectures": [ "XLMRobertaModel" ], "attention_probs_dropout_prob": 0.1, "bos_token_id": 0, "eos_token_id": 2, "gradient_checkpointing": false, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-05, "max_position_embeddings": 514, "model_type": "xlm-roberta", "num_attention_heads": 12, "num_hidden_layers": 12, "output_past": true, "pad_token_id": 1, "type_vocab_size": 1, "vocab_size": 250002 }

And here "vocab_size", I thought its some kind of a number of words.

Problem is that if I use an input word for example "autonomic dysreflexia" then the similarity score between these 2 words: ["dysreflexia is a syndrome", "latent pre"]:

"latent pre" has the highest match with "0.88", which i am not able to make sense of. Also, note that it works great for other examples.

[nreimers]

RoBERTa uses the same approach, but with a larger vocab size (more ngrams), so words are more seldom broken down into smaller pieces.

For individually words, contextualized word embeddings / sentence transformers does not work that well in my experience, especially as it was not trained for it. It was trained on complete sentences from "general english", i.e., for specialized terms, the results can be quite odd.

[harrypotter90]

Hi,
for example, can this be a vocabulary for XLM-R model : https://github.com/facebookresearch/XLM#pretrained-cross-lingual-language-models .

This means that these are the words that this model has seen.

Thoughts?

[harrypotter90]

I have realized that this is the list of vocabulary by which this model has been trained. for example "respira@@" is in that list, which will include respiratory, respiration, etc.
So words that are not in this list, means are out of vocabulary words.

[freeIsa]

Just wanted to add that some relevant OOV words are usually emojis; at least this is the case with distiluse-base-multilingual-cased.
Right now I am experimenting with xlm-r-distilroberta-base-paraphrase-v1 and it looks like emojis are in the tokenizer vocabulary, is that right?

[nreimers]

Have a look here, how XLM-R was created:
https://arxiv.org/abs/1911.02116

[freeIsa]

Have a look here, how XLM-R was created:
https://arxiv.org/abs/1911.02116

Thanks for the pointer, I will look into it - being based on CommonCrawls dumps rather than just Wikipedia data, XLM-R vocabulary should also contain emojis, but I will check this in depth.