Sentence-transformers: How to extract phrase embedding from a given sentence?

[farhaanbukhsh]

The basic requirement we have is we need to get embedding for a given phrase from a given sentences, this decision is based on the understanding that
extracting phrase embedding for a given sentence will hold some kind of contextuality from parent sentences so that we can use them to compare and find similar sentences and
phrases.

For example we have a sentence:

'Flipkart delivered products are defective'

and we need to find all the sentences which are similar and contextual in meaning with the phrase Flipkart delivered as used in the above sentence from our dataset.

Approach

This is how we approached it:

1. Get individual word embeddings through model.encode and passing output_value='token_embeddings'. For example the embedding returned back from our sentence is: 'flip' → E1, '##kar' → E2, '##t' → E3, 'delivered' → E4, 'product' → E5, 'are' → E6 and 'defective' → E7. Here each word is a token and E<number> is the embedding for that token.
2. Then take the embedding out for our interested tokens which is Flipkart and delivered i.e E1, E2, E3, E4
3. Do mean pooling on them to get the combine embedding for the phrase Flipkart delivered.
4. Compare this embedding using cosine similarity with the embeddings of other sentences in our dataset, in order to get contextually similar sentences.

Problems we are facing:

1. model.encode with output_value='token_embeddings' returns a list of token embeddings for a sentence, but it doesn't tell us which embedding belongs to which token, it gets more confusing with complex words for example Flipkart resolves to 'flip', '##kar', '##t' this will return 3 embeddings and we don't know how to map them back? This is required in order to get embedding for each token to do our own mean pooling.

2. While tracing back the code we found that [CLS] and [SEP] token is added to the original set of tokens, For example the above sentence:

'flip' → T1
'##kar' → T2
'##t' → T3
'delivered' → T4
'products' → T5
'are' → T6
'defective' → T7

here we have the tokens from the above sentence and now [CLS] and [SEP] is added to it, so we get

[CLS] T1 T2 T3 T4 T5 T6 T7 [SEP]

and when we embed these we will get:

[CLS Embedding] E1 E2 E3 E4 E5 E6 E7 [SEP Embedding]

how should these be used if we need single token embedding back to be used in mean pooling in order to get the combined embedding out?
For example if I need to pass embedding for Flipkart delivered to my pooling step should I be passing "E1 E2 E3 E4" or "[CLS embedding] E1 E2 E3 E4 [SEP embedding]"?

Any insight for our approach and help to do the same will be highly appreciated.

Thanks in advance.

[nreimers]

Hi @farhaanbukhsh
Yes, these word pieces can drive you crazy. Especially as the different models (BERT, RoBERTa, XLM-R etc.) use different tokenizers with different strategies to break-up words and to encode these break-ups.

You would need to find where 'Flipkart delivered' starts in the tokenized version of the sentence. I don't have a perfect strategy yet, but what you could do is to look at offsets:

1) Find the offset of 'Flipkart delivered' in your sentences. You maybe want to remove white spaces, as the BERT tokenizers ignores the white spaces. In the sentence 'Hello, Flipkart delivered ...' would be normalized to 'Hello,Flipkartdelivered...' and the offset for 'Flipkartdelivered' would be 7 to 24.
2) Then, you tokenize your sentence (with white spaces) with the BERT tokenizers. This gives You something like 'hel' '##llo,' 'flip' '##kar' '#t'....
3) For each word piece, you count what the offset is and if it is in your range of 6 - 23
hel => Offset 0 => Don't include

llo, => Offset 3=> Don't include

flip => Offset 7 => include
...

When computing the offset for the word pieces, you have to remove the ## at the beginning of the word piece to get the right counts.

This should give you which word pieces represent your phrase.

For your second question:
I would not include CLS and SEP token into the mean computation.

Best
Nils Reimers

[shameelct]

Does the new release and wkpool ease arrive at these embeddings?