Sentence-transformers: Question: Why BERT underperforms

[aclifton314]

I was wondering what your thoughts were on why BERT underperforms at sentence embeddings (i.e. averaging the last output layer or using the CLS token)?

[nreimers]

Hi @aclifton314
See this comment from Jacob Devlin:

Cosine-Similarity / Manhatten-Distance / Euclidean-Distance treat all dimension equally, with the same weight. I.e., to get well working sentence embeddings for unsupervised tasks, it is important that all dimensions "make sense" and have the "same scale".

BERT was not optimized for this, that all output-dimensions must contribute equally. For supervised tasks, the classifier learns which dimension contribute and what the "scale" of those are. But for unsupervised tasks, we sadly don't have this luxury.

I think that is the reason why BERT out-of-the-box does not produce well working sentence representations.

Best
Nils Reimers

[aclifton314]

@nreimers Got it! Thanks!

[sumit11112]

Hi,
How can I check cosine similarity between 2 sentences whose embeddings are scored at different time?

Below code is not working and gives error at cdist(a1, a2, 'cosine')[0][0]

sentence_embeddings = model.encode(['This framework generates embeddings for each input sentence']])
a1 = 1
for sentence, embedding in zip(sentences, sentence_embeddings):
print("Sentence:", sentence)
print("Embedding:", embedding)
a1 = embedding
print("")

sentence_embeddings = model.encode(['Sentences are passed as a list of string.']])
a2 = 1
for sentence, embedding in zip(sentences, sentence_embeddings):
print("Sentence:", sentence)
print("Embedding:", embedding)
a2 = embedding
print("")

import scipy.spatial
yy = scipy.spatial.distance.cdist(a1, a2, "cosine")[0]
print(yy)

Y = cdist(a1, a2, 'cosine')[0][0]
print(Y)