Sentence-transformers: Same model & data, similarity scores changed

[freeIsa]

Hi there,
I've just upgraded to version 0.3.8 from 0.3.2. After the update, I noticed that running the same notebook I was working on, which finds the most similar texts to predefined queries, returned different rankings than before (e.g. one sentence that had similarity score about 0.9 wrt the query - and was therefore ranked as the most similar text in the corpus - is now ranked second with a score of about 0.6).
I am loading the same corpus, using the same text queries and the same model (distiluse-base-multilingual-cased) as before.
Any idea what may have caused this change? I realize that several releases occurred in between so it may be close to impossible to answer, but if you have any clue I'd be curious to know. Thanks in advance!

[nreimers]

Hi,
yes, there was a lot of changes in between. Also the DistilUSE model might have changed in a more recent version, even though that there shouldn't be any major changes in that model.

Do you still have the old model? You can also check in the cache (~/.cache/transformers/sentence_transformer) if you have the old public....._distiluse-base-multilingual-cased model. You could then load it from there and see if you get the old results

[freeIsa]

Thanks for your reply! I managed to load the old model in my cache (hint for anyone that might want to try the same: you will need to add a "model_type": "distilbert" line to the related config.json file to be able to load it). Scores and rankings appear to be the same as the ones I got before the update, so it looks like something may have changed with the model itself.
Was it further fine-tuned on additional data maybe?

[freeIsa]

If you are curious and want to look into this, this is a few lines of code demonstrating that embeddings do change when switching model versions:

MODEL_NAME = '../../../.cache/torch/sentence_transformers/public.ukp.informatik.tu-darmstadt.de_reimers_sentence-transformers_v0.2_distiluse-base-multilingual-cased.zip'
model = SentenceTransformer(MODEL_NAME)
old_model_emb = model.encode("Love is love", convert_to_tensor=True)

MODEL_NAME = 'distiluse-base-multilingual-cased'
model = SentenceTransformer(MODEL_NAME)
new_model_emb = model.encode("Love is love", convert_to_tensor=True)

print(new_model_emb[:5])
print(old_model_emb[:5])

resulting in:

tensor([ 0.0848, -0.0112, -0.0094, -0.0278, -0.0173], device='cuda:0')
tensor([ 0.0863, -0.0242, -0.0092, -0.0280,  0.0026], device='cuda:0')

🤔

[zeno17]

Can confirm @freeIsas post. This resulted in quite a few changes in closest embeddings etc. in my own project

[zeno17]

Would it be possible to leave up the old version somewhere? I dont think i have the old one cached anymore

[nreimers]

Hi,
I uploaded the old version as distiluse-base-multilingual-case-v1, which is a version that supports (only) 15 languages.

The version that supports 50+ languages is uploaded as distiluse-base-multilingual-case-v2.

[zeno17]

Do you know in which version the model was changed? Ive recently finetuned a whole set of models on version 0.3.7 but all prior work etc was done way before that. Id like to know if the work is consistent for the sake of reproducibility

[nreimers]

From the timestamp on the file server, it appears they were uploaded on 29-Sep-2020 20:45.

Sorry for the issues this produces.