Sentence-transformers: Doubt about distiluse-base-multilingual-cased

[Guillermogsjc]

HI @nreimers,

first congratulations for the library and the papers about sentence embeddings.

I have a doubt about distiluse-base-multilingual-cased. Why the output tensor dimension for one sentence is 512 instead typical 768? Is not it trained with DistiBERT?

Regards.

[nreimers]

Hi @Guillermogsjc
It is based on DistilBERT, however, with a dense layer, the 768 dimension output were down projected to 512 dimensions.

The model is a distillation of multilingual Universal Sentence Encoder which uses 512 dimension, hence, this down projection was necesary.

Best
Nils Reimers

[Guillermogsjc]

Thank you very much for the quick response.

Kind regards.

[Guillermogsjc]

Hi @nreimers ,

another question is, it is usable this model as it were a transformer from hugginface collection? In that case do you plan to add it to the pool?

The use case is to use it for cold embeddings work like dataset creation etc, then transfer with a softmax or multisigmoidal layer for a text classification problem for example.

Regards.

[nreimers]

Hi @Guillermogsjc
I am working on it to add the models to the huggingface models collection. I should be able to add them within the next 2 weeks.

Best
Nils

[bernardocaldas]

Hi @nreimers , thanks for your work! Are you still planning to add the models to the huggingface collection?

Best,
Bernardo

[nreimers]

Hi @bernardocaldas
Sorry to be behind schedule. Yes, models will be added to the models collections. Hope I can do that soon.

Best
Nils

[Ierezell]

Hi, @nreimers

I have a pipeline for a company to save huggingface model in onnx format and run them in NodeJs. Mostly thanks to hugging face models, scripts for converting and the rust tokenizer repo with bindings in Node.

However, for semantic search, the best results on french text were with the distilUSE sentence bert model you gave. I tried to add a custom hugging face french model (camembert) but I lost perfomance.

I would like to know if adding the models to hugging face can come soon so I will export it from HF because it's really easy and run it in Node. Else if you have another way to pass your SBERT model to onnx i'm good with it.

If you need any help to port SBERT to HF I would be glad to help on a PR.

[nreimers]

Hi @Ierezell
I started to work on to create the needed accounts and to add the models. Should be added soon to HF hub and I will add documentation how the models can be used from there.

For the DistilUSE model I am not yet sure if it will be so easy to add it to Transformers Hub, as it uses a custom down-projection layer on top DistilmBERT.

Hope to update you soon.

Best
Nils

[Ierezell]

Thanks for the quick reply !

It's okay, even if I don't have the distilUSE model I will fine tune it with some (ro)bert-like model and export it to onnx getting the equivalent perfomances.

The biggest problem is to export the all model to onnx and even if I could do it for SBERT by hand, HF offers some nice scripts to do it.

Thanks a lot for you work, sentence embeddings are the next step in language understanding ;)

[nreimers]

Hi @Ierezell
Accessing the hugging face models is quite easy.

If you download the models from here:
https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/

And unzip them, you have two folders: 0_BERT and 1_Pooling.

0_BERT is your huggingface BERT model. 1_Pooling just operates the mean pooling. So the 0_BERT folder contains all needed files and can directly be loaded by huggingface code. It should also be able to export this model to ONNX using their scripts.

Your custom models follow the same format, that 0_... contains your huggingface model.

[nreimers]

Hi @Ierezell
I started to upload the models to HuggingFace models hub:
https://huggingface.co/sentence-transformers

All the other models will follow soon. I need to do some changes by hand to the uploaded files, so that the files are compatible with the most recent version of HuggingFace.

[Ierezell]

Hi @nreimers
Thanks a lot, it's really nice to see someone investing time on user feedback.

As I said before, I could have export everything on onnx and ran it piece by piece (bert => pool => fc) but a hugging face model allow to wrap everything in one place (since I knew they would be exported one day).

Do you plan to do the same for distilUse to have multilingual models accessibles or should I finetune bert models on other languages ?

Again thanks a lot for your work (which beat other embeddings I tried).

[nreimers]

Hi @Ierezell
Do you know how you can export custom layers (mean pooling and down projection) so that it can be loaded via ModelsHub from HuggingFace? Sadly haven't found that function yet.

Otherwise I am not sure what you mean.

[Ierezell]

Hi @nreimers, I read the HF source code and indeed all their poolings are only Linear functions... (or fixed stuff for existing models)
I guess the best option should be to create a new class, respecting the format of the others (Albert / Roberta etc...) and making a push request to HF so they will add a new SentenceBert Model to the model Zoo.

I'm sure they will be happy to help and I would be happy to help you with that.

Bests,