Sentence-transformers: Electra model

[e-ndrus]

@nreimers, would be great to have Electra among the available models. Are there plans to include it, as well? It's already available in HuggingFace Transformers.

[nreimers]

It will be available soon. The next version is nearly done, I currently do the last tests.

Best regards
Nils Reimers

[nreimers]

Hi @e-ndrus
A new version was released, with huggingface transformers version 2.8.0. You can use models.Transformer to load the google/electra models.

I trained them with NLI data but sadly didn't get great results so far (not better than the BERT models).

Best
Nils Reimers

[omerarshad]

@nreimers Did you tried the same pipeline? i.e. first trained on snli/Mnli and then trained on STS

[nreimers]

@omerarshad Run it on NLI, but sadly did not yield convincing results (about 5-10 points worse than BERT if I remember correctly).

[omerarshad]

ah.. i have a question, if a model is trained on nli+STS and i want to transfer it to another domain. Is their any technique which preserves semantic capturing of model and just pre-train model on new corpus. Which means that i dont have to retrain new model again on nli+STS.

[nreimers]

Hi @omerarshad
Sadly not as far as I am aware of. After fine-tuning the language model, you would need to re-fine-tune it on e.g. NLI data or something similar suitable for sentence embeddings.

[omerarshad]

ok. can you please share electra trained on NLI model?

[nreimers]

Haven't saved that model. But you can train it yourself by running:
https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_nli.py

And passing the right google/electra model to it.

[PhilipMay]

@omerarshad Run it on NLI, but sadly did not yield convincing results (about 5-10 points worse than BERT if I remember correctly).

Concerning this I opened a new issue: #362

@omerarshad did you find out anything about the electra performance?