Sentence-transformers: camemBERT not available

[dataislife]

Hi,

I just saw that you added camemBERT in your models which is fantastic! How one can use it?

model = SentenceTransformer("camembert-base-nli-mean-tokens")

returns an error since it is not in the list of pretrained models in your server.

Cheers,

[nreimers]

Hi,
you have to construct it like this:

# Use CamemBERT for mapping tokens to embeddings
word_embedding_model = models.CamemBERT('camembert-base')

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

[dataislife]

Thanks for your quick answer!

I expected there would be a pretrained-model on NLI-like dataset already available for this model.

So, do you know any NLI-like dataset for training French data? It would be cool to add that to the repo in the future. I can help with this :)

[nreimers]

Sadly I have no idea which datasets are available in French. If you have something, feel free to add it (highly welcome).

Currently I work on multi-lingual Sentence-BERT, which will support 16+ languages including French. I hope the experiments will finish soon.

Best
Nils Reimers

[dataislife]

Hi,

After training my model and fine tuning it in something like NLI for French, I see that there are some discrepancies when loading the model.
Indeed, I had to change the file modules.json in the output folder :
From

[
  {
    "idx": 0,
    "name": "0",
    "path": "0_CamemBERT",
    "type": "CamemBERT"
  },
  {
    "idx": 1,
    "name": "1",
    "path": "1_Pooling",
    "type": "sentence_transformers.models.Pooling"
  }
]

to

[
  {
    "idx": 0,
    "name": "0",
    "path": "0_camemBERT",
    "type": "sentence_transformers.models.CamemBERT"
  },

  {
    "idx": 1,
    "name": "1",
    "path": "1_Pooling",
    "type": "sentence_transformers.models.Pooling"
  }
]

My only question here is: even though I state the "type" to sentence_transformers.models.CamemBERT, will the encode function use the correct weights modified during additional training from the camembert-base model?

How do you advice not to have to override manually the "type" field?
Fyi, it was raising:
```
ImportError: CamemBERT doesn't look like a module path
````

Cheers,

[nreimers]

I was sadly not able to reproduce your error. In my modules.json, the correct (full) name was written:

from sentence_transformers import models, SentenceTransformer

# Use CamemBERT for mapping tokens to embeddings
word_embedding_model = models.CamemBERT('camembert-base')

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

#Save model
model.save('output/camembert')

#Load from disc
model_loaded = SentenceTransformer('output/camembert')

I'm sadly not sure why this happen on your computer.

[pommedeterresautee]

For the French data, you can check Flaubert project (another French BERT), it comes with Flue which tries to be the French Glue and includes NLI and classification among other tasks.
https://github.com/getalp/Flaubert

[dataislife]

@nreimers, I built my own class camemBERT in the same fashion you did using nn.Module (before I saw it was already present in the list of models in the repo). It may be linked to that!

Cheers for the recommendation @pommedeterresautee, will check that now! :)