Farm: Masked Language Modelling for QA domain adaptation

[sbhttchryy]

Dear developers, if I have a corpus of .txt files, how can I perform MLM on an existing model? Thank you very much.

[Timoeller]

Hey @sbhttchryy thanks for using farm (again). So you want to continue pretraining on some domain corpus?

We do have an example script for Bert style pretraining with both MLM and NSP. We do not have the option to disable NSP but you can just initialize the AdaptiveModel without the NSP PredictionHead like:

```
model = AdaptiveModel(
language_model=language_model,
prediction_heads=[lm_prediction_head],#, next_sentence_head],
embeds_dropout_prob=0.1,
lm_output_types=["per_token", "per_sequence"],
device=device,
)

If you want to pretrain other models than Bert (electra, roberta) you probably will need to adjust the input tensors to match the model you are using.

[sbhttchryy]

Dear @Timoeller, thank you very much (again). Yes, indeed, I want to continue pretraining for a domain corpus. Using a model that has been finetuned to SQUAD will not be a problem, I am guessing, apart from the adjusting of input tensors? Thanks again (again). :P

[Timoeller]

: )

Sounds interesting, can you elaborate on the use case.
So you trained on QA data, then you want to do domain adaptation with MLM? Do you plan to do QA afterwards on domain data?

[sbhttchryy]

Say I want to use https://huggingface.co/deepset/xlm-roberta-large-squad2 for domain adaptation with MLM. Then I want to use it later for QA on the domain data.

[Timoeller]

Ok I suspected so. I doubt this will work, since when you do MLM you will disturb the contextualized representations (top layer of Bert) that are being used for QA. So it should not work well but I am more than happy to be convinced otherwise. If you succeed this would be worth a paper! So good luck on that journey.

---

Maybe using MLM besides a QA prediction head could help improve domain QA perforamnce, but for this you would nevertheless need domain QA labels.
We do have a label tool as part of haystack which you possibly already have seen: https://annotate.deepset.ai/index.html

[sbhttchryy]

I do have annotated QAs. My plan is to first adapt to the domain using MLM (and NSP?) and then finetune it using the domain labels. For the annotation I have used the cdqa-annotator tool. However the number of QA pairs is around 2500, so I don't know if it might generalize well, which is why I thought doing the MLM and NSP might be good idea.
be:
I understand your point about disturbing the contexualized representation. Should the following be a better course of action b:

1. Using XLM-r
2. Doing MLM and NSP first
3. Training it with SQUAD v2
4. Finetune using our QA pairs.

Or Should finetuning it directly be better?

[Timoeller]

If you have annotated QA pairs the best you could do is https://github.com/deepset-ai/FARM/blob/master/examples/question_answering_crossvalidation.py to see how it performs. Are the labels in a non english language? IF so use the xlm-r model we put on the modelhub. You have to use a low batch size with 1 epoch, or figure out the memory leak. XLM-r somehow does not clear all variables from gpu cache, even across different epochs...

[sbhttchryy]

Thank you so much. I'll keep you updated.