Farm: Finetuning German BERT for QA on biomedical domain

[sbhttchryy]

Hello there and thank you very much for this wonderful work. I am relatively new to this field, so please bear with my amateur question. I want to perform question-answering on a German Biomedical text. From what I understand up to now, I need to fine-tune German BERT on biomedical QA datasets. Is there any script/pipeline that I should be using for this?
I have also posted in StackOverflow and the huggingface forum, but to no avail as of now.
Thank you very much in advance.

[vblagoje]

Hey, I am new to FARM as well but I'll attempt to answer nonetheless. I believe you should continue fine-tuning the German BERT model by continuing training German BERT on your biomedical texts. More precisely, you need to continue the masked language modelling task used for pretraining, which should not be too hard if you have a lot of unlabeled data (your biomedical texts). Huggingface has a blog post and a colab notebok with step-by-step instructions. Note however that you would start from the German BERT model and continue training on your corpora. Once you have the German biomedical language model, you can publish it to hugging face and then load it to FARM as you would any other model.

[Timoeller]

As mentioned by @tholor in haystack issue https://github.com/deepset-ai/haystack/issues/192#issuecomment-655389015
you should start with training a multilingual model on english SQuAD and then finetune to your domain with domain QA labels.

We are currently evaluating a multilingual model and how it performs for German QA data. We will update you soon on this progress.

[vblagoje]

@Timoeller I am curious if the approach I suggested works as well? Or be at least be an intermediate pre-training step for further refinement in downstream QA tasks? Thanks in advance!

[Timoeller]

please have a look in the linked haystack issue for your answer.

[Timoeller]

As mentioned in #475
We have trained an XLM-r large on SQuAD v2 and evaluated it on the German parts of XQuAD and MLQA.
We uploaded it to the hf modelhub and added the model card PR
Maybe you can use this model for your use case? We evaluated it on some other German domain QA data and found it to be OK (but not great).

We are also working on a proper German QA dataset, though this will take some more time. Please stay tuned.

[sbhttchryy]

Hello, thank you and congratulations on the great work. I tried using the xlm-roberta-large-squad2 on a simple case on my biomedical corpus. Unfortunately, it does not return the correct answer. Additionally, there is this curious warning:

:07/31/2020 10:20:50 - WARNING - haystack.reader.transformers_utils - Could not map predicted span (0,0) back to token space. Skipping this prediction ..."

@Timoeller could you please shed some light on this? Thank you again.

[Timoeller]

Please try loading the model as a FARMReader. We integrated "no answer" also for the Transformersreader but their pipeline has some peculiarities when it comes to "no answer" handling.

I know this topic is highly related to this thread but please create a haystack issue next time since in FARM there is no haystack.reader.transformers_utils.py

[Timoeller]

Hey @sbhttchryy are there any updates on the progress? Did loading the model as FARMReader help? If so, please close the issue.