Farm: Cannot extract QA answer at beginning of document

[Timoeller]

Describe the bug
Answer at beginning of documents cannot be extracted.

Error message
No errors, just the predictions are wrong.

Additional context
I suspect the logit verification is not correct. We do check weather a start/end logit is within the question tokens, this is possibly flawed.
Doesnt predict correctly neither for bert nor for roberta type QA models.

To Reproduce
compare

inferencer = Inferencer.load(model_name_or_path="deepset/roberta-base-squad2", task_type="question_answering")

    qa_input = [
        {
            "qas": ["What is the largest city in Germany?"],
            "context": "Berlin is the capital and largest city of Germany by both area and population.",
        }
    ]

    results = inferencer.inference_from_dicts(qa_input)
    print(results)


vs 

qa_input = [
    {
        "qas": ["What is the largest city in Germany?"],
        "context": "Document testing. With short text before Berlin it is still no answer. Berlin is the capital and largest city of Germany by both area and population.",
    }
]

vs
```
qa_input = [
{
"qas": ["What is the largest city in Germany?"],
"context": "Document testing this weird behaviour for all kinds of cases. With short text before Berlin it is still no answer. Berlin is the capital and largest city of Germany by both area and population.",
}
]

[ftesser]

I had the same feeling in some test I made, I noticed that a suspected percentage of un-predicted answers was located at the beginning of the documents.

[antoniolanza1996]

Comment from https://github.com/deepset-ai/haystack/issues/476#issuecomment-708447639:

@Timoeller however, even though you have written that QA doesn't predict correctly neither for BERT nor for RoBERTa type QA models, I've noted that for BERT there are no problems. The problems come up only with RoBERTa-Base and ELECTRA-Base. Even with RoBERTa-Large and ELECTRA-Large is ok.

from haystack.reader.farm import FARMReader
models = ["deepset/roberta-base-squad2","deepset/electra-base-squad2", "twmkn9/bert-base-uncased-squad2","twmkn9/albert-base-v2-squad2","deepset/minilm-uncased-squad2","ahotrod/electra_large_discriminator_squad2_512","phiyodr/roberta-large-finetuned-squad2"]
readers = []
for model in models:
  reader = FARMReader(model_name_or_path = model, use_gpu = True)
  readers.append(reader)

question = "What is the largest city in Germany?"
context = "Berlin is the capital and largest city of Germany by both area and population."
for model,reader in zip(models,readers):
  prediction = reader.predict_on_texts(question,[context],5)
  print(f"Model {model}: {[answer['answer'] for answer in prediction['answers']]}")

Output:

Model deepset/roberta-base-squad2: ['Germany by both area and population.']
Model deepset/electra-base-squad2: ['capital']
Model twmkn9/bert-base-uncased-squad2: ['Berlin']
Model twmkn9/albert-base-v2-squad2: ['Berlin']
Model deepset/minilm-uncased-squad2: ['Berlin']
Model ahotrod/electra_large_discriminator_squad2_512: ['Berlin']
Model phiyodr/roberta-large-finetuned-squad2: ['Berlin']

[Timoeller]

@brandenchan the analysis over different model types might help you in debugging that bugger.

It is quite weird that other roberta or electra models (from ahotrod or phiyodr) can predict that first token as answer.
It might be that our deepset models wont predict the first token, since they have never seen such training examples, because of the bug we address in #564
looking forward seeing this fixed

[antoniolanza1996]

@Timoeller but consider that deepset/minilm-uncased-squad2 is also one of your models and, in this case, it correctly predicts 'Berlin'.
If you've trained this model using the same strategy of deepset/roberta-base-squad2 and deepset/electra-base-squad2 it's even weirder.

[brandenchan]

So thank you @antoniolanza1996 for all the analysis and experiments!

I've just uploaded deepset/roberta-base-squad2-v2 which was trained after fixing the train-time first token issue . The performance is better than v1 and we have made sure that it can in fact predict first token. Could you try this out and see if it also fixes the problem on your side?

It is also odd to us that deepset/minilm-uncased-squad2 does not have the first token problem. Perhaps it has learned to focus more on word semantics and less on word position.

[antoniolanza1996]

Hi @brandenchan , thank you too for working on this! I'll try ASAP (hopefully later this week) and I'll let you know on my original issue on Haystack: https://github.com/deepset-ai/haystack/issues/476

[brandenchan]

The model update of deepset/roberta-base-squad2 has fixed the problem. Confirmed in #476