Haystack: How to use no_ans_boost in FARMReader?

[antoniolanza1996]

Question

In my setup I want that FARMReader can predict a "no-answer" when there is no answer in the given document.
Then, I've decided to use no_ans_boost parameter but I don't have any idea how to set the best value for it.

First of all, is there a range in which this parameter could vary? In addition, do you have some reference to read about it (e.g. paper, blog post)?

I know that this can be more flexible (in some behaviour) compared to True/False choice in TransformersReader. However, it could be mislead if we don't have any advice to follow during its setup.

[antoniolanza1996]

@tholor @Timoeller I'm just pinging you if you can help me on this issue, please.

[Timoeller]

I am not aware of any papers or blog posts about better QA model confidence and correct handling of no_answer. Unfortunately the no_answer handling is rather complex - though I am sure you could dig into it with a bit of our help : )

Some Background
In the official squad eval script there is the option to include "na_prob_file" scores and do some analysis on it. But getting this na_prob_file is currently not possible in FARM nor haystack.

Explanation of no_ans_boost
The no_ans_boost is changing the no_answer_logit, so positive values make predicting no_ans more likely. Hence the naming "boost". You could try out different boost values to see how your predictions change. But this is a lot of tedious and manual work.

In the end what you need is the no_answer score compared to the text answer score, aggregated over all predictions. This value called "max_no_ans_gap" is returned in the results dictionary (when doing FARMReader.predict()) and computed in this function: https://github.com/deepset-ai/haystack/blob/55552741702dd18384d9b38a436d9c96fb2fdf10/haystack/reader/base.py#L22
You could collect the max_no_ans_gap value for all your QA pairs and see how much you would need to change no_ans_boost in order to optimally return the no_answer.

[Timoeller]

Seems fixed, right @antoniolanza1996 ?
Closing this now, feel free to reopen

[antoniolanza1996]

Seems fixed, right @antoniolanza1996 ?

Yes @Timoeller , thanks.

I'm using no_ans_boost=0 or no_ans_boost=None based on whether I wanna allow a no answer or not. I'm not "boosting" because I prefer to use your algorithm in a more "standard" way.