Haystack: Possible tokenization bug that affect answers accuracy.

[kuatroka]

Describe the bug
When running the Tutorial - 3 with multiple models and asking a question "Who killed Ned Stark?" the first result is almost in all models is
1st Answer "n. When Ned\'s father and brother went south to reclaim her, the "Mad King" Aerys Targaryen burned both of them alive. Ned and Robert Baratheon led the"

Some of the 2nd and 3rd answers are actually correct (especially in deepset/minilm-uncased-squad2 and deepset/electra-base-squad2 ), but the very top answer is factually not correct. It's a shame because the rest of the answers are good.

I think that the reason could be a bad tokenization of some cases. In this case the text reads Ned\'s father and brother and maybe the algorithm thinks that Ned is one of the characters in the this very story that later is burned alive when in reality it's Ned's farther and brother are burned.

How can we remove this \ from the text so that the model starts understanding the text better?

Error message
Error that was thrown (if available)

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

To Reproduce
Steps to reproduce the behavior

System:

• OS: Colab Notebook
• GPU/CPU:
• Haystack version (commit or version number): 0.4.0
• DocumentStore:
• Reader:
• Retriever:

[tholor]

Thanks for reporting. I am not sure yet if this is really a tokenization bug, but we will investigate it in our next sprint.

[kuatroka]

If you want me to help you out and test anything, let me know

[tholor]

If you want to help:

• Check how the raw text for the above example looks like. Maybe it already contains "Ned\ 's"? You can find the raw text in the Game of thrones .txt files that we download in the tutorial.
• Check how the tokenizer tokenizes the raw text snippet. Is the \ really introduced there? Is there a difference between models? You could use the Transformers tokenizer directly for this purpose:

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('deepset/electra-base-squad2')
>>> tokenizer.tokenize("When Ned\'s father and brother went south ....")

[kuatroka]

1. The original text in the .txt file #349 from Game of Thrones: "When Ned's father and brother went south to reclaim her, the "Mad King" Aerys Targaryen burned both of them alive" - conclusion - no funny stuff here.

2. When running this function dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True) from the Tutorial #3 (with In-Memory DataStore) - the content of the dicts is all good too - When Ned's father and brother went south to reclaim her, the "Mad King"

3. But when I run predictions with prediction = finder.get_answers(question="Who killed Ned Stark?", top_k_retriever=10, top_k_reader=5) and check the predictions dictionary in its compete form it shows a field called context and it's here where the \
   appears
   'context': ' Ned\'s father and brother went south to reclaim her, the "Mad King" Aerys Targaryen burned both of them alive. Ned and Robert Baratheon led the rebell',

I'm not sure where this contextcomes from and if it affects the results or not.

I don't know if this issue is worth spending your time on or not, it's just I was wondering that if this is a hidden type of an issue that can result in lowering accuracy of semantic similarity tasks or QA. Feel free to close it if not important. Thanks

[lalitpagaria]

To me looks like it's Escape Sequence for Single Quote('). Refer https://docs.python.org/2.0/ref/strings.html

[tholor]

Yes, that was also my first thought.
@bogdankostic Can you please quickly verify that the tokenized passage input to the model doesn't contain the "\ " for this example?

[bogdankostic]

I just checked this and I could not observe any bug there. @lalitpagaria is right, the backslash in this case is used to distinguish single quotes that enclose the string and single quotes that are part of the string. You can check this like this: '\'' == "'" returns True