Haystack: Annotator for Evaluation

[mathieudumayet]

Question
Hi, is there an annotation machine for evaluation of models?

I used Haystack Annotation Tool to train my models, which worked perfectly. But in order to evaluate the model, the JSON file I created via Haystack Annotation Tool did not work. I get the error: "KeyError: 'title'". Because the json documents the annotation tool creates do not have "title" but "document_id". I changed every single "document_id" to "title" but I get still the same problem.

So do you know any tool that helps to create JSON files that Haystack could read?

Thanks!

Additional context
Add any other context or screenshots about the question (optional).

[Timoeller]

Hey @mathieudumayet thanks for using haystack also in combination with the annotation tool.
I came across the same problem and implemented a fix in https://github.com/deepset-ai/haystack/pull/494

Which version of haystack are you using? Could you try either using current github master or the newest release 0.5.0?
That should fix the Keyerror you are getting - it should only display related warnings.

[alpaltnsy]

Hey @Timoeller ,

I am using version 0.4. I started all my trainings with this version and did not want to have new problems by installing 0.5 in case my current environment would not work with version 0.5, actually.

But I will try it.

Thanks for the solution!

[Timoeller]

K thanks for reporting back. A haystack update should resolve your KeyError issue and hopefully : ) not introduce more errors.

If it works please let us and the rest of the community know.

[alpaltnsy]

Ok, will do!

In case I would like to revert the installation and go back to previous version, I need to run "pip install farm-haystack==0.4", right?

[mathieudumayet]

Thanks @Timoeller . I will try the solution you suggested and also will report back.

[mathieudumayet]

Hi @Timoeller ,

it worked fine. Thanks for the solution.

[Timoeller]

Nice! Will close the issue then.