Haystack: [Epic] Add Benchmarking

[tholor]

There's an overwhelming number of options for retrievers and readers out there. For users, it's often difficult to select the appropriate models / methods for their use case and understand the implications.

Let's add a benchmarking section to the project website (haystack.deepset.ai).

Subtasks:

• Eval reader speed #409
• Eval reader accuracy #410
• Eval retriever speed #411
• Eval retriever accuracy #412
• Design eval charts & Integration on website #413
• Automating runs & storing results #414

[guillim]

That would be very helpful indeed.

And also, would be great to specify the language you are doing the Benchmark with ! (obviously EN, but since you're a German team, and we're French one, let's clear things up 😄 )

Last thing, here at @etalab we have been working on which "k" to choose for the retriever, and in our case, it was pointless to go above 5 since the gain was minimal after 5. A paragraph about _How to choose k_ could be useful for new comers

[tholor]

Good points!

We will specify the language and comply with the bender rule :)

A section about top k is definitely a good idea. On what dataset and retriever did you observe this saturation at 5? We didn't do a study here yet, but I believe the curve also depends on your doc splitting strategy.

[guillim]

Answering _On what dataset and retriever did you observe this saturation at 5?_ :

We are working with @psorianom on this dataset. it's basically this website made into a dataset. The study of k was done with a list of 400 questions on this dataset. Each question was manually linked to its json.

Definitivly, the curve depends on the length of your splitting strategy. We are looking closely to this as well. @psorianom could talk about it better than me. It's our biggest concern at the moment.