Haystack: Combination of multiple retrievers

[tholor]

Originating from a discussion in #104.

What
Combine multiple scores during retrieval (e.g. embeddings similarity of multiple text fields, or BM25 + Embedding similarity ...)

Options

a) Concatenation of Retrievers: Make the Finder accept a list of Retrievers and combine their scores (incl. weights) before feeding results to the Reader
Pro: most generic solution also allowing combination of BM25 + EmbeddingRetriever
Con: Two separate retriever queries (suboptimal efficiency in some cases)

b) New MultiEmbeddingRetriever: Extend the EmbeddingRetriever to the case of multiple Embeddings
Pro: simple implementation without side effects on the regular user interface
Con: won't allow the combination of other retriever methods (e.g. ElasticsearchRetriever + EmbeddingRetriver)

I am leaning towards a), but could see b) as a simpler short-term solution for the original use case discussed in #104 .

┆Issue is synchronized with this Jira Task by Unito

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs.

[tholor]

Still a useful feature. While nothing immediate on our roadmap I would still opt for keeping this one open.

[guillim]

Hey @tholor , I believe this could be a great feature. Especially since BM25 cost almost nothing, and can sometimes bring more suitable documents than EmbeddingRetriver or DensePassageRetriever. We can imagine retrieving k texts for each Retriever, and letting the reader rank all answers afterwards.

[tholor]

Yes, totally agree @guillim. It's on our roadmap for later this year / Q1 2021.
While it's quite straightforward from a technical perspective, I think we need to put some thoughts into the design of the user-facing API (should Finder get a list of retrievers or maybe becomes a fully flexible pipeline where you just stack retrievers, readers, rerankers ...). It will also become tricky in case we support the combination of retrievers that use different documentstores (e.g. DPR with FAISS + Elastic's BM25 with Elasticsearch)

[tholor]

@guillim Quick update: We have the first version now implemented and you can use it like this:

from haystack.pipeline import DocumentSearchPipeline, ExtractiveQAPipeline, Pipeline, JoinDocuments
from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
from haystack.reader.farm import FARMReader
from haystack.retriever.sparse import ElasticsearchRetriever

# Building blocks (= Nodes)
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")
es_retriever = ElasticsearchRetriever(document_store=document_store)
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=False)
...

# Combine via default pipeline (here: QA)
qa_pipe = ExtractiveQAPipeline(reader=reader, retriever=retriever)
res = qa_pipe.run(question="Who is the father of Sansa Stark?", top_k_retriever=2, top_k_reader=5)
print(res)
qa_pipe.draw()


# Or build your own custom pipeline to model complex search routes (e.g. combine multiple retrievers)
p = Pipeline()
p.add_node(component=es_retriever, name="ESRetriever1", inputs=["Query"])
p.add_node(component=dpr_retriever, name="DPRRetriever1", inputs=["Query"])
p.add_node(component=JoinDocuments(join_mode="concatenate"), name="JoinResults", inputs=["ESRetriever1", "DPRRetriever1"])

@tanaysoni We still need to add other options for join_mode (e.g. combining doc scores + weighting)

[guillim]

Nice ! I'll give it a shot very soon.