Haystack: finder.get_answers : Improve Search / Answer time

[nsankar]

I had processed 6 mutual fund brochures each with around 65 pages (pdf) using Google Colab ( GPU)

Inferencing samples for a question using finder.get_answers with (top_k_retriever=5, top_k_reader=5 ) takes about 4 minutes as you can see from the %timeit below.

This inferencing performance should be considerably improved through multiprocessing or a suitable method. Otherwise this is a major bottleneck for near real-time answering (response rendering)

/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py:177: ElasticsearchDeprecationWarning: The vector functions of the form function(query, doc['field']) are deprecated, and the form function(query, 'field') should be used instead. For example, cosineSimilarity(query, doc['field']) is replaced by cosineSimilarity(query, 'field').
warnings.warn(message, category=ElasticsearchDeprecationWarning)
07/27/2020 13:53:13 - INFO - elasticsearch - POST http://localhost:9200/document/_search [status:200 request:0.027s]
07/27/2020 13:53:13 - INFO - haystack.finder - Reader is looking for detailed answer in 1513010 chars ...
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.27s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.25s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.25s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.29s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]
07/27/2020 13:54:31 - INFO - elasticsearch - POST http://localhost:9200/document/_search [status:200 request:0.019s]
07/27/2020 13:54:31 - INFO - haystack.finder - Reader is looking for detailed answer in 1513010 chars ...
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.28s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.25s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.29s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]
07/27/2020 13:55:50 - INFO - elasticsearch - POST http://localhost:9200/document/_search [status:200 request:0.020s]
07/27/2020 13:55:50 - INFO - haystack.finder - Reader is looking for detailed answer in 1513010 chars ...
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.28s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.25s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.29s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]
07/27/2020 13:57:09 - INFO - elasticsearch - POST http://localhost:9200/document/_search [status:200 request:0.022s]
07/27/2020 13:57:09 - INFO - haystack.finder - Reader is looking for detailed answer in 1513010 chars ...
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.29s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.25s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:14<00:00, 1.29s/ Batches]
Inferencing Samples: 100%|██████████| 11/11 [00:13<00:00, 1.26s/ Batches]1 loop, best of 3: 1min 18s per loop

┆Issue is synchronized with this Jira Task by Unito

[Timoeller]

Hey @nsankar thanks for using haystack.

The long processing time comes from the powerful reader (Bert based DNN) having to process nearly all documents and your documents are quite large. Quick fix: please split up your documents into smaller meaningful chunks.

Nearly all docs:
You say you have 6 very long documents. When using finder.get_answers with (top_k_retriever=5, top_k_reader=5 ) you preselect 5 of the 6 documents as potential candidates.

Documents being large
You write your docs are about 65 pages each. I would split them up into smaller parts and then index them in haystack. With smaller documents the speed is drastically improved as the reader gets fewer text to go through. IT will still get 5 candidates but these candidates are much shorter.

[nsankar]

Thanks @Timoeller I will try that.

[nsankar]

@Timoeller I was able to improve it using the combination of a) NLTK text splitter b) a hosted elasticsearch server in the cloud and setting the index priority 'higher' in elastic search.. Thanks.

[Timoeller]

Nice. Happy to be able to make the app run faster. Could you give some more insights to us and our community? Improving inference speed is always very important.
E.g.

1. how many text snippets you created with nltk text splitting and how many characters there are per text snippet?
2. processing time before and after splitting + retriever settings in haystack?

[RaviTezu]

@nsankar Could you please provide us more details of your solution so it will help our community ?

[antoniolanza1996]

Hi @nsankar ,

b) a hosted elasticsearch server in the cloud and setting the index priority 'higher' in elastic search

How do you do that? My Elastic Cloud indexes are really slow at search time and I don't have any idea how to speed up.

Thank you

[Timoeller]

Issue seems solved, having additional infos on the solution would be nice though : )

Closing this now, feel free to add infos @nsankar or reopen the issue if needed.

[tholor]

Probably not what @nsankar did, but there's another option to speed up dense retrieval with elastic by using the open distro of elastic with NMSLIB: https://aws.amazon.com/de/about-aws/whats-new/2020/07/cosine-similarity-support-in-amazon-elasticsearch-service/