Haystack: Additional way for reducing computation.

[zshnhaque]

I would like to know about reducing computation time for a response from Finder (I am using Elasicticsearch Retriever). For reference, I have implemented the suggestions given on issue #472 , like using RoBERTa_squad_v2 and deepset/mini_lm , splitting documents which has a total number of characters more than 50k, using lesser value of parameters like top-k-retriever = 5 and top-k-reader = 3 . In case of GPU, I am using AWS EC2 Instance having 16 GB GPU RAM, on average the response time is 4 seconds and for a CPU of 16 GB RAM the response time is 30-40 seconds. I would like to know is there any other recommendation which can reduce the response time in both environments, especially in CPU.

[tholor]

Hey @zshnhaque ,

This seems very slow. What GPU / AWS instance type are you using (RAM is not so important for inference speed)?

I could still see some potential in reducing the split length. If I got you right, you currently feed 5 docs with ~ 50k chars each to the reader. With our new Preprocessor class, you can also split based on the number of words while respecting sentence boundaries. I would really try to reduce this as much as possible until you see a drop in answer accuracy.

In a CPU environment, using the onnx-runtime can yield significant speed-ups. We have this already available in FARM (https://github.com/deepset-ai/FARM/blob/master/farm/modeling/adaptive_model.py#L541). However, it's not integrated yet upstream in Haystack's FARMReader. If you are interested in tackling this with a PR, we could guide you on this.

Another thing you could explore if you are not already using it: multiprocessing for the preprocessing. You can specify the number of processes via FARMReader(num_processes=...)

Regarding, the models, I would also encourage you to try MiniLM again with the latest FARM version (0.5.0). We got a 5x speed improvement on a certain type of questions (where it's hard to find the answer or there's no answer) due to some better vectorization in the postprocessing.

There's also more exciting stuff coming up in Haystack to improve inference speed in the next months (AMP, Pruning, Distillation...)

[zshnhaque]

Hello @tholor , thank you for sharing this.

1) I changed the number of processes using FARMReader(num_processes= ), and time computation is less in case of
deepset/roberta-base-squad2 and deepset/minilm-uncased-squad2.

2) In the case of splitting, I made document total character limit below 50k (<50k) , this splitting method is applied for all documents, and time computation is less in CPU system but I would say changing parameter in FARMReader(num_processes= ) has good impact in reducing computation.

[tholor]

Glad to hear that. To what value did you change num_processes to?

[zshnhaque]

Hello @tholor, in the beginning, while using deepset/roberta-base-squad2 , I kept num_processes = 4 .
For a local machine with 4 GB GPU (use_gpu=True), the time computation was around 7-11 sec.
Then I changed num_processes = None (as suggested here - https://github.com/deepset-ai/haystack/blob/master/haystack/reader/farm.py#L82) . By changing to num_processes = None, I get time computation, between 4 - 7 sec, sometimes even luckier less than 3 seconds. So this kind of time computation is working well in my case.

[tholor]

Ok great! For others who might read this: num_processes=None adjusts the number of processes automatically to your machine by setting it to NUM_CPU_CORES - 1

@zshnhaque FYI we are currently doing a big refactoring of QA preprocessing and expect quite some additional speed improvements there soon :)

[tholor]

Closing this for now. Feel free to re-open if further questions pop up...