Faiss: Are search operations thread safe?

[burk]

Summary

Clarification around multithreaded use of faiss.

Platform

OS: Ubuntu, Python 2.7

Running on :

• [x] CPU
• [ ] GPU

Reproduction instructions

In the FAQ you state that

Concurrent searches are supported. Concurrent search/add or add/add are not supported. There is no locking mechanism in place to protect against this, but it is easy to maintain a lock in the calling code.

While the page about Threads and asynchronous calls states

Faiss CPU and GPU indices are not thread-safe with respect to multiple user calling threads.

I would like to run a faiss in a gunicorn worker with multiple threads (this means only one copy of the faiss index). If I do locking so that I only have concurrent reads, and not concurrent read+write or write+write, will I be safe? I've tested and it seems to work..

Thanks again!

[wickedfoo]

GPU Faiss is definitely not thread-safe with respect to multiple readers even.

I do not know about the CPU side, but it is probably best to assume that it is not thread-safe either.

Generally, you don't want concurrent reads anyways, unless your index is really small. It would be better to batch the read requests if possible and call search once, as Faiss CPU attempts to use as many cores as possible to parallelize the request. Generally, attempting to multithread on top of Faiss for performance doesn't make sense as Faiss does this internally.

If your requests are coming in on multiple threads, you can send them via a multi-producer, single-consumer queue to a single processing thread, which will batch up the requests if possible (if they are mutually compatible) and search in a batch, then send out the results to the originating threads.

[mdouze]

Hi,
About CPU threading: the documentation was wrong, CPU search is thread-safe.
Please consider the section on search performance
https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls#performance-of-search

[burk]

Thanks! I've understood that batching is the recommended way to achieve best performance. It requires a bit more work when using faiss in a on-line setting though, so that's why we're considering multiple read threads.

[wxingda]

good