Faiss: Running on GPU slower than CPU?

[hminle]

Summary

I use faiss for my own dataset.
First, I try IndexFlatL2 on cpu, it takes around 90 seconds for my dataset
And then, I try multiple gpus by the code below, and it takes around 400 seconds for my dataset.

cpu_index = faiss.IndexFlatL2(d)

gpu_index = faiss.index_cpu_to_all_gpus(  # build the index
    cpu_index
)

So, for the normal index like IndexFlat2D, how can I optimize the performance?

[mdouze]

Hi,
What is the number of vectors, their dimension and how are you performing the searches (by batch or one by one)?

[wickedfoo]

Also, how are you timing the search on the GPU? Are you including the copy of the index to the GPUs?

[hminle]

@mdouze Hi, the size of my embeddings is (23600, 128)
D = 128
I perform the search one by one, not by batch

[hminle]

@wickedfoo I run my script on my own dataset,
First, I run it with simple index (IndexFlat2D).
And then I modify my code to transfer the index to the gpu, and run my script again.

[mdouze]

If you run the search one by one, you cannot take advantage of the GPU because of insufficient inherent parallelism and the synchronization and memory transfer overheads.

[hminle]

@mdouze Thank you a lot. I got it.