Faiss: FAISS and PyTorch GPU - questions

[shaibagon]

After looking at the very good example here,
I have a few questions regarding interfacing faiss and pytorch:

1. I would like to use float16 ("half" precision) elements instead of float32. I set
   flat_config.useFloat16 = True for the config when building my index, but I can only see faiss.cast_integer_to_float_ptr - so it seems like I am passing float32 from pytorch to faiss and I get garbage.
   How can I make faiss work with pytorch float16 arrays?

2. My pytorch arrays are stored "transposed" to what faiss expects: pytorch dim-num_vectors, while faiss is num_vectors-dim. Transposing the arrays and making them contiguous() is quite demanding.
   I noticed the index config has an option flat_config.storeTransposed would setting this to True solve my problem? I did not manage to make it work.

3. In your example, you call torch.cuda.synchronize() after searching the index. I noticed that without this call I get garbage - why is that? why do I need to sync if all data and index are already on the GPU (using a single GPU)? Is there a way around this, it seems like this sync operation is time consuming?

I would appreciate a more comprehensive faiss-pytorch example.

Thanks!

OS: Linux

Faiss version: latest (from conda)

Running on:

• [ ] CPU
• [* ] GPU

Interface:

• [ ] C++
• [ *] Python + PyTorch

[mdouze]

My understanding on these points (@wickedfoo can correct me if necessary):

1. useFloat16 is only for the internal storage of some indexes. The data should still be passed as float32 in the interface.

2. again storeTransposed is for the internal data format. It is not supported in the interface.

3. currently the Faiss computations are done in a specific CUDA stream. To synchronize it with the default pytorch cuda stream, you can either use torch.cuda.synchronize() (slow) or res.syncDefaultStreamCurrentDevice() where res is the StandardGpuResources object.

We are not planning to support much more in the following release of Faiss, except searches on a Pytroch tensor allocated externally (without explicitly constructing a GpuIndexFlat).

[shaibagon]

@mdouze thank you for the prompt reply!

Although not on your current workplan, it would be very nice to see more integration between faiss and pytorch.

[wickedfoo]

Adding a way to support passing transposed data to the flat index is something we could probably do, but it isn't in my immediate plans (I'm only working on Faiss GPU very very part time at the moment).

Same thing with regards to float16 support.

How streams are dealt with between Faiss and pytorch is a complication that I need to clean up at some point.

Hitherto all users of Faiss have been for the most part from the CPU only. Faiss does all of its work ordered with respect to its own streams; work performed on the default stream is not ordered wrt work in Faiss.

Calling setDefaultNullStreamAllDevices() on the StandardGpuResources is probably more what you want, but even then, the current pytorch stream need not be the default stream.

https://github.com/facebookresearch/faiss/blob/master/gpu/StandardGpuResources.h#L49

In order to synchronize properly, one would need to be able to call setDefaultStream() passing the current pytorch stream before any Faiss GPU call (unless you know you are only ever on one stream), which does require trying to figure out how to pass a cudaStream_t across the Python boundary. I don't think anyone on our side has done this yet (the GPU APIs were C++ only at one point).

https://github.com/facebookresearch/faiss/blob/master/gpu/StandardGpuResources.h#L45

This is something that I do think we need to fix somehow. I haven't taken a look in a long time what the type of the pytorch stream wrapper is in Python.

[shaibagon]

@wickedfoo Thank you for the detailed response. As a pytorch user I am less familiar (to put it mildly) with GPU streams and the way they are arranged and carried out internally on device. However, being able to call faiss directly from pytroch is a great thing already allowing me to perform all sort of operation I couldn't do before. I really appreciate the efforts you put into this complex integration.

[mdouze]

no activity, closing.