Faiss: Get index -1 when doing search on GPU

[fabulousfeng]

Summary

Platform

OS:

Faiss version:

Faiss compilation options:

Running on:

• [ ] GPU

Interface:

• [ ] Python

Reproduction instructions

Below is my code, I got -1 in Index I, which is strange, someone tells me what does -1 in index mean? Thank you.

import numpy as np

d = 100                           # dimension
nb = 6000000                      # database size
np.random.seed(1234)             # make reproducible
xb = np.random.random((nb, d)).astype('float32')
xb[:, 0] += np.arange(nb) / 1000.
ns = len(xb)
k = 21
import faiss                     # make faiss available

res = faiss.StandardGpuResources()  # use a single GPU

faiss.normalize_L2(xb)


nlist = 2**12
quantizer = faiss.IndexFlatIP(d)
index_ivf = faiss.IndexIVFFlat(quantizer, d, nlist, faiss.METRIC_INNER_PRODUCT)

# make it an IVF GPU index
gpu_index_ivf = faiss.index_cpu_to_gpu(res, 0, index_ivf)

assert not gpu_index_ivf.is_trained
gpu_index_ivf.train(xb)        # add vectors to the index
assert gpu_index_ivf.is_trained

gpu_index_ivf.add(xb)          # add vectors to the index
print(gpu_index_ivf.ntotal)
gpu_index_ivf.nprobe = int(nlist/10)
_, I = gpu_index_ivf.search(xb[:ns], k)  # actual search
index = faiss.index_gpu_to_cpu(gpu_index_ivf)
faiss.write_index(index, "file.gpu.index")
for i in I:
    print(i)

[fabulousfeng]

Below is an example for one line
[4652 6844 5553 6670 5073 5849 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1]
@mdouze what does these -1 mean?
BTW, can we use add_with_ids for GPU index?

[mdouze]

see
https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids
And yes add_with_ids is supported for GPU.

[fabulousfeng]

see
https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids
And yes add_with_ids is supported for GPU.

@mdouze , I see, it seems that I need to increase the nprobe and gpu_index_ivf.nprobe = int(nlist/10)may not work. However, when I use ParameterSpace().set_index_parameter(index, 'nprobe', 100), it
comes NameError: name 'ParmeterSpace' is not defined, same with GpuParameterSpace

[mdouze]

Prefix with faiss.

[fabulousfeng]

Prefix with faiss.

@mdouze I had tried this, AttributeError: module 'faiss' has no attribute 'ParmeterSpace'
And I installed faiss with conda install faiss-gpu cuda90 -c pytorch

[mdouze]

Check syntax: 'ParmeterSpace' instead of 'ParameterSpace'

[fabulousfeng]

ParmeterSpace

@mdouze Thanks so much, problem solved, ParmeterSpace should be ParameterSpace. However, you may need to fix the below sentence in wiki since I copy from there.

If you use GPU indices, replace ParameterSpace with GpuParmeterSpace.