Faiss: hnsw result index contains -1

[eshaoliu]

Summary

Platform

centos

OS:

Faiss version:1.6.1

Installed from:

Faiss compilation options:

Running on:

• [ 1] CPU
• [ ] GPU

Interface:

• [ ] C++
• [ 1] Python

Reproduction instructions

My data scale is about 100000000，and the dimension of data is 32.
after build index either by index =faiss.IndexHNSWFlat(len(data[0]),8) or faiss.index_factory(len(data[0]),"HNSW8")
after I search result e.g. for top 10000.
The tail result index is -1
Just like this
dis, ind = index.search(np.array([[0.0,0.0,0.0,0.0,0.0,0.0,5.348159313201904,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,3.13490629196167,0.0,0.0,6.721113204956055,0.0,0.0,0.0,0.02932894229888916,0.35066425800323486,0.0,4.0010085105896,2.194624423980713,0.0,0.0,0.28953737020492554],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.513262927532196,0.0,0.0,0.0,0.0,0.0,0.0,18.823591232299805,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,5.756752967834473,0.0,0.0,0.0,0.0,1.0187498331069946,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,3.2329189777374268,0.0,0.0,0.0,0.0,0.0,15.415619850158691,0.0,0.0,0.0,0.0,6.2131266593933105]]).astype('float32'), 10000)

And for different queries,the number of -1 is different.
When I use the brute-force index(IndexFlatL2),this problem doesn't occur.
I just wonder whether my data scale is so large that hnsw can't index all the data.

[Enet4]

As HNSW indexes are not 100% accurate, a search might yield less results than the ones requested. I am not entirely sure if this is applicable to HNSW, but it seems to be for algorithms with a non-exhaustive search algorithm. See the relevant FAQ. Accuracy can be improved by increasing the HNSW parameters described here.

[mdouze]

Updated the FAQ to mention efSearch for HNSW.

[eshaoliu]

Thanks，so how can I find the default value of the three parameters，I failed to find a parameter api documention for hnsw in faiss(I use python).

As HNSW indexes are not 100% accurate, a search might yield less results than the ones requested. I am not entirely sure if this is applicable to HNSW, but it seems to be for algorithms with a non-exhaustive search algorithm. See the relevant FAQ. Accuracy can be improved by increasing the HNSW parameters described here.