Faiss: index -1?

Created on 11 May 2020  路  4Comments  路  Source: facebookresearch/faiss

hi,dear
When I query 1000@100 in 10,000,000, I got -1 in index
the codes below refer the script

import faiss
import numpy as np

d = 64                           # dimension
nb = 10000000                      # database size
nq = 10000                       # nb of queries
np.random.seed(1234)             # make reproducible
xb = np.random.random((nb, d)).astype('float32')
xb[:, 0] += np.arange(nb) / 1000.
xq = np.random.random((nq, d)).astype('float32')
xq[:, 0] += np.arange(nq) / 1000.

## Using an IVF index

nlist = 10000
quantizer = faiss.IndexFlatL2(d)  # the other index
index_ivf = faiss.IndexIVFFlat(quantizer, d, nlist, faiss.METRIC_L2)
# here we specify METRIC_L2, by default it performs inner-product search

# make it an IVF GPU index
res = faiss.StandardGpuResources()  # use a single GPU
gpu_index_ivf = faiss.index_cpu_to_gpu(res, 0, index_ivf)

assert not gpu_index_ivf.is_trained
gpu_index_ivf.train(xb)        # add vectors to the index
assert gpu_index_ivf.is_trained

gpu_index_ivf.add(xb)          # add vectors to the index
print(gpu_index_ivf.ntotal)


k = 100                          # we want to see 4 nearest neighbors

D, I = gpu_index_ivf.search(xq, k)  # actual search

print(I[:5])                   # neighbors of the 5 first queries
#1209 print(I[-5:])                  # neighbors of the 5 last queries

results

[[ 846  263  425  711  334  541   18  617  108   58    2  199  175 1071
    26   57    4  411  119  363  794  393  268  160 1214  243  287  483
   923  331  215  637  179 1292  784  282  678  406  112  439 1008   32
  1306  543 1308  288  214  493  614  277   25  788  188    1  653  833
    39  250  317  357  308   69  201  817 1040  236  157  531  674  975
   132  917  178  806  166  186  602  558   27   77  364 1027  170  257
   443 1187  592 1130  144  924  774  616  463  779  271  918  114  515
   478   73]
 [ 429  614  439  935  578   95 1071  653  617  758  663  250  500  492
   766  107  164   58  363  552  883  268   39  698    4  199  400   26
   535  282  784   69  813    2  426  543  910  132  430  215  616  588
    21   77 1027  911  600  756 1022  308 1124  786  381  372  114  917
   305  144  175  700  161  186  448  763  549   64  695  484  174  277
   263  397   73  271  794  613  364  533  642  820  352  214  674  289
   918  331 1292  670  159  288  343  237  515  721  406 1130  531  461
    18  127]
 [ 208  907  227 1053  811  398  419  417   41  689  491  574  392  611
   283  235  294   76   44  703  403  117   59  291  101  193  673   81
   845  387  796  274  173  510  848 1060  482  198  444  153  661   78
    42  529  705  121  409  118  780  133  142  383  467  842   53  532
    60 1038  196  440  218   22  527  253  116  692   85  494  100  192
   356  223  560  373  158 1083   50 1063  244  787  350  944  608  278
     7  258  391   74   47  140   51  498 1219   48  206  110  194  247
   389  877]
 [ 364  600  237  343  461  918  425  924  573  215  766  147 1122  614
   601  381  575  493  109  989  224   64  808 1026 1187  217  239  698
  1174  155  526  884  263  457  833  847  616  150  186  174  166  935
   637  539  363  794  504 1160  923  650  500  478  846  424  183  426
   653  886   34  492  883  919  544  588  282 1032 1699 1143  397  819
   270 1040  786  308  740  738  487  361  617  670  641 1008  809  359
  1090  869  515  289  784  700  868  393  377  528  268 1364  107 1292
   721 1141]
 [ 767 1228  230 1196  706 1365  256 1488  562  940  718  346  658   43
  1431 1232  761  837 1108  593 1674  633 1363  152  477 1139  171 1198
   938  672  190 1328  472  525  885  855  977  704  694   86 1150  321
   486 1128  659 1019  881  354  850  742  412 1203  957 1179  496  905
   229 1119  464 1798 1118  402 1186  360 1200  629  985 1519 1483   -1
    -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1
    -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1   -1
    -1   -1]]

I have run twice,also the same problem
could you pls help me ?

thx

OS:
Faiss version:
Running on:

  • [x] GPU
    Interface:
  • [x] Python
help wanted

All 4 comments

Ubuntu1804 faiss-gpu

Frequently asked question: https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids

Consider increasing the nprobe parameter (or decreasing nlist).

hi,dear
If I want to use xb=100M or more ,then how to set the index ?
thx
@Enet4

No activity, closing.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

zoe-cheung picture zoe-cheung  路  3Comments

daniellevy picture daniellevy  路  3Comments

brunodoamaral picture brunodoamaral  路  3Comments

0DF0Arc picture 0DF0Arc  路  3Comments

lukedeo picture lukedeo  路  3Comments