hi,dear
When I query 1000@100 in 10,000,000, I got -1 in index
the codes below refer the script
import faiss
import numpy as np
d = 64 # dimension
nb = 10000000 # database size
nq = 10000 # nb of queries
np.random.seed(1234) # make reproducible
xb = np.random.random((nb, d)).astype('float32')
xb[:, 0] += np.arange(nb) / 1000.
xq = np.random.random((nq, d)).astype('float32')
xq[:, 0] += np.arange(nq) / 1000.
## Using an IVF index
nlist = 10000
quantizer = faiss.IndexFlatL2(d) # the other index
index_ivf = faiss.IndexIVFFlat(quantizer, d, nlist, faiss.METRIC_L2)
# here we specify METRIC_L2, by default it performs inner-product search
# make it an IVF GPU index
res = faiss.StandardGpuResources() # use a single GPU
gpu_index_ivf = faiss.index_cpu_to_gpu(res, 0, index_ivf)
assert not gpu_index_ivf.is_trained
gpu_index_ivf.train(xb) # add vectors to the index
assert gpu_index_ivf.is_trained
gpu_index_ivf.add(xb) # add vectors to the index
print(gpu_index_ivf.ntotal)
k = 100 # we want to see 4 nearest neighbors
D, I = gpu_index_ivf.search(xq, k) # actual search
print(I[:5]) # neighbors of the 5 first queries
#1209 print(I[-5:]) # neighbors of the 5 last queries
results
[[ 846 263 425 711 334 541 18 617 108 58 2 199 175 1071
26 57 4 411 119 363 794 393 268 160 1214 243 287 483
923 331 215 637 179 1292 784 282 678 406 112 439 1008 32
1306 543 1308 288 214 493 614 277 25 788 188 1 653 833
39 250 317 357 308 69 201 817 1040 236 157 531 674 975
132 917 178 806 166 186 602 558 27 77 364 1027 170 257
443 1187 592 1130 144 924 774 616 463 779 271 918 114 515
478 73]
[ 429 614 439 935 578 95 1071 653 617 758 663 250 500 492
766 107 164 58 363 552 883 268 39 698 4 199 400 26
535 282 784 69 813 2 426 543 910 132 430 215 616 588
21 77 1027 911 600 756 1022 308 1124 786 381 372 114 917
305 144 175 700 161 186 448 763 549 64 695 484 174 277
263 397 73 271 794 613 364 533 642 820 352 214 674 289
918 331 1292 670 159 288 343 237 515 721 406 1130 531 461
18 127]
[ 208 907 227 1053 811 398 419 417 41 689 491 574 392 611
283 235 294 76 44 703 403 117 59 291 101 193 673 81
845 387 796 274 173 510 848 1060 482 198 444 153 661 78
42 529 705 121 409 118 780 133 142 383 467 842 53 532
60 1038 196 440 218 22 527 253 116 692 85 494 100 192
356 223 560 373 158 1083 50 1063 244 787 350 944 608 278
7 258 391 74 47 140 51 498 1219 48 206 110 194 247
389 877]
[ 364 600 237 343 461 918 425 924 573 215 766 147 1122 614
601 381 575 493 109 989 224 64 808 1026 1187 217 239 698
1174 155 526 884 263 457 833 847 616 150 186 174 166 935
637 539 363 794 504 1160 923 650 500 478 846 424 183 426
653 886 34 492 883 919 544 588 282 1032 1699 1143 397 819
270 1040 786 308 740 738 487 361 617 670 641 1008 809 359
1090 869 515 289 784 700 868 393 377 528 268 1364 107 1292
721 1141]
[ 767 1228 230 1196 706 1365 256 1488 562 940 718 346 658 43
1431 1232 761 837 1108 593 1674 633 1363 152 477 1139 171 1198
938 672 190 1328 472 525 885 855 977 704 694 86 1150 321
486 1128 659 1019 881 354 850 742 412 1203 957 1179 496 905
229 1119 464 1798 1118 402 1186 360 1200 629 985 1519 1483 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
-1 -1]]
I have run twice,also the same problem
could you pls help me ?
thx
OS:
Faiss version:
Running on:
Ubuntu1804 faiss-gpu
Frequently asked question: https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids
Consider increasing the nprobe parameter (or decreasing nlist).
hi,dear
If I want to use xb=100M or more ,then how to set the index ?
thx
@Enet4
No activity, closing.