Faiss: Issue while using add_with_ids with large 64-bit faiss ids

[adityasundaram]

Summary

Getting an error while calling index.add_with_ids()

Platform

OS: Ubuntu 16.04

Faiss version: Faiss pytorch release 1.4

Running on:

• CPU

Interface:

• Python

Reproduction instructions

import numpy as np
import faiss

d = 512 # dimension
nb = 1000 # database size
nq = 100 # nb of queries
np.random.seed(1234) # make reproducible
xb = np.random.random((nb, d)).astype('float32')
xb[:, 0] += np.arange(nb) / 1000.
ids = np.arange(xb.shape[0]) + 999999990100000000
print(type(ids))
print(len(str(ids[0])))
print(ids.shape)
print(ids)

Program to read a pre-trained index for 512 dimensions and adding some vectors to it -

xq = np.random.random((nq, d)).astype('float32')
xq[:, 0] += np.arange(nq) / 1000.
nlist = 4096
index = faiss.read_index("emptyTrainedIVF16384.index")

The index generated was using -> index = faiss.index_factory(d,'IVF4096,Flat')

index.add_with_ids(xb, ids)
res = index.search(xq,10)
print(res)

Error:
Traceback (most recent call last):
File "/home/aditya/test/test_add_with_ids.py", line 23, in
index.add_with_ids(xb, ids)
File "/home/aditya/anaconda3/lib/python3.6/site-packages/faiss/__init__.py", line 106, in replacement_add_with_ids
self.add_with_ids_c(n, swig_ptr(x), swig_ptr(ids))
ValueError: did not recognize array type

[mdouze]

Can you print ids.dtype? It should be int64.

[adityasundaram]

ids.dtype came out to be float64

[adityasundaram]

Is there anyway that faiss would support ids longer than 64 bit integers in the near future?

[beauby]

@adityasundaram Probably not, as we do not support indexing more than 2^64 (~10^19) vectors. If your ids happen to be bigger, you could use sequential ids in Faiss and maintain a map between your ids and Faiss'.