Faiss: Visualization of centroids, inverted lists and compact codes?

[ashfaq92]

nlist = 100
k = 4
d = 2
quantizer = faiss.IndexFlatL2(d)  
index = faiss.IndexIVFFlat(quantizer, d, nlist)
index.train(xb)
index.add(xb)  

I want to print coordinates of centroids, inverted lists and compact codes of vectors for visualization. I can easily get if I run kmeans separately by kmeans.centroids. But how to get them in the above example?

[QwertyJack]

Get the pointer of the object (centroids, invlist, etc.) directly from the index and view it in a proper type. I'll paste an example afterward.

[ashfaq92]

@QwertyJack sure! a (python) example will be good. Thanks

[QwertyJack]

A toy example might seem like this:

# suppose there is an ivf-index, which is already trained and built (add)
index = ...
index.train(...)
index.add(...)

inv = index.invlists
code_size = inv.code_size

# loop through invlists
for i in range(index.nlist):
  # get length of *this* invlist
  list_size = inv.list_size(i)

  # get the centroid of *this* invlist
  centroid = index.quantizer.reconstruct(i)

  # get ids of *this* invlist
  ids = faiss.rev_swig_ptr(inv.get_ids(i), list_size)

  # get codes of *this* invlist; the pointer is represented in `char*` and flattened
  codes = faiss.rev_swig_ptr(inv.get_codes(i), list_size * code_size).view('float32')
  codes.shape = list_size, index.d

  # for `Flat` index, one can assert that codes is equal to the original vectors
  np.all(codes == xb[ids])

[mdouze]

See also
https://github.com/facebookresearch/faiss/wiki/Python-C---code-snippets