Faiss: more memory is needed when load a small index through read_index

[kakashidan]

Summary

I trained and added a IndexIVFPQ index with a small size vectors (eg. 13119 vectors, 128 dims per each), the IndexIVFPQ params are below:

nlist = 1024
d = 128
m = d / 4
quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFPQ(quantizer, d, nlist, m, 8)
index.train(train_features)
index.add(train_features)
faiss.write_index(index, 'test.index')

Thetest.index on the disk is about 1.2MB, but when I load the index file into memory through read_index, the memory increase about 34MB !! Is something wrong ?

test code is below:

faiss_indexes = list()

while True:
    try:
        message = input("one more try: ")
        index = faiss.read_index('test.index')
        faiss_indexes.append(index)
    except:
        break

for each loop, using ps aux to see the memory occupied by the process.

Platform

OS: Ubuntu16.04

Faiss version:

Faiss compilation options:

Running on:

• [x] CPU
• [ ] GPU

Interface:

• [x] C++
• [x] Python

Reproduction instructions

the test.index is on google drive

[mdouze]

You may want to disable the precomputed tables

index.use_precomputed_tables = 0
index.precomputed_table.resize(0)

This sets the tradeoff between speed and mem usage to favor mem usage.

Even with this, the index will not be as compact in RAM as on disk because of object overheads and std::vector geometric reallocation overheads.

[kakashidan]

@mdouze thank you for your reply. It works for me. In my case, resize is not enough to release precomputed_table, swap(empty_vector) works. :)