Faiss: Memory error on standard aws machine

[batrlatom]

Summary

I am building an image search service. I tried to make an index with 12500 dimensions ( naive image bottlenecks ) and 5000 images. It is running on standard vps machine with 2GB of ram. But I am getting the memory error. I know that the number of dimensions is pretty high, but I thought that this will fit into memory easily.

Does everything needs to be in memory or is the index able to use some harddisk space? Are there some optimalizations besice reducing number of dimensions which I can do?

I am using IndexFlatL2(d) quantizes and index faiss.IndexIVFFlat(quantizer, d, nlist, faiss.METRIC_L2)

Platform

OS:

Faiss version:

Faiss compilation options:

Running on:

• [x] CPU
• [ ] GPU

Interface:

• [ ] C++
• [x] Python

Reproduction instructions

[beauby]

Your best bet is probably do reduce dimensions first. Otherwise, you can also use an index with compression (see wiki/Lower-memory-footprint). If none of that works for you, you could indeed use on-disk IVF lists.

[batrlatom]

Thanks. I will try it.