Faiss: how to dump faiss index to disk?

[hbyang2]

Summary

Platform

OS:

Faiss version:

Faiss compilation options:

Running on :

• [x] CPU
• [x] GPU

Reproduction instructions

[hbyang2]

how to dump faiss index to disk?

[FlYWMe]

You can use write_index (const Index *idx, FILE *f) in index_io.h , and use read_index when you want to use the index.

[beauby]

Closing this, as @FlYWMe provided the answer. @hbyang2 feel free to keep commenting if you need further help.

[hbyang2]

but i want to dump to the disk by using python language，how to do ? thank you .

[reshu-b7]

Hi @FlYWMe

I need to read_index an write_index in c++
I have written this in c++ faiss::write_index(index, filename);

But I am getting the following errors:

demo_sift1M.cpp:192:31: error: no matching function for call to ‘write_index(faiss::Index**, const char*&)’
   write_index(&index, filename);

I am trying to make these changes in demo_sift1M.cpp

How can I accomplish this in c++

[FlYWMe]

Hi~ @reshu-b7
Did you add #include "faiss/index_io.h" in your demo?

[reshu-b7]

yes, i have added that.

#include <cmath>
#include <cstdio>
#include <cstdlib>
#include <cassert>
#include <cstring>
#include <iostream>
#include <thread>
#include <vector>
#include <boost/filesystem.hpp>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <faiss/IndexPQ.h>
#include <faiss/IndexIVFPQ.h>
#include <faiss/IndexFlat.h>
#include <faiss/index_io.h>
#include <sys/time.h>

#include "../AutoTune.h"

[reshu-b7]

a sample of the script just illustrating the training regime.

int main()
{
    double t0 = elapsed();

    // this is typically the fastest one.
    const char *index_key = "IVF4096,Flat";

    // these ones have better memory usage
    // const char *index_key = "Flat";
    // const char *index_key = "PQ32";
    // const char *index_key = "PCA80,Flat";
    // const char *index_key = "IVF4096,PQ8+16";
    // const char *index_key = "IVF4096,PQ32";
    // const char *index_key = "IMI2x8,PQ32";
    // const char *index_key = "IMI2x8,PQ8+16";
    // const char *index_key = "OPQ16_64,IMI2x8,PQ8+16";

    faiss::Index * index;

    size_t d;
        {

        printf ("[%.3f s] Loading train set\n", elapsed() - t0);

        size_t nt;
        float *xt = fvecs_read("/mnt/kann/data/SIFT1M/sift_learn.fvecs", &d, &nt);
        printf ("[%.3f s] Preparing index \"%s\" d=%ld\n",
                elapsed() - t0, index_key, d);

        const char *filename = "/mnt/kann/data/SIFT1M/sift1M_IVF4096,Flat_trained.index";

        if(boost::filesystem::exists( filename )){
                printf("loading file %s", filename);
                //index = read_index(filename);
                index = faiss::read_index(filename);
        }
        else{
                printf("file does not exist, training and storing the file");
                index = faiss::index_factory(d, index_key);
                printf ("[%.3f s] Training on %ld vectors\n", elapsed() - t0, nt);
                index->train(nt, xt);
                faiss::write_index(&index, filename);

        }
        delete [] xt;

}

[FlYWMe]

How about faiss::write_index(index, filename); ? @reshu-b7

[reshu-b7]

Yes that worked thanks

[carlosost]

First I create the structures...
faiss::IndexFlatIP index(d); faiss::IndexIDMap mapedIndex(&index);

Then insert the vectors into them...
mapedIndex.add_with_ids(n, vectors, ids);

Then write the index to a file...
const char *filename = "teste.index"; faiss::write_index(&mapedIndex, filename);

And then try to load the index again. Here is my problem, because when I try to load the index as an faiss::IndexIDMap it gives me an error. If I load it as an faiss::Index, I don't have the original ID anymore.
faiss::IndexIDMap * mapedIndex2 = faiss::read_index(filename); // It is not implemented faiss::Index * index2 = faiss::read_index(filename); // I loose the original ids

Is there a way to load an index file, originally written from an IndexIDMap index, as an IndexIDMap?

[Enet4]

@carlosost You should be able to explicitly cast the output of read_index to a pointer to IndexIDMap:

auto mappedIndex = static_cast<faiss::IndexIDMap*>(faiss::read_index(filename));

[carlosost]

@carlosost You should be able to explicitly cast the output of read_index to a pointer to IndexIDMap:

auto mappedIndex = static_cast<faiss::IndexIDMap*>(faiss::read_index(filename));

Thank you so much. It worked perfectly.