Faiss: Segfault when reconstructing non-existing id

[mdouze]

You could catch the exception then.
There is no other function to check if an id exists.

@mdouze when I reconstruct from a id not existing in the index, it gives me "Segmentation fault", which leads to python interpreter crash and cannot be caught. I am using 1.4 version from conda. Any workaround to check if an id exists? Thanks.

_Originally posted by @mylyu in https://github.com/facebookresearch/faiss/issues/439#issuecomment-451914888_

[mdouze]

@mylyu could you post a short reproduction script?

[mylyu]

@mylyu could you post a short reproduction script?

@mdouze Here is one, which is simply modified from https://github.com/facebookresearch/faiss/blob/master/tutorial/python/2-IVFFlat.py
by adding

index.reconstruct(100001)
index.reconstruct(999100001)

the former line gives

python: InvertedLists.cpp:56: virtual const uint8_t* faiss::InvertedLists::get_single_code(size_t, size_t) const: Assertion `offset < list_size (list_no)' failed.
Aborted.

the latter line gives

Segmentation fault

Neither can be caught.

reconUnknownIds.zip

[Enet4]

I can reproduce the problem on this side.

Line 40 is expected to fail, because the given key is outside the limits of the index, which is checked internally by assert. I guess that making this a recoverable error rather than a hard assertion would be preferable.

    print(index.reconstruct(100001))

A more serious problem occurs in the following instruction (line 49), where a key of greater magnitude is passed to reconstruct. Even though the key is an integer and theoretically fits in a idx_t, the program crashes.

    print(index.reconstruct(999100001))

[mdouze]

Fixed in internal FB version. Will be included in the next Faiss release.
Diff here:
https://gist.github.com/mdouze/ad61c0f345c75303ca1f032550279ace