Faiss: Question on Optimized Product Quantization?

[mucun1988]

A quick question.

I am wondering whether the method proposed in “Optimized product quantization” by T. Ge et al. (CVPR 2013) is implemented in FAISS or not.

Thanks,
Matt

[mdouze]

Hi
Quick answer: https://github.com/facebookresearch/faiss/wiki/Pre--and-post-processing#pre-transforming-the-data

[mucun1988]

Dear @mdouze

That's exactly what I am looking for!

Many thanks,
Matt

[yupbank]

The OPQ is implementing non parametric version or parametric version ?

[yupbank]

@mdouze it's the non parametric version right?

https://github.com/facebookresearch/faiss/blob/master/VectorTransform.cpp#L736

[beauby]

@yupbank Yes, it is the non parametric version.

[mucun1988]

So is there a way to call this optimized product quantization (like the way to use faiss.ProductQuantizer() function)?

Thanks,
Matt

[yupbank]

i think somewhere in the LOPQ paper was mentioning non parametric doesn't scale well? what do you think of add a eigenvalue allocation version?

@mdouze

[mdouze]

Hi
The eigenvalue allocation is implemented in PCAMatrix, train with PCAMatrix::balanced_bins set to true. We have not tested it thoroughly though.

[yupbank]

nice, but since @skamalas also works for facebook now, does it make sense to help out here? i think he also do a lot of PQ, maybe add LOPQ and the latest Locally Optimized Hashing here?

[mucun1988]

So is there a way to call this optimized product quantization (like the way to use faiss.ProductQuantizer() function)?

Thanks,
Matt

[skamalas]

@yupbank For now there are no plans for adding LOPQ/LOH in FAISS, I don't really have the time for this, sorry.

In practice I remember I ended up using the parametric version almost everywhere for LOPQ, for efficiency (as the optimization happens locally at each coarse bin and therefore needs to be computed multiple times).