faiss::gpu::allocMemorySpaceV?

[ahappycutedog]

Summary

Recently I upgraded the cuda version(cuda9.0), and faiss-gpu was reinstalled with conda. However, there is always an out of memory error when using faiss.GpuIndexFlatL2, but the same dataset (shape = (2000000, 512)) has no error with cpu version. Even if the dimension of the data set is reduced (10000*512), there is still an error that the memory is not enough。Why is that?

Platform

ubuntu 16.04
faiss-gpu cudatoolkit=9.0
nvidia GTX 1080ti 11G nums: 2
Faiss compilation options: conda

Running on:
GPU

Interface:
Python

ERRORS:
RuntimeError: Error in void faiss::gpu::allocMemorySpaceV(faiss::gpu::MemorySpace, void**, size_t) at gpu/utils/MemorySpace.cpp:27: Error: 'err == cudaSuccess' failed: failed to cudaMalloc 1610612736 bytes (error 2 out of memory)

[SimlaBurcu]

I have the same problem. Any solutions?

[ckghosted]

I encountered a similar problem and fixed it by limiting the GPU memory usage, say, to 70%, for my program. For example, in Tensorflow:

import tensorflow as tf

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.7)
with tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
    ...

I'm not sure if this is related to the "temporary scratch space" mentioned in https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU
I guess that if I didn't limit the GPU memory usage at the very beginning, the allocation of the "temporary scratch space" will fail and lead to this problem in the run time. Any comment?

[mdouze]

No activity, closing.

[wassryan]

Hi, guys, how do you fix this problem as you mentioned, I encountered this problem too. @ahappycutedog

[ucasiggcas]

dear
have you solved the problem?
Now I have the same questiom
need help
SOS

[hosea7456]

I have the same problem. But I finally resolved it.

This problem maby cause by the " temporary scratch space" which mentioned in :
https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU

So I set the usage of the GPU memory as follows:

config = tf.compat.v1.ConfigProto(allow_soft_placement=True)
config.gpu_options.per_process_gpu_memory_fraction = 0.7
sess = tf.compat.v1.Session(config=config)

Just put it at the head of your run file. And then the code code runs successfully.
Hope this could work for all your guys.