Theano: Multiple GPU cublasSgemm failed

[ghost]

The _cublasSgemm failed_ error arises because of using _theano.sandbox.cuda.use()_ twice on a node with _multiple GPUs_ (NVidia Titan Z dual card, Ubuntu 14.04 + CUDA 7.0 + bleeding-edge Theano).
Both GPUs have EXCLUSIVE_PROCESS enabled. So, only 1 process can be mounted on 1 card (using in a cluster).
It order to verify which GPU is not used, I test sequentially: I first try to use the first GPU, and then the second one.

In my python script I do:
1) theano.sandbox.cuda.use('gpu0');
2) when it fails because gpu0 already has a process running on it, I call theano.sandbox.cuda.use('gpu1') second time;
3) everything seems OK at this point, but when Theano starts to use cuBlas for matrix operations on this second card (lasagne rnn example using scan()), it fails with cublasSgemm error.

The problem arises when I reuse theano.sandbox.cuda.use() within the same python interpreter.
When I launch theano.sandbox.cuda.use() in 2 different interpreters sequentially, there problem disappears.
It seems to me that cuBlas still tries to map on the first GPU.

I suggest that not all initializations (e.g. cuBlas initializations) are performed when using theano.sandbox.cuda.use() second time within the same process.

The similar issue arose before and it was partially solved, but I did not found clear solution for my use-case.
https://groups.google.com/forum/#!topic/theano-users/Pu4YKlZKwm4/discussion
http://stackoverflow.com/questions/28187868/theano-cublassgemm-failed-14-an-internal-operation-failed

Thank you.

[nouiz]

Just use the flag device=gpu

this let the driver select the frist GPU available. We use the exclusive
mode here too and it work correctly. Theano do not support changing GPUs
once initialized, but this isn't needed.

On Fri, Sep 11, 2015 at 12:35 PM, Viacheslav Khomenko <
[email protected]> wrote:

The _cublasSgemm failed_ error arises because of using
_theano.sandbox.cuda.use()_ twice on a node with _multiple GPUs_ (NVidia
Titan Z dual card, Ubuntu 14.04 + CUDA 7.0 + bleeding-edge Theano).
Both GPUs have EXCLUSIVE_PROCESS enabled. So, only 1 process can be
mounted on 1 card (using in a cluster).
It order to verify which GPU is not used, I test sequentially: I first try
to use the first GPU, and then the second one.

In my python script I do:
1) theano.sandbox.cuda.use('gpu0');
2) when it fails because gpu0 already has a process running on it, I call
theano.sandbox.cuda.use('gpu1') second time;
3) everything seems OK at this point, but when Theano starts to use cuBlas
for matrix operations on this second card (lasagne rnn example using
scan()), it fails with cublasSgemm error.

The problem arises when I reuse theano.sandbox.cuda.use() within the same
python interpreter.
When I launch theano.sandbox.cuda.use() in 2 different interpreters
sequentially, there problem disappears.
It seems to me that cuBlas still tries to map on the first GPU.

I suggest that not all initializations (e.g. cuBlas initializations) are
performed when using theano.sandbox.cuda.use() second time within the same
process.

The similar issue arose before and it was partially solved, but I did not
found clear solution for my use-case.
https://groups.google.com/forum/#!topic/theano-users/Pu4YKlZKwm4/discussion

http://stackoverflow.com/questions/28187868/theano-cublassgemm-failed-14-an-internal-operation-failed

Thank you.

—
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/3383.

[nouiz]

I'll close this as there is no activity and the last comment should fix the problem.