Theano: float64 is faster than float32??

[junku901]

It seems using float64 is faster than float32 in matrix muliplication. I got the better result using float32 only if I used Out(sandbox.cuda.basic_ops.gpu_from_host(a*b),borrow=True). Should I use float64 rather than float32? I'm really confused b/c the documentation is recommending using float32.

import numpy as np
import time
import theano
from theano import sandbox, Out, shared

a = shared(np.array(np.random.rand(500,500),dtype='float32'))
b = shared(np.array(np.random.rand(500,500),dtype='float32'))
f = theano.function([],a*b)
start = time.time()
for i in xrange(5000):
    c = f()
end = time.time()
print "(float32) Time elapsed : %f second " % (end-start)

a = shared(np.array(np.random.rand(500,500),dtype='float64'))
b = shared(np.array(np.random.rand(500,500),dtype='float64'))
f = theano.function([],a*b)
start = time.time()
for i in xrange(5000):
    c = f()
end = time.time()
print "(float64) Time elapsed : %f second " % (end-start)

a = shared(np.array(np.random.rand(500,500),dtype='float32'))
b = shared(np.array(np.random.rand(500,500),dtype='float32'))
f = theano.function([],sandbox.cuda.basic_ops.gpu_from_host(a*b))
start = time.time()
for i in xrange(5000):
    c = f()
end = time.time()
print "(float32 optimized) Time elapsed : %f second " % (end-start)

a = shared(np.array(np.random.rand(500,500),dtype='float32'))
b = shared(np.array(np.random.rand(500,500),dtype='float32'))
f = theano.function([],Out(sandbox.cuda.basic_ops.gpu_from_host(a*b),borrow=True))
start = time.time()
for i in xrange(5000):
    c = f()
end = time.time()
print "(float32 highly-optimized) Time elapsed : %f second " % (end-start)

Result

(float32) Time elapsed : 2.470041 second 
(float64) Time elapsed : 1.159385 second 
(float32 optimized) Time elapsed : 1.616751 second 
(float32 highly-optimized) Time elapsed : 0.410361 second 

I'm using
2.8 GHz Intel Core i7 / NVIDIA GeForce GT 650M / OS X 10.9.2

[abergeron]

First off, 'float64' computations always run on the CPU with the cuda backend (which is why float32 is recommended to actually use the GPU).

Secondly you matrices are somewhat small, only 250000 elements, and you are not doing matrix multiplication (dot) you are doing elementwise multiplication. To actually do a dot product use theano.tensor.dot.

Finally your GPU is an M version which are somewhat under-powered and is probably using shared memory. Comparing to your CPU which is a rather fast Core i7 and probably has a much better caching mechanism for memory loads.

All these factors mean that the transfer time between CPU and GPU of the result at the end in the float32 case completely overshadow the computing time. This is supported by the fact that the only fair comparison, using gpu_from_host and borrow=True, is much faster than the one on the CPU.

It is possible that a more complex operation would still benefit from the GPU on your machine, but you would have to measure that.

[junku901]

Thank you for your great answer. I made a mistake in doing matrix multiplication. I fixed my code to do T.dot() with two 2500x2500 matrices and not to use shared when defining float64 matrices, but I got more strange result than previous one. Using float64 was nearly twice as fast as using gpu_from_host and borrow=True. Here's the following code I used.

a = shared(np.array(np.random.rand(2500,2500),dtype='float32'))
b = shared(np.array(np.random.rand(2500,2500),dtype='float32'))
f = theano.function([],T.dot(a,b))
start = time.time()
for i in xrange(5):
    c = f()
end = time.time()
print "(float32) Time elapsed : %f second " % (end-start)

a = np.array(np.random.rand(2500,2500),dtype='float64')
b = np.array(np.random.rand(2500,2500),dtype='float64')
f = theano.function([],T.dot(a,b))
start = time.time()
for i in xrange(5):
    c = f()
end = time.time()
print "(float64) Time elapsed : %f second " % (end-start)

a = shared(np.array(np.random.rand(2500,2500),dtype='float32'))
b = shared(np.array(np.random.rand(2500,2500),dtype='float32'))
f = theano.function([],sandbox.cuda.basic_ops.gpu_from_host(T.dot(a,b)))
start = time.time()
for i in xrange(5):
    c = f()
end = time.time()
print "(float32 optimized) Time elapsed : %f second " % (end-start)

a = shared(np.array(np.random.rand(2500,2500),dtype='float32'))
b = shared(np.array(np.random.rand(2500,2500),dtype='float32'))
f = theano.function([],Out(sandbox.cuda.basic_ops.gpu_from_host(T.dot(a,b)),borrow=True))
start = time.time()
for i in xrange(5):
    c = f()
end = time.time()
print "(float32 highly-optimized) Time elapsed : %f second " % (end-start)

Result

(float32) Time elapsed : 1.297062 second 
(float64) Time elapsed : 0.059254 second 
(float32 optimized) Time elapsed : 0.578303 second 
(float32 highly-optimized) Time elapsed : 0.113903 second 

Is this because my GPU is underpowered GPU or matrix multiplication is not enough complex operation?

[abergeron]

Now the float64 code is using constant inputs so the dot() is constant-folded at compilation time and the constant result is returned every time you call the function. This is why it's much faster.

[junku901]

Thanks!