Theano: BaseCorrMM: Failed to allocate output

[vshebunyaev]

I tried( i use CPU and openBLAS) to test sample

import numpy as np
from scipy.misc import ascent
from theano import tensor as T,function
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers.convolutional import Convolution2D, MaxPooling2D
img = ascent()
I = img.reshape((1,1, img.shape[0], img.shape[1]))
w = np.random.randn(1,1,5,5)
nb_output_chan = 1
nb_rows = 5
nb_cols = 5
pool_row = 3
pool_col = 3
model = Sequential()
model.add(Convolution2D(nb_output_chan,nb_rows,nb_cols,border_mode='valid',input_shape=I.shape[1:]))
model.add(MaxPooling2D(pool_size=(pool_row,pool_col)))
model.compile(loss='mse',optimizer='sgd')
out = model.predict(I)

And get error

RuntimeError: BaseCorrMM: Failed to allocate output of 1 x 1 x -3 x 508
Apply node that caused the error: CorrMM{valid, (1, 1)}(InplaceDimShuffle{0,3,1,2}.0, Subtensor{::, ::, ::int64, ::int64}.0)
Toposort index: 5
Inputs types: [TensorType(float32, 4D), TensorType(float32, 4D)]
Inputs shapes: [(1L, 512L, 1L, 512L), (1L, 512L, 5L, 5L)]
Inputs strides: [(2048L, 4L, 1048576L, 2048L), (2048L, 4L, -10240L, -2048L)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Add}[(0, 0)](CorrMM{valid, (1, 1)}.0, InplaceDimShuffle{0,3,1,2}.0)]]

[lamblin]

Please do not use tickets for help requets, use the theano-users mailing list or StackOverflow.

In your case, it looks like you are trying to do "valid" convolution with an image size of (1, 512) and a filter size of (5, 5), which does not make sense. For a valid convolution, the filter has to be smaller than the image.

[nouiz]

Reopening, we should give a better error message, not an unrelated one like now, so reopening.

[vshebunyaev]

I am not sure that because of memory
below i have some additional

Debugprint of the apply node:
CorrMM{valid, (1, 1)} [id A] |InplaceDimShuffle{0,3,1,2} [id B] | |convolution2d_input_1 [id C] |Subtensor{::, ::, ::int64, ::int64} [id D] |InplaceDimShuffle{3,2,0,1} [id E] | |convolution2d_1_W [id F] |Constant{-1} [id G]
|Constant{-1} [id G]

Storage map footprint:

• InplaceDimShuffle{0,3,1,2}.0, Shape: (1L, 512L, 1L, 512L), ElemSize: 4 Byte(s), TotalSize: 1048576 Byte(s)
• convolution2d_input_1, Input, Shape: (1L, 1L, 512L, 512L), ElemSize: 4 Byte(s), TotalSize: 1048576 Byte(s)
• convolution2d_1_W, Shared Input, Shape: (3L, 3L, 512L, 1L), ElemSize: 4 Byte(s), TotalSize: 18432 Byte(s)
• Subtensor{::, ::, ::int64, ::int64}.0, Shape: (1L, 512L, 3L, 3L), ElemSize: 4 Byte(s), TotalSize: 18432 Byte(s)
• TensorConstant{(4L,) of 1}, Shape: (4L,), ElemSize: 4 Byte(s), TotalSize: 16 Byte(s)
• Constant{-1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
• convolution2d_1_b, Shared Input, Shape: (1L,), ElemSize: 4 Byte(s), TotalSize: 4 Byte(s)
  TotalSize: 1085468.0 Byte(s) 0.001 GB
  TotalSize inputs: 1067036.0 Byte(s) 0.001 GB

Just small memory use,

[vshebunyaev]

Problem was in keras,not connected with memory
Needs to set
set_image_dim_ordering

[gvtulder]

This issue, or at least the bit about the error message, might be connected with #5267. That PR changes some error messages as a side effect.

Instead of 'failed to allocate output', this particular error message would become:

BaseGpuCorrMM: impossible output shape 1 x 1 x -3 x 508

which perhaps is still not as informative as it could be.

The cuDNN ops raise this:

RuntimeError: Could not set tensorNd descriptor: CUDNN_STATUS_BAD_PARAM

Printing all three shapes might be better? For example, something like this:

ValueError: GpuCorrMM: impossible output shape
  bottom shape: 1 x 1 x 4 x 10
  weights shape: 1 x 1 x 5 x 5
  top shape: 1 x 1 x 0 x 2

Adding the extra checks and error messages would slightly complicate the CorrMM code.

Improving the cuDNN error requires more extra code, because the shapes are currently computed by cuDNN. An more informative error message with extra shape checks would require a separate shape computation before calling cuDNN. (But the code for these checks could be shared with GpuCorrMM.)

[lamblin]

For cuDNN, I think we are actually computing the shape in the graph before, and passing them to GpuAlloc or GpuEmpty, since the output buffer is passed as an input. I find it strange that this is not the Op that fails.

Edit: Maybe for a shape with a dimension of 0, GpuAlloc would work, but not cuDNN. In that case, we should update the Theano wrapper of cuDNN so that it skips the call and just returns the 0-size array. For negative shapes, though, I would expect GpuAlloc to fail.

[nouiz]

I think gh-5267 goes in the right direction, to at least tell that the
error is shapes related and not memory related.

When a node crash at run time, we print its inputs shapes, so that should
already be avaiable. So I don't understand why you tell it isn't the case.

On Tue, Nov 29, 2016 at 12:22 PM, Pascal Lamblin [e-mail]
wrote:

For cuDNN, I think we are actually computing the shape in the graph
before, and passing them to GpuAlloc or GpuEmpty, since the output buffer
is passed as an input. I find it strange that this is not the Op that fails.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/5284#issuecomment-263636814, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-6Pa_ltTLMuIFU4V9JBoNEjsYzL_ks5rDF9HgaJpZM4K8fn_
.

[gvtulder]

@lamblin: You're right, the shape is computed for cuDNN (just not in the C code, which is where I looked). You're also correct about the GpuAlloc, that seems to be the most common point failure. I got the cuDNN error with a valid convolution with a 5-pixel filter on a 4-pixel input. The output shape in that case was indeed 0. I would say that that's not a valid convolution and should return an error. So that's correct, but it's not the right type of error.

Is there a neat way to check the output shape before it reaches GpuAlloc? A GpuConvSizeCheck Op might be overkill just to generate a nice error message.

[lamblin]

It is always possible to introduce an AssertOp during the optimization, with a custom error message. I don't think it would interfere with the other optimizations that late in the process.
The other option would be to check the size of the output buffer in the C code of DnnConv*, but that would only work for the 0-size ones, not negative-size.
If we want to intercept the negative size before GpuAlloc and override the error message, I would say AssertOp would be the way to go.

[nouiz]

This was fixed. Now we have strick shape check before the computation in all convolution ops.