Keras: ValueError: GpuReshape: cannot reshape input of shape (1472, 1000) to shape (1, 1000, 1000)

Created on 9 Dec 2016 · 9Comments · Source: keras-team/keras

https://github.com/baidu-research/ba-dls-deepspeech

I was trying to train the model here for speech recognition. It runs fine when I train it using just a CPU, but gives the above error on GPU.

b6c9a140-bc94-11e6-9553-c36871e347a0

Am I doing something wrong? Or is it a bug?

stale

Source

singlasahil14

👍1

Most helpful comment

@fchollet: I was using keras-1.1.2. I switched to 1.1.0, and now this error is not showing up. This is most probably a bug in keras-1.1.2.

singlasahil14 on 14 Dec 2016

👍2

All 9 comments

1472 * 1000 != 1000 * 1000, therefore reshape fails. The total size should be the same.

keunwoochoi on 10 Dec 2016

Can you post the output of model.summary() for this model?

fchollet on 13 Dec 2016

@fchollet : Here
screen shot 2016-12-14 at 7 23 45 am

singlasahil14 on 14 Dec 2016

@fchollet: I was using keras-1.1.2. I switched to 1.1.0, and now this error is not showing up. This is most probably a bug in keras-1.1.2.

singlasahil14 on 14 Dec 2016

👍2

@singlasahil14:
I had the same problem when I tried to load weights from a different architecture by accident. I found out by loading the complete models and comparing the summaries.

error:
ValueError: GpuReshape: cannot reshape input of shape (128) to shape (1, 1, 1, 96).
Apply node that caused the error: GpuReshape{4}(convolution2d_4_b, TensorConstant{[ 1 1 1 96]})
Toposort index: 41
Inputs types: [CudaNdarrayType(float32, vector), TensorType(int64, vector)]
Inputs shapes: [(128,), (4,)]
Inputs strides: [(1,), (8,)]
Inputs values: ['not shown', array([ 1, 1, 1, 96])]
Outputs clients: [[GpuElemwise{add,no_inplace}(GpuDimShuffle{0,2,3,1}.0, GpuReshape{4}.0)]]

coristus on 1 Feb 2017

I am having a similar problem when I use K80 Tesla GPU. It Please look into this issue. I am posting below the error trace and my model.summary().
Thanks
Traceback (most recent call last):
File "", line 1, in
File "/N/u/achbogga/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 1486, in fit
initial_epoch=initial_epoch)
File "/N/u/achbogga/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 1141, in _fit_loop
outs = f(ins_batch)
File "/N/u/achbogga/anaconda2/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1122, in __call__
return self.function(*inputs)
File "/N/u/achbogga/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 898, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "/N/u/achbogga/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/N/u/achbogga/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", line 884, in __call__
self.fn() if output_subset is None else\
ValueError: GpuReshape: -1 axis found at index 0 in new shape but the total size of the array (6400000) is not divisible by the given shapes (43264).
Apply node that caused the error: GpuReshape{4}(GpuSubtensor{::, ::, int64::, int64::}.0, TensorConstant{[-1 64 26 26]})
Toposort index: 631
Inputs types: [GpuArrayType(float32, (False, False, False, False)), TensorType(int64, vector)]
Inputs shapes: [(160, 64, 25, 25), (4,)]
Inputs strides: [(173056, 2704, 104, 4), (8,)]
Inputs values: ['not shown', array([-1, 64, 26, 26])]
Outputs clients: [[GpuContiguous(GpuReshape{4}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Layer (type) Output Shape Param # Connected to

input_1 (InputLayer) (None, 16, 3, 48, 48) 0

timedistributed_1 (TimeDistribut (None, 16, 64, 24, 24 9472 input_1[0][0]

timedistributed_2 (TimeDistribut (None, 16, 64, 26, 26 0 timedistributed_1[0][0]

timedistributed_3 (TimeDistribut (None, 16, 64, 26, 26 0 timedistributed_2[0][0]

timedistributed_4 (TimeDistribut (None, 16, 64, 12, 12 0 timedistributed_3[0][0]

timedistributed_5 (TimeDistribut (None, 16, 64, 12, 12 0 timedistributed_4[0][0]

timedistributed_6 (TimeDistribut (None, 16, 64, 12, 12 4160 timedistributed_5[0][0]

timedistributed_7 (TimeDistribut (None, 16, 192, 12, 1 110784 timedistributed_6[0][0]

timedistributed_8 (TimeDistribut (None, 16, 192, 12, 1 0 timedistributed_7[0][0]

timedistributed_9 (TimeDistribut (None, 16, 192, 14, 1 0 timedistributed_8[0][0]

timedistributed_10 (TimeDistribu (None, 16, 192, 14, 1 0 timedistributed_9[0][0]

timedistributed_11 (TimeDistribu (None, 16, 192, 6, 6) 0 timedistributed_10[0][0]

timedistributed_13 (TimeDistribu (None, 16, 96, 6, 6) 18528 timedistributed_11[0][0]

timedistributed_15 (TimeDistribu (None, 16, 16, 6, 6) 3088 timedistributed_11[0][0]

timedistributed_17 (TimeDistribu (None, 16, 192, 6, 6) 0 timedistributed_11[0][0]

timedistributed_12 (TimeDistribu (None, 16, 64, 6, 6) 12352 timedistributed_11[0][0]

timedistributed_14 (TimeDistribu (None, 16, 128, 6, 6) 110720 timedistributed_13[0][0]

timedistributed_16 (TimeDistribu (None, 16, 32, 6, 6) 12832 timedistributed_15[0][0]

timedistributed_18 (TimeDistribu (None, 16, 32, 6, 6) 6176 timedistributed_17[0][0]

inception_3a/output (Merge) (None, 16, 256, 6, 6) 0 timedistributed_12[0][0]
timedistributed_14[0][0]
timedistributed_16[0][0]
timedistributed_18[0][0]

timedistributed_20 (TimeDistribu (None, 16, 128, 6, 6) 32896 inception_3a/output[0][0]

timedistributed_22 (TimeDistribu (None, 16, 32, 6, 6) 8224 inception_3a/output[0][0]

timedistributed_24 (TimeDistribu (None, 16, 256, 6, 6) 0 inception_3a/output[0][0]

timedistributed_19 (TimeDistribu (None, 16, 128, 6, 6) 32896 inception_3a/output[0][0]

timedistributed_21 (TimeDistribu (None, 16, 192, 6, 6) 221376 timedistributed_20[0][0]

timedistributed_23 (TimeDistribu (None, 16, 96, 6, 6) 76896 timedistributed_22[0][0]

timedistributed_25 (TimeDistribu (None, 16, 64, 6, 6) 16448 timedistributed_24[0][0]

inception_3b/output (Merge) (None, 16, 480, 6, 6) 0 timedistributed_19[0][0]
timedistributed_21[0][0]
timedistributed_23[0][0]
timedistributed_25[0][0]

timedistributed_26 (TimeDistribu (None, 16, 480, 8, 8) 0 inception_3b/output[0][0]

timedistributed_27 (TimeDistribu (None, 16, 480, 8, 8) 0 timedistributed_26[0][0]

timedistributed_28 (TimeDistribu (None, 16, 480, 3, 3) 0 timedistributed_27[0][0]

timedistributed_30 (TimeDistribu (None, 16, 96, 3, 3) 46176 timedistributed_28[0][0]

timedistributed_32 (TimeDistribu (None, 16, 16, 3, 3) 7696 timedistributed_28[0][0]

timedistributed_34 (TimeDistribu (None, 16, 480, 3, 3) 0 timedistributed_28[0][0]

timedistributed_29 (TimeDistribu (None, 16, 192, 3, 3) 92352 timedistributed_28[0][0]

timedistributed_31 (TimeDistribu (None, 16, 208, 3, 3) 179920 timedistributed_30[0][0]

timedistributed_33 (TimeDistribu (None, 16, 48, 3, 3) 19248 timedistributed_32[0][0]

timedistributed_35 (TimeDistribu (None, 16, 64, 3, 3) 30784 timedistributed_34[0][0]

inception_4a/output (Merge) (None, 16, 512, 3, 3) 0 timedistributed_29[0][0]
timedistributed_31[0][0]
timedistributed_33[0][0]
timedistributed_35[0][0]

timedistributed_36 (TimeDistribu (None, 16, 512, 1, 1) 0 inception_4a/output[0][0]

timedistributed_37 (TimeDistribu (None, 16, 128, 1, 1) 65664 timedistributed_36[0][0]

timedistributed_38 (TimeDistribu (None, 16, 128) 0 timedistributed_37[0][0]

bidirectional_1 (Bidirectional) (None, 16, 512) 788480 timedistributed_38[0][0]

dropout_1 (Dropout) (None, 16, 512) 0 bidirectional_1[0][0]

bidirectional_2 (Bidirectional) (None, 16, 512) 1574912 dropout_1[0][0]

dropout_2 (Dropout) (None, 16, 512) 0 bidirectional_2[0][0]

bidirectional_3 (Bidirectional) (None, 16, 512) 1574912 dropout_2[0][0]

dropout_3 (Dropout) (None, 16, 512) 0 bidirectional_3[0][0]

bidirectional_4 (Bidirectional) (None, 16, 512) 1574912 dropout_3[0][0]

dropout_4 (Dropout) (None, 16, 512) 0 bidirectional_4[0][0]

timedistributed_39 (TimeDistribu (None, 16, 1024) 525312 dropout_4[0][0]

dropout_5 (Dropout) (None, 16, 1024) 0 timedistributed_39[0][0]

loss1/classifier (Dense) (None, 16, 7) 7175 dropout_5[0][0]

activation_1 (Activation) (None, 16, 7) 0 loss1/classifier[0][0]

Total params: 7,164,391
Trainable params: 7,164,391
Non-trainable params: 0

None

achbogga on 6 Apr 2017

👎1 👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

stale[bot] on 6 Jul 2017

I am facing the same issue regarding the array structure. I am using theano=0.8.2 with GPU, cuda=8.0 , keras=1.1.2.

2017-07-27 03:22:55,651 INFO (data_generator) Iters: 326
Traceback (most recent call last):
File "../train.py", line 155, in
args.sortagrad)
File "../train.py", line 133, in main
do_sortagrad=sortagrad)
File "../train.py", line 89, in train
label_lengths, True])
File "/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py", line 811, in __call__
return self.function(*inputs)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in __call__
outputs = self.fn()
ValueError: GpuReshape: cannot reshape input of shape (1776, 1000) to shape (1, 1000, 1000).
Apply node that caused the error: GpuReshape{3}(GpuElemwise{add,no_inplace}.0, MakeVector{dtype='int64'}.0)
Toposort index: 395
Inputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)]
Inputs shapes: [(1776, 1000), (3,)]
Inputs strides: [(1000, 1), (8,)]
Inputs values: ['not shown', array([ -1, 1000, 1000])]
Outputs clients: [[GpuJoin(TensorConstant{2}, GpuReshape{3}.0, GpuReshape{3}.0, GpuReshape{3}.0)]]