Keras: 5d Input array shape for ConvLSTM2D layer

[vteodorescu]

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [x ] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [ x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [x ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [ x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Hello

I am trying to use the ConvLSTM2D layer, tf backend - channels last.
I have read the reference doc (present only in tensorflow and not in keras docs, for some reason): https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/keras/layers/ConvLSTM2D

So, I have in my code:

model.add(ConvLSTM2D(filters=32, kernel_size=(3, 3),
activation='tanh',
return_sequences=True,
padding='same',
input_shape=InpShape,
name='FirstCLSTMv2'))

InpShape = (None, 15, 128, 1). (2d data of 15 rows by 128 columns, channels = 1 (only one channel))

My understanding of this is: (input samples(or batchsize, or num samples), dim1, dim2, channels)

full model definition:

# define the model
model = Sequential()
model.add(ConvLSTM2D(filters=32, kernel_size=(3, 3),
                     activation='tanh',
                     return_sequences=True,
                     padding='same',
                     input_shape=InpShape,
                     name='FirstCLSTMv2'))
model.add(BatchNormalization())
model.add(ConvLSTM2D(64, (3, 3), activation='tanh', name='secondLSTM'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(outVectorLen, activation='softmax'))

print(model.summary())

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

******

All good so far.

Then I fit the model with data of shape (52, 15, 128, 1). 52 "images" of size 15x128, 1 channel (no colour)
Keras displays the following error message:

Traceback (most recent call last):
File "C:/Users/Teo/OBtrainv1/getDatafitKerasModelv8.py", line 128, in
callbacks=[ffCallBack])
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\models.py", line 867, in fit
initial_epoch=initial_epoch)
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\engine\training.py", line 1522, in fit
batch_size=batch_size)
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\engine\training.py", line 1378, in _standardize_user_data
exception_prefix='input')
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\engine\training.py", line 132, in _standardize_input_data
str(array.shape))
ValueError: Error when checking input: expected FirstCLSTMv2_input to have 5 dimensions, but got array with shape (52, 15, 128, 1)

I understand I need another dimension, as per documentation above:

if data_format='channels_last' 5D tensor with shape: (samples,time, rows, cols, channels)

As I have an image per timestep, I am a bit stumped as to how I can format my data so it fits this input requirement.

I tried adding a 5th dimension, with value 1, (1 sample), making the input array of shape (1, 52, 15, 128, 1). Also added an extra dimension to the labels array.

As a result of this, I got a different error message:

Traceback (most recent call last):
File "C:/Users/Teo/OBtrainv1/getDatafitKerasModelv8.py", line 130, in
callbacks=[ffCallBack])
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\models.py", line 867, in fit
initial_epoch=initial_epoch)
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\engine\training.py", line 1522, in fit
batch_size=batch_size)
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\engine\training.py", line 1382, in _standardize_user_data
exception_prefix='target')
File "C:\Users\Teo\Anaconda3\lib\site-packages\keras\engine\training.py", line 132, in _standardize_input_data
str(array.shape))
ValueError: Error when checking target: expected dense_2 to have 2 dimensions, but got array with shape (1, 52, 31)

I thought that the layers figured out the dimensions in between them - and we are only supposed to be careful with the dimensions at the beginning - by the time we get to the last layer it should be all worked out. - am I doing something wrong?

The model printout:

---

Layer (type) Output Shape Param #

FirstCLSTMv2 (ConvLSTM2D) (None, None, 15, 128, 32) 38144

---

batch_normalization_1 (Batch (None, None, 15, 128, 32) 128

---

secondLSTM (ConvLSTM2D) (None, 13, 126, 64) 221440

---

max_pooling2d_1 (MaxPooling2 (None, 6, 63, 64) 0

---

dropout_1 (Dropout) (None, 6, 63, 64) 0

---

flatten_1 (Flatten) (None, 24192) 0

---

dense_1 (Dense) (None, 128) 3096704

---

dropout_2 (Dropout) (None, 128) 0

---

dense_2 (Dense) (None, 31) 3999

Total params: 3,360,415
Trainable params: 3,360,351
Non-trainable params: 64

---

Thank you in advance for any help!

[jokandre]

It depends on the problem you are trying to solve. Your target ( y ) should be changed accordingly.
Try to see if your problem is a many-to-one or many-to-many.

By your model definition seem that you are doing classification ( many-to-one). Which means, many time steps pointing to one class.

By definition you need:
(samples,time, rows, cols, channels)
X: (1, 52, 15, 128, 1) means that you have only one sample that is a sequence of 52 images.

Your target y, should be (1, outVectorLen) . Which is 2D not 3D

[ggsf13]

I have the same question as you, could you tell me whether you solved it?

[HareshKarnan]

having the same issue

[abbassi007]

having same issue please any one can help??