Keras: Adding more layers causes 'Exception: The shape of the input to "Flatten" is not fully defined'

[Gandhi-Sagar]

In a task of an Image Classification, following works perfectly:

batch_size = 128
nb_classes = 5
nb_epoch = 48
img_rows, img_cols = 32, 32
nb_filters = 64
nb_pool = 3
nb_conv = 3
X_train, y_train, X_test, y_test = Load_Some_Custom_Data()
X_train = X_train.reshape(1638, 3, img_rows, img_cols)
X_test = X_test.reshape(409, 3, img_rows, img_cols)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255

Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test,nb_classes)

model = Sequential()
model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
            border_mode = 'valid',
            input_shape = (3, img_rows, img_cols)))
model.add(Activation('relu'))

### -- **Problem is reproducible by uncommenting below lines** 
### if I add this layer, it produces an exception
###model.add(MaxPooling2D(pool_size  = (nb_pool, nb_pool)))
###model.add(Dropout(0.40))
###
model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
model.add(Activation('relu'))

model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size  = (nb_pool, nb_pool)))
model.add(Dropout(0.40))

model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size  = (nb_pool, nb_pool)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(256))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

model.compile(      loss = 'categorical_crossentropy', 
                        optimizer = 'adadelta',
                        metrics = ['accuracy'])

model.fit(X_train, Y_train, batch_size = batch_size, nb_epoch = nb_epoch, verbose = 1, validation_data = (X_test, Y_test), shuffle = True)
score = model.evaluate(X_test, Y_test, verbose = 0)
print('Test score: ', score[0])
print('Test accuracy: ', score[1])

This works perfectly, and gives me a certain accuracy. I am not satisfied with this accuracy, so I want to add more layers. But If I add any single layer further to this model, I get an exception:
Exception: The shape of the input to "Flatten" is not fully defined (got (64, 0, 0). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

I tried changing batch_input_shape = (3, img_rows, img_cols)) to batch_input_shape = (batch_size, 3, img_rows, img_cols)) in the first layer, but no resolution.

I also read somewhere that batch_size has to be factor of number of training_samples, I did that as well, but still the error.

Can you help in this regard, by either helping me eliminate the error or pointing out WHY can't I add more layers? Your help is already appreciated.

[Gandhi-Sagar]

Further, If I change the size of max-pooling window, i.e. nb_pool to 5, then also it produces the exact same exception.

BTW, I also upgraded my keras version, using 1.0.3, and issue persists

[joelthchao]

Your height, width are not large enough for this network. It will become zero before Flatten layer. Please make good use of model.summary() to see how your network looks like.

convolution2d_1 (Convolution2D)    (None, 64, 30, 30)  1792        convolution2d_input_1[0][0]
activation_1 (Activation)          (None, 64, 30, 30)  0           convolution2d_1[0][0]
maxpooling2d_1 (MaxPooling2D)      (None, 64, 10, 10)  0           activation_1[0][0]
dropout_1 (Dropout)                (None, 64, 10, 10)  0           maxpooling2d_1[0][0]
convolution2d_2 (Convolution2D)    (None, 64, 8, 8)    36928       dropout_1[0][0]
activation_2 (Activation)          (None, 64, 8, 8)    0           convolution2d_2[0][0]
convolution2d_3 (Convolution2D)    (None, 64, 6, 6)    36928       activation_2[0][0]
activation_3 (Activation)          (None, 64, 6, 6)    0           convolution2d_3[0][0]
maxpooling2d_2 (MaxPooling2D)      (None, 64, 2, 2)    0           activation_3[0][0]
dropout_2 (Dropout)                (None, 64, 2, 2)    0           maxpooling2d_2[0][0]
convolution2d_4 (Convolution2D)    (None, 64, 0, 0)    36928       dropout_2[0][0]
activation_4 (Activation)          (None, 64, 0, 0)    0           convolution2d_4[0][0]
maxpooling2d_3 (MaxPooling2D)      (None, 64, 0, 0)    0           activation_4[0][0]
dropout_3 (Dropout)                (None, 64, 0, 0)    0           maxpooling2d_3[0][0]