Keras: Keras Error when input Dense layer with a reshaped layer

[dzhao123]

I am trying to implement a lstm model which process 100 texts and then concatenate them together to feed into a dense layer. However there comes an error, which is:

TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

The code shows below:

from keras.layers import Embedding, Dense, LSTM, Masking, concatenate, Flatten, Activation, RepeatVector, Permute, Multiply, Lambda, TimeDistributed, Reshape
from keras import Input, Model
from keras import backend as K


lstm = LSTM(100)
embd = Embedding(1000, 128)
dense = Dense(1, activation='tanh')#(lstm_outputs) 


inputs = []
outputs = []


for _ in range(2):
    encoder_inputs = Input(shape=(160,))
    print(encoder_inputs.shape)
    masked_outputs = Masking(0)(encoder_inputs)
    encoder_embd = embd(masked_outputs)
    print(encoder_embd.shape)
    lstm_outputs = lstm(encoder_embd)
    attention = dense(lstm_outputs)
    attention = Activation('softmax')(attention)
    attention = RepeatVector(100)(attention)
    attention = Permute([2, 1])(attention)

    utterance_representation = Multiply()([lstm_outputs, attention])
    print(utterance_representation.shape)

    utterance_representation = K.squeeze(utterance_representation,axis=1)

    utterance_outputs = K.reshape(utterance_representation, [-1, K.shape(utterance_representation)[0]*K.shape(utterance_representation)[1]])

    encoder_outputs = Dense(200, activation='relu')(utterance_outputs)
    model_outputs = Dense(1, activation='softmax')(encoder_outputs)

    inputs.append(encoder_inputs)
    outputs.append(model_outputs)


encoder_outputs = Dense(200, activation='relu')(outputs_one)
model_outputs = Dense(1, activation='softmax')(encoder_outputs)

model = Model(input=inputs, output=outputs)
model.compile(optimizer='rmsprop', loss='mse', metrics=['accuracy'])


model.fit_generator(feed_generator('data.h5'), steps_per_epoch=250, epochs=1, verbose=1, callbacks=[EarlyStopping(monitor='val_loss', patience=4)])

Can anyone gives some help, thanks!

[maxpumperla]

@dzhao123 after you squeeze utterance_representation has shape (?, 100), which makes K.shape(utterance_representation)[0]*K.shape(utterance_representation)[1] evaluate to None, which in turn makes fan_in of your Dense layer None, which breaks weight init. The error that ultimately causes this is

scale /= max(1., float(fan_in + fan_out) / 2)

since you can't add ints and None.

The flexibility Reshape gives you comes at a cost: you need to carefully evaluate if the shapes you provide make sense. Replace your line above with

    utterance_outputs = K.reshape(utterance_representation, [-1, 100])

and watch it succeed. Note that I have no clue what outputs_one is though.