Keras: How to combine CNN & LSTM for two feature sets?

[hkmztrk]

Hello,

I'm having troubles to understand whether something is possible for such example:

For instance I have two different feature sets (A & B) and I want to use CNN for A and LSTM for B and combine their learned representations before feeding it to FC layer. Is it possible and how do I do it with Keras? Should I use two different fit functions? But then how do I use predict?

model.fit(train_A, trainY)
model.fit(train_B, trainY)

I'm really confused about this architecture, I'd appreciate it a lot if you guide me or even refer some reading material.

Thanks in advance!

[samre12]

@hkmztrk one simple possible solution is to train the two models separately, and then fix their weights when training the FC layer.
Another solution is to use the Functional API in Keras. It allows to create models with multiple inputs, multiple outputs and also allows to define and train multiple losses. So you could have one loss for model A, one loss for model B and one loss for the FC layer and train them all simultaneously. However, this could pose convergence issues for the parameters if not trained properly.

[hkmztrk]

@samre12 thank you for your comment! I actually want to have a single loss function, I am just not that sure about how to combine them before feeding them to FC.
Also, sorry if this is a trivial question but how fixing weights will affect the model?

[samre12]

@hkmztrk sorry for the misinterpretation, I thought you would like to fix the models generating the representation when training the FC layer. However, you could always fine tune them when training the FC layer.
A trivial way of combining representations is to just concatenate them and then pass through multiple FC layers to generate a common representation of the required size.

[hkmztrk]

thank you for your help @samre12! Do you happen to know such example that you can share?

[samre12]

Visual Question Answering (VQA) Models happen to combine the visual and textual representations of the image and question respectively using the same technique. You could have a look there. :)

[hkmztrk]

Ok, thanks a lot @samre12 !

[hkmztrk]

Hello again,

I actually found the model I want to implement, here (https://arxiv.org/abs/1605.07333)
They have two different contexts as inputs which are processed by two independent conv and max-pooling layers. After pooling they concat the results. What I want is similar to this. So if I have the following CNN for instance, how do I do above concat operation:

def baseline_cnn(activation='relu'):

model = Sequential()
model.add(Embedding(SAMPLE_SIZE, EMBEDDING_DIMS, input_length=MAX_SMI_LEN))
model.add(Dropout(0.2))
model.add(Conv1D(NUM_FILTERS, FILTER_LENGTH, padding='valid', activation=activation, strides=1))
model.add(GlobalMaxPooling1D())
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam',  metrics=['accuracy'])

return model

[samre12]

@hkmztrk , just to clarify, are you making two copies of this model and then you want to concat the results of the two models?

[hkmztrk]

@samre12 yes, similar. I have two copies of this model or I have this model and one LSTM or sth. What I want is a model just like in the figure, but I don't understand how can I code this concat step.
( e.g. Should it happen before compile? Do I still need Dense layer? How do fit function will work? )

[samre12]

@hkmztrk what is the loss that you are using to fit the parameters and is it over the sentence representation?

[hkmztrk]

@samre12 I think it's over the sentence and they use sth called ranking loss. This is the exact phrase:

"Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the
middle context; and (2) a combination of the middle context, the right entity and the right context. The two contexts are processed by two independent convolutional and max-pooling layers. After pooling, the results are concatenated to form the sentence representation"

[samre12]

@hkmztrk again I would suggest you using the Keras Functional API. Here you could define a functional model which takes in two inputs (the two contexts that you mention) runs the two models on the two inputs (check if they share weights or not), then concatenate them into a single sentence representation.
You could do something like:

x1 = keras.layers.Input()
x2 = keras.layers.Input()
y1 = base_line_cc(x1) #apply all the convolutional and max pool operations
y2 = base_line_cc(x2) #apply all the convolutional and max pool operations
y = concatenate(y1, y2)
model = Model(inputs=[x1, x2], outputs=[y])
model.compile()
model.fit()

[primeMover2011]

@hkmztrk like @samre12 already said: Just take a look at the Functional API where you can find samples that show how to create models with multiple inputs.

[samre12]

@hkmztrk please close the issue if you think it is resolved. :)

[hkmztrk]

Thanks a lot, I will try that! @samre12 @primeMover2011 Hope it will work!