Keras: Masking the output of a conv layer

[bernardohenz]

Hi,
I am trying to apply a given mask in the output of a conv layer. The simplest form of my problem can be seen in the img

The mask should be considered as an input when training/predicting. I have already tried to use the Merge layer (mode='mul') to apply the input mask as follows:

main_input= Input(shape=(3, 64, 64))
mask1_input = Input(shape=(1, 64, 64))
mask2_input = Input(shape=(1, 64, 64))

conv1 = Convolution2D(1,7,7, border_mode='same')(main_input)
merged_model1 = Sequential()
merged_model1.add(Merge([conv1, mask1_input], mode='mul'))

conv2 = Convolution2D(1, 7,7, border_mode='same')(main_input)
merged_model2 = Sequential()
merged_model2.add(Merge([conv2, mask2_input], mode='mul'))

model = Sequential()
model.add(Merge([merged_model1,merged_model2],mode='sum'))

But it is not working, maybe because I'm trying to merge a layer with a Tensor. But even if I could do that, I don't feel this is the right way to do that. Can someone help?

Please make sure that the boxes below are checked before you submit your issue. Thank you!

• [X] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
• [X] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[dzordzyk]

Is there a reason why you don't write your model using Graph blocks? I tried to rewrite and it seems to be ok.

main_input= Input(shape=(3, 64, 64))
mask1_input = Input(shape=(1, 64, 64))
mask2_input = Input(shape=(1, 64, 64))

conv1 = Convolution2D(1,7,7, border_mode='same')(main_input)
merged_c1 = merge([conv1, mask1_input], mode='mul')

conv2 = Convolution2D(1,7,7, border_mode='same')(main_input)
merged_c2 = merge([conv2, mask2_input], mode='mul')

merged_all = merge([merged_c1, merged_c2], mode='sum')

Then, if you build the model using Model() and check the output shape, it looks fine:

>>> model = Model(input=[main_input, mask1_input, mask2_input], output=merged_all)
>>>
>>> model.output_shape
(None, 1, 64, 64)
>>>

I used functional merge merge() instead of layer Merge(). Functional works on Tensors, Merge() works on Layers.
Also, remember that now you have to pass the list of 3 numpy arrays when fitting/predicting.

[bernardohenz]

Thank you. It seems to work. I'll keep coding.