Keras: FCN for image segmentation

[GPistre]

I am trying to implement pixel-wise image segmentation, but am having trouble figuring out to shape the network to get the right output.

I have X = n images of 60 * 80
and y = n binary matrices of the same size

In the simplest case, I would have:

model = Sequential()
model.add(Convolution2D(32, 3, 3, border_mode='same', input_shape=(1, X.shape[2], X.shape[3])))
model.add(Activation('sigmoid'))
sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=sgd)
model.fit(X, y)

Which results in this:
Exception: A target array with shape (n, 60, 80) was passed for an output of shape (None, 32, 60, 80) while using as loss binary_crossentropy. This loss expects targets to have the same shape as the output.

And I cannot figure out which layers to add to get the right output.

[GPistre]

My confusion was due a misuse of reshape... The network now compiles, but is far from being performant.

It would be great to have examples of image segmentation in Keras. I've had trouble finding any code examples, in Keras or others, and "raw" papers are a bit above my level.

[joelthchao]

I am also interested in such FCN implement. Your model has a serious problem, that is, conv layer produces 32 channels feature map, while your y has only 1 channel. Then you must face the shape mismatch problem.

[GPistre]

My model so far is this. My problem was indeed to merge the channels of the convolutional layers to match the shape of the output matrix. What I did compiles, but I am not sure it makes sense.

X.shape = (nSample, 1, 60, 80) # 60 * 80 black and white image
y.shape = (nSample, 60, 80) # 60 * 80 binary matrix

model = Sequential()
model.add(Convolution2D(32, 3, 3, border_mode='same', input_shape=(1, X.shape[2], X.shape[3])))
model.add(Activation('relu'))

model.add(Convolution2D(32, 3, 3, border_mode='same'))
model.add(Activation('relu'))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))

model.add(Convolution2D(1, 1, 1, border_mode='same'))
# This outputs (nSample, 1, 60, 80), so in effects it seems to merge the channels of the previous layers, but does it make sense?

model.add(Reshape((60, 80)))
model.add(Activation('sigmoid'))

sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=sgd)

model.fit(X, y)

[joelthchao]

You can read FCN for Semantic Segmentation and its code, it's a famous work from Berkeley. Reshape has no problem. However, you can also reshape your y into (nSample, 1, 60, 80) to prevent reshape.

[pengpaiSH]

@GPistre Did you finally finish the FCN for segmentation with Keras? Could you share your implementation?

[gravity1989]

Yes, can someone please share the code?

[Irtza]

@GPistre did you complete the semantic segmentation. Can you share the code?

[GPistre]

I did some experiments, but nothing very conclusive yet. I'm in the process of a long move, and my workstation is in boxes in the middle of the indian ocean. I'll resume my experiments once I'm settled and post it if it works.

[tonyabracadabra]

@GPistre What did you use for your final layer with pixel labeling?

[kevz93]

Can softmax be applied and then get one hot arrays for each pixel ?

[mzaradzki]

Hi @GPistre, @gravity1989, @pengpaiSH,

In case you are still interested in FCN code using Keras I shared a post on Medium (and a corresponding Git repo about it) explaining it :

https://medium.com/@m.zaradzki/image-segmentation-with-neural-net-d5094d571b1e

https://github.com/mzaradzki/neuralnets/tree/master/vgg_segmentation_keras

[CreatCodeBuild]

@mzaradzki what is the loss function you used? I browsed through your code but didn't find any

[mzaradzki]

@CreatCodeBuild very good question : in fact in the script I load the weights that were already trained !

The source for the pre-trained weights is :
http://www.vlfeat.org/matconvnet/pretrained/#semantic-segmentation

How to read the weights file into Keras is shown in the Notebook

[CreatCodeBuild]

@mzaradzki Thank you. I am trying to figure out how to train a FCN. The high level idea of FCN is very appearing. But I can hardly find enough tutorials. The paper FCN for Semantic Segmentation doesn't discuss details very much.

[ahundt]

See the keras-contrib repository and the keras-fcn repository they have examples and training code. I'm on my phone otherwise I'd post a link.