Keras: The loss function can not contain the output of intermediate layers ?

[Imorton-zd]

I need to add a loss term of two intermediate layers. For example:

x1 = Input(shape=(input_size,))
ld = Dense(128, activation='relu')
out1 = Dense(1, activation='sigmoid')

x2 = Input(shape=(input_size,))
rd = Dense(128, activation='relu')
out2 = Dense(1, activation='sigmoid')

def three_loss(x1,x2,ld,rd,out1,out2):
        loss1 = K.sum(objectives.binary_crossentropy(x1, out))
# The gap between `ld` and `rd` is expected as small as possible
        loss2 = K.sum(objectives.binary_crossentropy(ld, rd))
        loss3 = K.sum(objectives.binary_crossentropy(x2, out2))

model = Model(input=[x1,x2], output=[ld2,rd2])
model.compile(optimizer='adadelta',
                  loss = three_loss, metrics=['accuracy'])

However, the error tells me that the three_loss need six input but only given four. How can I solve this problem? To my knowledge, the input and output must be the actual data sets and observed targets, I can not add the rd or ld in them.

[unrealwill]

Hello,

You usually can pass an array of losses in compile.
model.compile( ..., loss = [ loss1, loss3] )
This will work for loss 1 and loss 3 but not for loss2 which is more like a regularization term.
Regularization has recently been refactorized, so the documentation is probably not yet up to date.

But to add a custom loss. You create a custom layer (returning anything as you won't use it) , and inside call you do add_loss()

Running code (not tested for bugs):

import numpy as np

from keras.models import Model
from keras.layers import Input

import keras.backend as K
from keras.engine.topology import Layer
from keras.layers.core import  Dense

from keras import objectives

def zero_loss(y_true, y_pred):
    return K.zeros_like(y_pred)


class CustomRegularization(Layer):
    def __init__(self, **kwargs):
        super(CustomRegularization, self).__init__(**kwargs)

    def call(self ,x ,mask=None):
        ld=x[0]
        rd=x[1]
        bce = objectives.binary_crossentropy(ld, rd)
        loss2 = K.sum(bce)
        self.add_loss(loss2,x)
        #you can output whatever you need, just update output_shape adequately
        #But this is probably useful
        return bce

    def get_output_shape_for(self, input_shape):
        return (input_shape[0][0],1)

input_size= 100
output_dim = 1
x1 = Input(shape=(input_size,))
ld = Dense(128, activation='relu')(x1)
out1 = Dense(output_dim, activation='sigmoid')(ld)

x2 = Input(shape=(input_size,))
rd = Dense(128, activation='relu')(x2)
out2 = Dense(output_dim, activation='sigmoid')(rd)

cr = CustomRegularization()([ld,rd])

m = Model( [x1,x2], [out1,out2,cr])
m.compile( loss=[K.binary_crossentropy,K.binary_crossentropy,zero_loss], optimizer="adam")

nb_examples = 32
print m.predict( [np.random.randn(nb_examples,input_size),np.random.randn(nb_examples,input_size)] )
print m.fit(  [np.random.randn(nb_examples,input_size),np.random.randn(nb_examples,input_size)], [np.random.randn(nb_examples,output_dim),np.random.randn(nb_examples,output_dim), np.random.randn(nb_examples,1) ]  )

[Imorton-zd]

Thanks for your reply.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[xinghuokang]

@unrealwill I have the need for computing loss using the output of intermediate layers, and I use the code you give above; The part train result as follow:
loss: 0.4426 - hks-fc_loss: 0.3693 - custom_regularization_1_loss: 0.0000e+00
Question : the total loss is 0.4426(!=0.3693 why???) and custom_regularization_1_loss: 0.0000e+00 ????
If custom_regularization_1_loss is 0, how is it to adjust weights of every layer??

[raginisharma14]

@unrealwill where is this function "get_output_shape_for " being called? because when I run the above code, I get the dimension mismatch error, as the shape of the output of custom regularization layer is the shape of bce and not the shape of get_output_shape_for function.
Error: Cannot feed value of shape (45, 1) for Tensor u'custom_regularization_1_target:0', which has shape '(?, ?, ?)'

[teaghan]

If I am calculating multiple custom losses throughout the model, using this method, how can I return each loss separately when running m.fit()? For instance, in this example the first loss returned is the total loss, the second and third are the binary cross-entropy losses, and the last returned loss is just zero. Therefore, it is easy to calculate the custom loss, as it is just the remainder of the binary cross-entropy losses subtracted from the total loss. What if I have two separate custom losses?

[hoangcuong2011]

@unrealwill: Thank you for your code. I was wondering about the role of zero_loss in your code. Here, zero_loss function returns a tensor with zero values for all elements.

What does it do here as the loss of the full model exactly? I was very confused about this. If the loss is such a constant, what would happen during training?

Also do you have any reference about this?

[unrealwill]

@hoangcuong2011 Hello,

First of all, this is kind of old thread digging. Not still sure if the code works again or if it has rotten, but in principle (i.e. adding a custom loss the same way we add a regularization term is still sound).
It dates from the time before Keras was "merged" into Tensorflow.

Nowadays, when I use Keras, I write my losses in Tensorflow. So I'm not aware if there is a newer neater way of doing things staying Keras only.

If I remember correctly, the issue was mostly about getting Keras to compute a specific loss function.
Because of the specificity of the loss function, Keras was complaining about shapes, or was not able to feed it the inputs in the appropriate way.

The zero_loss is a trick : it's just a way to ignore the output value (yet not have it simplified by the computation graph), because what matter is the "self.add_loss" inside the CustomRegularization Layer, which taps directly at a lower level into the back-end (theano or tensorflow) to allow us to add any custom term.

Thanks.