Keras: Metrics ignored when using `model.add_loss()` (like in VAE example)

[danielegrattarola]

Hello,

I'm working with a network similar to the VAE defined in this example, and I've noticed that when a model is defined like:

>>> model = Model(inputs, outputs)
>>> model.add_loss(custom_loss)
>>> model.compile(optimizer=optimizer, metrics=['acc', custom_metric])
>>> model.fit(x_train)  # The model does not require explicit target, like an autoencoder

the metrics seem to be ignored.
I've tried debugging the problem, but it seems that the metric-defining function itself never gets called.

A solution to this is to define a second model identical to the first, but with a loss passed to compile() and an explicit target passed to evaluate():

>>> # Define two models
>>> model_1 = Model(inputs, outputs)
>>> model_2 = Model(inputs, outputs)
>>> 
>>> # Compile the two models separately
>>> model_1.add_loss(custom_loss)
>>> model_1.compile(optimizer=optimizer)
>>> model_2.compile(optimizer='...', loss='...', metrics=['acc', custom_metric])
>>> 
>>> # Fit one, evaluate the other
>>> model_1.fit(x_train)
>>> metrics = model_2.evaluate(x_test, x_test)[1:]
>>> print(metrics)
[acc_value, custom_value]

but besides being inconvenient, this also means that there is no way of monitoring the metrics during the training of the first model.

I'm on Keras 2.1.4, Tensorflow 1.5.0.
Is there a workaround for this or should I just hack a solution together like the one I tried above?

Thanks,
Daniele

[paulcx]

I have same question and any progress?

[emnajaoua]

I am still facing the same problem. any progress ?
@danielegrattarola I didn't really get your workaround. why did you use two separate models ?

[danielegrattarola]

@emnajaoua model_1 is compiled with the custom loss that you add via add_loss(), model_2 is compiled with any usual loss (e.g. MSE) and the metrics. Since the two models share the same weights, training model_1 and evaluating model_2 will give the desired behavior (just ignore the loss in model_2).

[emnajaoua]

@danielegrattarola Thank you for the clarification. I didn't know previously that by instantiating the models , they will share the same weights.

[tik0]

This issue still exists. However, instead of evaluating two models it would be more straight forward to evaluate the metric in a callback.

Furthermore, I found out that, if compile is called, that the metrics are just skipped if the corresponding loss per output is None in line 443. Removing the conditional fixes the issue!

[May4m]

Hey here's a dirty work around this issue

model.compile('adam')

# add your custom metric
model.metrics_tensors.append(K.mean(metric_tensor))
model.metrics_names.append("my_custom_metric")

Note: add this code after you compiled your model

[sometimescasey]

Wow, ok. A big thanks to @tik0 and @May4m here. I'm working on the same VAE example and wanted to do (what I thought would be) the dead-simple task of tracking the KL portion of the loss and the reconstruction portion of the loss separately during training. Have been banging my head against a wall for hours and had no idea why my metrics were being completely ignored.

@tik0 maybe I'm just misunderstanding but does line 443 still line up with the conditional that was causing the problem? I'm not seeing the connection but I 100% believe that there's a non-sensical conditional check in there somewhere since May4m's workaround works.

FYI for clarity, the "metric_tensor" portion of the code snippet that @May4m provided is where I put my custom metric function, which can take more arguments than just (y_true, y_pred). So I'm doing something like the following:

model.metrics_tensors.append(calc_mse_alone(inputs, outputs, input_dim))
model.metrics_names.append("mse")

model.metrics_tensors.append(calc_kl_alone(z_mean, z_log_var))
model.metrics_names.append("kl")

Thanks for help everyone!

[oarriaga]

@sometimescasey one way to track the KL portion could be by making a custom KL loss function that "hacks" the value of y_true to be always None

Therefore your model should return the reconstruction and the latent_variable :

model = tf.keras.Model(image, [reconstruction, latent_variable], 'VAE')

You can create your custom KL function as:

def CustomKL(y_true=None, y_pred=None):
     return compute_KL(y_pred)

Assuming the layers that output reconstruction and latent_variable are called reconstruction and latent_variable you can then define the loss as a dictionary:

all_losses = {'reconstruction': 'mse', 'latent_variable': CustomKL}

Finally you can pass this to model.compile

model.compile(Adam(), all_losses)

Then model.fit should track all the losses separately