Keras: Unknown metric error when loading a model trained with stateful metrics

[lucagrementieri]

Hi everyone,

I'm trying to load a saved model which has been trained with a custom stateful metric (object of the class ValidAccuracy using the code

dependencies = {
     'ValidAccuracy': ValidAccuracy
}
model = keras.models.load_model(model_file, custom_objects=dependencies)

and I get the error
ValueError: Unknown metric function: {'config': {'batch_size': 32, 'name': 'valid_accuracy', 'trainable': False}, 'class_name': <class 'spearec.run.metrics.ValidAccuracy'>}.

Is there any solution or workaround?

[brge17]

dependencies = {
    'valid_accuracy': ValidAccuracy
}

should solve the issue.

Stateful metric inherits from layer and forces it to lower_case_underscore. As per the error message, it is looking for valid_accuracy and not ValidAccuracy.

[sx5640]

I'm running into the same problem, and it seems @brge17 's solution was not the fix. Here is my error message: {'class_name': 'BinaryRecall', 'config': {'name': <metric.BinaryRecall object at 0x000002674E7EE860>, 'trainable': True}}

And my implementation:

global_recall = metric.BinaryRecall()
global_kappa = metric.BinaryKappa()

self.model = keras.models.load_model(
    model_path,
    custom_objects={
        "batch_recall": metric.batch_recall,
        "global_recall": global_recall,
        "global_kappa": global_kappa
    }
)

I also tried to use metric class instead of the instance, and it also failed with similar error message. Is there something I'm missing or simply the code has been changed around Stateful metrics? I'm running 2.1.6

[saetlan]

I'm facing the exact same issue, does anyone have a solution ?

ValueError: Unknown metric function: {'class_name': <lib.metrics.custom_metrics.streaming_iou object at 0x7fd78b1ff5c0>, 'config': {'name': <lib.metrics.custom_metrics.streaming_iou object at 0x7fd78b1ff5c0>, 'trainable': True}}

The loading part :

custom_stream_iou = streaming_iou(num_classes=2, name="streaming_iou")
net = load_model(model_name, custom_objects={'loss_w': loss_w,  'streaming_iou': custom_stream_iou })

The metric implementation :

class streaming_iou(keras.layers.Layer):

"""Stateful Metric to count the total true positives over all batches.
Assumes predictions and targets of shape `(samples, 1)`.
# Arguments
    name: String, name for the metric.
"""

def __init__(self, num_classes, name=None, cl=None, **kwargs):
    super(streaming_iou, self).__init__(name=name, **kwargs)

    if(self.name==None):

        if(self.cl==None):
            name='streaming_iou_mean'
        else:
            name='streaming_iou_'+str(self.cl)

    self.cl=cl
    self.num_classes=num_classes
    self.stateful = True

    #Initialize confusion matrix
    self.confusion = K.variable(value=np.zeros((self.num_classes,self.num_classes)), dtype=tf.float64)

def reset_states(self):
    K.set_value(self.confusion, np.zeros((self.num_classes,self.num_classes)))

def __call__(self, y_true, y_pred):
    """Computes the iou mean or for a specific class
    # Arguments
        y_true: Tensor, batch_wise labels
        y_pred: Tensor, batch_wise predictions
    # Returns
        The total number of true positives seen this epoch at the
            completion of the batch.
    """

    """Compute the per-batch mean intersection-over-union."""

    # Check if shape is compatible.
    y_pred.get_shape().assert_is_compatible_with(y_true.get_shape())  

    y_true = K.argmax(y_true,axis=-1)
    y_pred = K.argmax(y_pred,axis=-1)

    y_true = tf.reshape(tensor=y_true, shape=(tf.shape(y_true)[0]*tf.shape(y_true)[1]*tf.shape(y_true)[2],))
    y_pred = tf.reshape(tensor=y_pred, shape=(tf.shape(y_pred)[0]*tf.shape(y_pred)[1]*tf.shape(y_pred)[2],))

    # Compute the confusion matrix.
    cm = tf.confusion_matrix(y_true,
                         y_pred,
                         self.num_classes,
                            dtype=tf.float64)

    updates = [
        K.update_add(
            self.confusion, 
            cm)]
    self.add_update(updates)

    # Compute the mean intersection-over-union via the confusion matrix.
    sum_over_row = tf.to_float(tf.reduce_sum(self.confusion, 0))
    sum_over_col = tf.to_float(tf.reduce_sum(self.confusion, 1))
    cm_diag = tf.to_float(tf.diag_part(self.confusion))
    denominator = sum_over_row + sum_over_col - cm_diag

    # The mean is only computed over classes that appear in the
    # label or prediction tensor. If the denominator is 0, we need to
    # ignore the class.
    num_valid_entries = tf.reduce_sum(
      tf.cast(
        tf.not_equal(denominator, 0), dtype=tf.float32))

    # If the value of the denominator is 0, set it to 1 to avoid
    # zero division.
    denominator = tf.where(
        tf.greater(denominator, 0), denominator,
        tf.ones_like(denominator))
    iou = tf.div(cm_diag, denominator)

    #If no class is specified, compute the mean
    if self.cl==None:
        #If the number of valid entries is 0 (no classes) we return 0.
        result = tf.where(
            tf.greater(num_valid_entries, 0),
            tf.reduce_sum(iou, name=self.name) / num_valid_entries, 0)
        return result
    else:
        return iou[self.cl]

[brge17]

Stateful Metrics are custom layers:

If you really want to load a custom model with custom layers: https://github.com/keras-team/keras/issues/4871.

Or you can just do load_weights.

I'm of the opinion that load_model shouldn't even exist. Have a function that defines your network, train your model, save the weights, create the network, load your weights, predict.

You shouldn't need the training metrics, or the optimizer to do predictions.

[StenkinVlad]

Or you can just do load_weights.
...
You shouldn't need the training metrics, or the optimizer to do predictions.

That works for me, nice and elegant solution, thx!

[tall-josh]

load_weights worked for me too :-) thanks.