Keras: Error using new keras (v2.2.0) custom_metric function

[paulafortuna]

When I try to customize a metric using keras in R I got an error then when changing epochs.
Here I send you an example of the same error I am having:

```{r}

example https://keras.rstudio.com/articles/examples/mnist_mlp.html

library(keras)

Data Preparation ---------------------------------------------------

batch_size <- 128
num_classes <- 10
epochs <- 10

The data, shuffled and split between train and test sets

c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))

Transform RGB values into [0,1] range

x_train <- x_train / 255
x_test <- x_test / 255

cat(nrow(x_train), 'train samplesn')
cat(nrow(x_test), 'test samplesn')

Convert class vectors to binary class matrices

y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)

Define Model --------------------------------------------------------------

model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dropout(rate = 0.4) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = 0.3) %>%
layer_dense(units = 10, activation = 'softmax')

summary(model)

custom metric change

metric_mean_pred <- custom_metric("metric_mean_pred", function(y_true, y_pred) {
k_mean(y_pred)
})

model %>% compile(
loss = 'categorical_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c(metric_mean_pred) # custom metric change #################################################
)

Training & Evaluation ----------------------------------------------------

Fit model to data

history <- model %>% fit(
x_train, y_train,
batch_size = batch_size,
epochs = epochs,
verbose = 1,
validation_split = 0.2
)

plot(history)

score <- model %>% evaluate(
x_test, y_test,
verbose = 0
)

Output metrics

cat('Test loss:', score[[1]], 'n')
cat('Test accuracy:', score[[2]], 'n')

The error I get is:

Error in py_call_impl(callable, dots$args, dots$keywords) :
RuntimeError: Evaluation error: arguments imply differing number of rows: 1, 0.
```

Does anyone have a working code with the function custom_metric?
Thank you in advance

[skeydan]

Hi,

thanks for reporting this!

This is not a problem with the custom metric, but a bug in keras training history. You can temporarily circumvent this changing

history <- model %>% fit(
  x_train, y_train,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.2
)

to

model %>% fit(
  x_train, y_train,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.2
)

It will train fine, and you can evaluate the model and make predictions, just not plot the training history (until the bug is fixed).

We'll keep you posted.

[paulafortuna]

Even if I change that the error persists.
Additionally, I can't see the evolution of the metric during the several epochs, and therefore I have no idea about the model performance. I am using this with cross validation and I would like to have access to F1, for avoiding overfitting.

Do you have an idea of how can I access the values of the new metric during this several iterations. even if is not through this way?

Thank you very much for your fast reply!

[jjallaire]

It looks like there was a change in the way Keras handles custom metric functions. This should be fixed on master here: https://github.com/rstudio/keras/commit/726c422cd25b0600db3553fc90f7000eff57a9b6

[skeydan]

Thanks @jjallaire , works great!
@paulafortuna please reinstall from github: devtools::install_github("rstudio/keras")

To correct what I wrote above, the root cause of the error was not in print.keras_training_history (that was just a side effect) but a change in the underlying Python implementation.

[paulafortuna]

Thank you @jjallaire and @skeydan for your fast reply.
The change solved the problem for the provided example :)
However in my bigger project I am also saving and loading the model and in that point I have found a new error that seems to me related. I provide you here a new example:

```{r}

library(keras)

Data Preparation ---------------------------------------------------

batch_size <- 128
num_classes <- 10
epochs <- 10

The data, shuffled and split between train and test sets

c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))

Transform RGB values into [0,1] range

x_train <- x_train / 255
x_test <- x_test / 255

cat(nrow(x_train), 'train samplesn')
cat(nrow(x_test), 'test samplesn')

Convert class vectors to binary class matrices

y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)

Define Model --------------------------------------------------------------

model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dropout(rate = 0.4) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = 0.3) %>%
layer_dense(units = 10, activation = 'softmax')

summary(model)

custom <- custom_metric("custom", function(y_true, y_pred) {
k_mean(y_pred)
})

model %>% compile(
loss = 'categorical_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c(custom)
)

Training & Evaluation ----------------------------------------------------

Fit model to data

history <- model %>% fit(
x_train, y_train,
batch_size = batch_size,
epochs = epochs,
verbose = 1,
validation_split = 0.2
)

plot(history)

score <- model %>% evaluate(
x_test, y_test,
verbose = 0
)

Output metrics

cat('Test loss:', score[[1]], 'n')
cat('Test custom:', score[[2]], 'n')

filename <- paste0("model", ".h5")
save_model_hdf5(model, filename, overwrite = TRUE,include_optimizer = TRUE)
remove(model)

model <- load_model_hdf5(filename) ###################### instruction leading to error

test_score <- model %>% predict_classes(x_test)

And the respective error with traceback:

```{r}

 Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Unknown metric function:custom 
11.
stop(structure(list(message = "ValueError: Unknown metric function:custom", 
    call = py_call_impl(callable, dots$args, dots$keywords), 
    cppstack = structure(list(file = "", line = -1L, stack = c("1   reticulate.so                       0x000000010a12a434 _ZN4Rcpp9exceptionC2EPKcb + 276", 
    "2   reticulate.so                       0x000000010a12a270 _ZN4Rcpp4stopERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE + 48",  ... 
10.
deserialize_keras_object at generic_utils.py#164
9.
deserialize at metrics.py#67
8.
get at metrics.py#75
7.
handle_metrics at training.py#901
6.
compile at training.py#934
5.
compile at models.py#824
4.
load_model at models.py#274
3.
(structure(function (...) 
{
    dots <- py_resolve_dots(list(...))
    result <- py_call_impl(callable, dots$args, dots$keywords) ... 
2.
do.call(keras$models$load_model, args) 
1.
load_model_hdf5(filename) 

I also don't understand why is it necessary the name of this metric in the moment of loading the model.
A way that seems to me that skips this error is by choosing a name of the new metric that is contained in the set of default metrics. But I don't know if this has implications.

Thank you again!

[skeydan]

Hi,

for loading a model with a custom metric, you need to specify the metric in the call:

model <- load_model_hdf5(filename, c("custom" = custom))

Please also see the documentation

https://tensorflow.rstudio.com/keras/reference/metric_binary_accuracy.html#custom-metrics

[paulafortuna]

Perfect. It is solved now! Here I leave the complete working example:

```{r}

example https://keras.rstudio.com/articles/examples/mnist_mlp.html

library(keras)

Data Preparation ---------------------------------------------------

batch_size <- 128
num_classes <- 10
epochs <- 10

The data, shuffled and split between train and test sets

c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))

Transform RGB values into [0,1] range

x_train <- x_train / 255
x_test <- x_test / 255

cat(nrow(x_train), 'train samplesn')
cat(nrow(x_test), 'test samplesn')

Convert class vectors to binary class matrices

y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)

Define Model --------------------------------------------------------------

model <- keras_model_sequential()
model %>%
layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>%
layer_dropout(rate = 0.4) %>%
layer_dense(units = 128, activation = 'relu') %>%
layer_dropout(rate = 0.3) %>%
layer_dense(units = 10, activation = 'softmax')

summary(model)

custom <- custom_metric("custom", function(y_true, y_pred) {
k_mean(y_pred)
})

model %>% compile(
loss = 'categorical_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c(custom)
)

Training & Evaluation ----------------------------------------------------

Fit model to data

history <- model %>% fit(
x_train, y_train,
batch_size = batch_size,
epochs = epochs,
verbose = 1,
validation_split = 0.2
)

plot(history)

score <- model %>% evaluate(
x_test, y_test,
verbose = 0
)

Output metrics

cat('Test loss:', score[[1]], 'n')
cat('Test custom:', score[[2]], 'n')

filename <- paste0("model", ".h5")
save_model_hdf5(model, filename, overwrite = TRUE,include_optimizer = TRUE)
remove(model)

model <- load_model_hdf5(filename, c("custom" = custom)) ###################### instruction leading to error

test_score <- model %>% predict_classes(x_test)
```

[paulafortuna]

Also in case someone is interested in the code for the F1 metric:

{r} F1 <- custom_metric("custom", function(y_true, y_pred) { y_correct <- y_true * y_pred sum_true <- k_sum(y_true, axis=1) sum_pred <- k_sum(y_pred, axis=1) sum_correct <- k_sum(y_correct, axis=1) precision <- sum_correct / sum_pred recall <- sum_correct / sum_true F1 <- 2*(precision*recall)/(precision + recall) return(k_identity(F1)) })