Keras: Custom metric function and K$eval() error on model compile.

[madpower2000]

I was try code from https://github.com/rstudio/keras/issues/59, with custom metric function, converting tensor to R vector using the K$eval().

metric_custom <- function( y_true, y_pred ) {
    # convert tensors to R objects
    K <- backend()
    y_true <- K$eval(y_true)
    y_pred <- K$eval(y_pred)

    # calculate the metric
    metric <- mean( y_true[ y_pred >= quantile(y_pred, .985) ] ) 

    # convert to tensor
    K$constant(metric)
}

Work fine with TensorFlow, when I run with dummy data, something like this:

sess <- tf$Session()
yy    <- matrix(c(2, -1, 2, 1, 0, 1, -3, 1,  2), c(9,1))
y_hat <- matrix(c(2,  1, 1, 1, 0, 1, -2, 3, -1), c(9,1))
print(sess$run(metric_custom(tf$constant(yy), tf$constant(y_hat))))

But always get error with K$eval(), when trying to compile my model with this custom metric function:

model %>% compile( 
  optimizer = optimizer_rmsprop(),
  loss = metric_custom
)

RuntimeError: Evaluation error: InvalidArgumentError: You must feed a value for placeholder tensor 'dense_38_target_11' with dtype float and shape [?,?]

[jjallaire]

The problem is that you can't actually convert tensors to R objects because many times a placeholder is passed as the tensor's value (and you can't convert a placeholder to R). You need to use keras backend function instead:

https://keras.rstudio.com/articles/backend.html

Perhaps the k_mean() and k_in_top_k() functions could be used?

[madpower2000]

Actually, I want something like this (if sign of predicted value different from true value, I penalize it):

require(keras)
library(tensorflow)
K <- backend()

metric_custom <- function(y_true, y_pred) {
  x <- K$eval(K$cast(K$equal(K$sign(y_true), K$sign(y_pred)), 'float32'))
  K$sum(tf$multiply(K$abs(y_true), K$cast(K$constant(replace(x,  x==0, -1)), 'float64')))
}
sess <- tf$Session()
yy    <- matrix(c(2, -1, 2, 1, 0, 1, -3, 1,  2), c(9,1))
y_hat <- matrix(c(2,  1, 1, 1, 0, 1, -2, 3, -1), c(9,1))

print(sess$run(metric_custom(tf$constant(yy), tf$constant(y_hat))))

[jjallaire]

Note that with more recent versions of keras you don't need the K <- backend() bit (we have e.g. k_cast(), k_constant(). Note also that these R wrappers use 1-based indexing rather than the 0-based indexing that the K$ variation does.

[madpower2000]

It’s OK to mix Keras backend function call k_function() with TensorFlow backend function tf$function() ?

[jjallaire]

Yes, if you are using the keras tensorflow backend you can indeed mix k_function() with tf$

[madpower2000]

One more thing. If I just want switch y_pred and y_true in my custom loss function

metric_mean_pred <- function(y_true, y_pred) {
  k_mean(y_true) 
} 

model compiles OK, but when I start train it, I’ve got error like this:

py_call_impl(callable, dots$args, dots$keywords) : ValueError: Tried to convert 'x' to a tensor and failed. Error: None values not supported.

[jjallaire]

There was a recent change to the tensorflow package which enabled tensor operations on combinations of tensor variables and tensor constants/placeholders. I don't know if that's the issue here but it's worth updating to the latest versions of all packages on CRAN to see.

If that doesn't resolve it then provide me with a complete (and minimal) reproducible example and I'll investigate further.

[madpower2000]

Nope. After updating all packages and reinstall TensorFlow (via keras_install()) error still persist.

library(keras)
keras:::keras_version() # ‘2.1.3’
tensorflow::tf_config() # v1.4.1

data <- matrix(runif(1000*100), nrow = 1000, ncol = 100)
labels <- matrix(round(runif(1000, min = 0, max = 1)), nrow = 1000, ncol = 1)

model <- keras_model_sequential() %>% 
  layer_dense(units = 32, activation = "relu", input_shape = c(100)) %>% 
  layer_dense(units = 1)

custom_metric <- function(y_true, y_pred) {
  # k_cast(k_equal(k_sign(y_true), k_sign(y_pred)), 'float32') # want to calc
  # k_mean(y_pred)  # It's OK
  k_mean(y_true)  # error
}

model %>% compile(
  optimizer = optimizer_rmsprop(),
  loss = custom_metric
)

model %>% fit(data, labels, epochs=10, batch_size=32)

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Tried to convert 'x' to a tensor and failed. Error: None values not supported.**

Detailed traceback: 
  File "/Users/madpower2000/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 965, in fit
    validation_steps=validation_steps)
  File "/Users/madpower2000/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/training.py", line 1646, in fit
    self._make_train_function()
  File "/Users/madpower2000/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/training.py", line 970, in _make_train_function
    loss=self.total_loss)
  File "/Users/madpower2000/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/Users/madpower2000/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/optimizers.py", line 245, in get_updates
    new_a = self.rho * a + (1. - self.rho) * K.square(g)
  File "/Users/madpower2000/.virtualenvs/r-



md5-5134b435dad5e1736bad08b8ea90ff8a



> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: OS X El Capitan 10.11.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] keras_2.1.3

loaded via a namespace (and not attached):
 [1] compiler_3.4.3  magrittr_1.5    R6_2.2.2        tools_3.4.3     whisker_0.3-2   base64enc_0.1-3 yaml_2.1.16    
 [8] Rcpp_0.12.15    reticulate_1.4  tensorflow_1.5  zeallot_0.0.6   jsonlite_1.5    tfruns_1.2   

[jjallaire]

It looks like you are passing your custom metric as the loss parameter rather than via the metrics parameter. Is it a custom metric or a custom loss function?

This code works for me:

model %>% compile(
  optimizer = optimizer_rmsprop(),
  loss = "mse",
  metrics = list(custom = custom_metric)
)

model %>% fit(data, labels, epochs=10, batch_size=32)

[madpower2000]

I’m apologies for ambiguity. Yes, at first, I was want to make made my custom metrics, but later I realized, I need it not as metric, but custom loss function. So actually, I need custom loss function.

[jjallaire]

This looks like a Keras/TF issue related to the fact that the loss function is not differentiable (perhaps because of the return value of 0 from the loss function)

https://github.com/keras-team/keras/issues/4175
https://github.com/keras-team/keras/issues/4920
https://stackoverflow.com/questions/47426764/custom-keras-loss-function-throws-valueerror-none?rq=1
https://stackoverflow.com/questions/43530157/exception-in-tensorflow-function-used-as-keras-custom-loss?rq=1

[madpower2000]

Thanks @jjallaire! Got it! It’s seems writing custom metric functions and custom loss functions is a little bit a tricky topic not only for me and deserve highlight in documentation and chapter in “Deep Learning with R” book too. 😈