Keras: Custom Mean Directional Accuracy loss function

[xenmind]

Hi,

I'm trying to convert the python loss function from here: https://stackoverflow.com/questions/46876213/custom-mean-directional-accuracy-loss-function-in-keras

I managed to get it working with hard-coded indices in this example:

require(keras)
K <- backend()

mda = function(y_true, y_pred) {
dir = K$equal(K$sign(y_true[1:8] - y_true[0:7]), K$sign(y_pred[1:8] - y_pred[0:7]))
m = K$mean(K$cast(dir, 'float32'))
return(m)}

sess = tf$Session()
yy = matrix(c(0, 1, 2, 1, 0, 1, 3, 1, 2), c(9,1))
y_hat = matrix(c(0, 1, 0, 1, 0, 1, 2, 3, 1), c(9,1))
print(sess$run(mda(tf$constant(yy), tf$constant(y_hat))))

The problem is I can't figure out how to do the first differencing, e.g. y_true(t) - y_true(t-1), with R Keras' tensor indexing. In python, the differencing is done like this: y_true[1:] - y_true[:-1], e.g. (in fudged R and python code):

dir = K$equal(K$sign(y_true[1:] - y_true[:-1]), K$sign(y_pred[1:] - y_pred[:-1]))

When I do this in R it throws this error: 'unexpected ']' in:...' Indicating to me that the indexing is not recognised. I have attempted several ways to solve this, including calculating the length of y_true and plugging the length variable into the index, but this doesn't seem to work either.

I was hoping someone might please help me convert this function.

Thanks for any help,
Gavin.

[dfalbel]

I would do something like this:

mda = function(y_true, y_pred) {

  shape <- y_true$shape$as_list()[1]

  dir = K$equal(K$sign(y_true[2:shape] - y_true[1:(shape-1)]), K$sign(y_pred[2:shape] - y_pred[1:(shape-1)]))
  m = K$mean(K$cast(dir, 'float32'))
  return(m)}

Note that I'm using the newest version of tensorflow package, that uses 1-based indexing.

[xenmind]

Awesome man, looks good, my intuition about using the tensor length seems to have been correct. I'll test it when I can, I just broke my installation updating tensorflow! Thanks very much.

[xenmind]

Hi, I've tested the function with test numbers and it works as expected. The problem is when I compile it:

mod %>% compile(optimizer='nadam', loss=mda)

I get the following error:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RuntimeError: Evaluation error: NA/NaN argument.

Detailed traceback: 
  File "C:\Users\xen\AppData\Local\conda\conda\envs\R-TENS~1\lib\site-packages\keras\engine\training.py", line 860, in compile
    sample_weight, mask)
  File "C:\Users\xen\AppData\Local\conda\conda\envs\R-TENS~1\lib\site-packages\keras\engine\training.py", line 460, in weighted
    score_array = fn(y_true, y_pred)
  File "C:/Users/xen/Documents/R/win-library/3.4/reticulate/python\rpytools\call.py", line 21, in python_function
    raise RuntimeError(res[kErrorKey])

Am I calling the function wrong? I have a more detailed traceback available if it's helpful.

Thanks,
Gavin.

[xenmind]

@dfalbel

Hi,
I noticed that compiling the following:

mda = function(y_true, y_pred) {
    shape <- y_true$shape$as_list()[1]
}

... yields the error:
AttributeError: 'list' object has no attribute 'get_shape'

Could this be the problem? Should the shape method not be called from K? There are some candidates, e.g. ndim, shape, int_shape, etc. Something like: K$ndim(y_true)[1]? I have tried various substitutes, but I get other errors.

Gavin.

[xenmind]

Based on the solution here: https://github.com/rstudio/keras/issues/59 I have create a hybrid R/Backend function:

require(rugarch)
require(quantmod)

mda <- function( y_true, y_pred ) {
    # convert tensors to R objects
    K <- backend()
    y_true <- K$eval(y_true)
    y_pred <- K$eval(y_pred)

    # calculate the metric
    metric = DACTest(na.omit(Delt(y_pred)), na.omit(Delt(y_true)))$DirAcc

    # convert to tensor
    K$constant(metric)
}

It uses R functions Delt and DACTest from the required packages. It yields the correct result with dummy data, but the following error when compiled:

RuntimeError: Evaluation error: InvalidArgumentError: You must feed a value for placeholder tensor 'conv9_target_28' with dtype float and shape [?,?,?] [[Node: conv9_target_28 = Placeholder[dtype=DT_FLOAT, shape=[?,?,?], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Perhaps this approach is easier to solve?
G

[dfalbel]

@xenmind can you provide a reproducible example of the aproach that gave an error when compiling?

y_true may have more then 1 one dimension. That's why you are evaluating in NA dim. I would try to change to something like this shape <- y_true$shape$as_list()[[2]], but I need a reproducible example to test.

[xenmind]

I see, I'll get to it soon.
Cheers.

[xenmind]

@dfalbel

Ok man this should reproduce the first error. Let me know if it doesn't work for you, and thanks for your time and effort, I really appreciate it.

[jjallaire]

The problem seems to be that when the custom loss function is called during compile this is the shape of y_true:

Tensor("conv9_target_1:0", shape=(?, ?, ?), dtype=float32)

So as_list() ends up returning NULL for each dimensions.

@dfalbel Is there a way you know of in the Keras backend to perform tensor slicing on dimensions with ? (I know this is currently a limitation of our [ operator, but perhaps direct calls to the backend can work around it?)

[dfalbel]

Sorry, I only found this now!

@xenmind I'm not getting what your loss function does. With batch_size of 1 you won't be able to do this: y_true[2:shape] - y_true[1:(shape-1). I would use the batch_size inside the loss function and using setting the dimension of y_true. Something like this will allow compiling your model, but are you sure you want to do this?

mda = function(y_true, y_pred) {
  y_true$set_shape(as.integer(c(batch_size, 1, 1)))
  y_pred$set_shape(as.integer(c(batch_size, 1, 1)))
  shape <- y_true$shape$as_list()[1]
  dir = K$equal(K$sign(y_true[2:shape,,] - y_true[1:(shape-1),,]), K$sign(y_pred[2:shape,,] - y_pred[1:(shape-1),,]))
  m = K$mean(K$cast(dir, 'float32'))
  return(m)}

[xenmind]

Hi,

I ran the reproducible example with the new function, but I got this error:

Evaluation error: decreasing indexing of Tensors is not currently supported with 0-based indexing.

Anyway on to your wider point, is this what I want to do? Let me explain my motivation. I'm using this network to forecast a time series 1 ahead. I've read that a batch size of 1 is good to use (for 1 ahead forecasting), although apparently I can use a larger batch size and just use the end value of the prediction.

In respect of directional accuracy, it is a classic forecasting metric for how often a forecast picks the t+1 direction accurately; ultimately I want to pick the direction accurately, I don't necessarily need to get the exact number right. I have a feature generator that had good results when I swapped RMSE for directional accuracy. I was hoping this might hold true for optimizing the network. So it was an experiment, but one that may have paid significant dividends. Anyway I'm happy to take your advice on this. And if it requires significant work to fix the above error, I'm also happy to move on without wasting more of your precious time.

Thanks,
Gavin.

[xenmind]

Having thought about it more, I'm now wondering why I don't just recast my model as a classification one. If I care mostly about direction then it makes sense, and in fact there may already be the equivalent of directional accuracy for classification in keras.

[mg64ve]

I have stateful GRU and my tensor is (batch_size,num_features), so I changed the mda function to:

mda = function(y_true, y_pred) {
  y_true$set_shape(as.integer(c(PRED_BATCH, 1)))
  y_pred$set_shape(as.integer(c(PRED_BATCH, 1)))
  shape <- y_true$shape$as_list()[1]
  dir = K$equal(K$sign(y_true[2:shape,] - y_true[1:(shape-1),]), K$sign(y_pred[2:shape,] - y_pred[1:(shape-1),]))
  m = K$mean(K$cast(dir, 'float32'))
  return(m)}

but I am getting the following error:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

Detailed traceback: 
  File "C:\PROGRA~3\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\models.py", line 1002, in fit
    validation_steps=validation_steps)
  File "C:\PROGRA~3\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\engine\training.py", line 1682, in fit
    self._make_train_function()
  File "C:\PROGRA~3\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\engine\training.py", line 992, in _make_train_function
    loss=self.total_loss)
  File "C:\PROGRA~3\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "C:\PROGRA~3\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\optimizers.py", line 445, in get_updates
    grads = self.get_gradients(loss, params)
  File "C:\PROGRA~3\ANACON~1\envs\R-TENS~1\lib\site-pa

do you have any update on this custom loss function? What should I have to change?