Keras: How to optimize AUC directly？

[lemuriandezapada]

You need a keras(theano/tensorflow) function in order to compute some objective
This gets compiled and run on the gpu(if available)
You can't just use an external function, you'd have to reimplement roc_auc_score yourself

[lukedeo]

Also, the AUC of the ROC is not a differentiable objective...

[fchollet]

The short answer is, you can't. You should optimize a different "proxy" objective, such a crossentropy.

[jerheff]

I've experimented with adding a rank-based loss to crossentropy with mixed results.

Example:
https://gist.github.com/jerheff/8cf06fe1df0695806456

[thouis]

AUC is not differentiable, but it's equivalent to the expected probability that a classifier will correctly rank a random positive and random negative example.

If your classifier outputs probabilities (or something you can treat as probabilities), you could try this as a proxy loss:

1 - E[f(pos)] E[1 - f(neg)]

The first expectation is over positive labels, the second over negative. This can be computed for minibatches, as long as you can ensure there's always a mix of positive and negative labels in training.

NB: I'm not making any arguments about whether this proxy is _good_ for optimization, just that it could be seen as an estimator of the AUC, under the predictions-are-class-probabilities assumption.

[fchollet]

For optimizing AUC indirectly, see this paper by my colleague Elad:
https://arxiv.org/abs/1608.04802

On 19 November 2016 at 09:07, Thouis (Ray) Jones [email protected]
wrote:

AUC is not differentiable, but it's equivalent to the expected probability
that a classifier will correctly rank a random positive and random negative
example.

If your classifier outputs probabilities (or something you can treat as
probabilities), you could try this as a proxy loss:

1 - E[f(pos)] E[1 - f(neg)]

The first expectation is over positive labels, the second over negative.
This can be computed for minibatches, as long as you can ensure there's
always a mix of positive and negative labels in training.

NB: I'm not making any arguments about whether this proxy is _good_ for
optimization, just that it could be seen as an estimator of the AUC, under
the predictions-are-class-probabilities assumption.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/1732#issuecomment-261725828,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArWb5Q2OMoGv7w3nV9lTAPuCoW1D2Eqks5q_yzngaJpZM4HaQf0
.

[josepablog]

@fchollet Now that TF has some support to calculating AUC*, would it be feasible to implement a custom Keras metric (not loss function) that supports this? I've been trying but I get an uninitialized tensor error. Not sure if I'm overlooking something

• https://www.tensorflow.org/api_docs/python/tf/contrib/metrics/streaming_auc

[rspadim]

any news?

[agamemnonc]

Tflearn provides the option of optimising ROC AUC directly using an approximation suggested in this paper.

If this is something people think it would be worth adding, I would be happy to give it a go.

[jiangnanhugo]

you may try pairwise ranking loss. As Minimizing ranking error is equivalent to maximizing AUC (area under ROC curve).

`

[steveyeh987]

@jerheff What is the ranking function you implemented call?

[don-tpanic]

you may try pairwise ranking loss. As Minimizing ranking error is equivalent to maximizing AUC (area under ROC curve).

`
Can you kindly point out the source of this claim? Thanks.

[rahimentezari]

I've experimented with adding a rank-based loss to crossentropy with mixed results.

Example:
https://gist.github.com/jerheff/8cf06fe1df0695806456

I've experimented with adding a rank-based loss to crossentropy with mixed results.

Example:
https://gist.github.com/jerheff/8cf06fe1df0695806456

@jerheff This implementation is based on which paper? This one? https://arxiv.org/abs/1608.04802?

[dsaada]

I use R for tensorflow,
it's possible to optimize on AUC?

[ecm0]

For optimizing AUC indirectly, see this paper by my colleague Elad:
https://arxiv.org/abs/1608.04802

@fchollet Are you aware of a demo/code where this idea is implemented
and that could be re-used or adapted? Thanks!

[ecm0]

I found out where the code associated with https://arxiv.org/abs/1608.04802
is located. It is now here:

https://github.com/tensorflow/models/tree/archive/research/global_objectives

This can is not compatible with TF2. However, it can be made compatible with
a couple of small changes:

*** loss_layers_example_orig.py 2020-08-27 18:01:12.096020097 +0200
--- loss_layers_example.py  2020-08-27 17:35:05.029580911 +0200
***************
*** 25,30 ****
--- 25,33 ----
  import numpy as np
  from sklearn.metrics import precision_score
  import tensorflow as tf
+ tf.compat.v1.disable_eager_execution()
+ 
+ 
  from global_objectives import loss_layers

  # When optimizing using global_objectives, if set to True then the saddle point
***************
*** 176,182 ****
          _, loss_value, step = sess.run([dual_update_op, loss, global_step])

        if use_global_objectives:
!         go_outputs = sess.run(other_outputs.values())

        if step % checkpoint_step == 0:
          precision = precision_at_recall(
--- 179,186 ----
          _, loss_value, step = sess.run([dual_update_op, loss, global_step])

        if use_global_objectives:
!           #go_outputs = sess.run(other_outputs.values())
!           go_outputs = sess.run(list(other_outputs.values()))

        if step % checkpoint_step == 0:
          precision = precision_at_recall(