Addons: Multiclass AUC

[ychervonyi]

System information

• TensorFlow version (you are using): 1.13
• TensorFlow Addons version: -
• Is it in the tf.contrib (if so, where): No.
• Are you willing to contribute it (yes/no): Yes.
• Are you willing to maintain it going forward? (yes/no): Yes

Describe the feature and the current behavior/state.
Multi-class AUC is a measure to examine the performance of multi-class classifiers. It is a natural extension of the well known and widely used AUC (Area Under the ROC curve) metric, which is defined for only binary classifiers. This measure was introduced in the paper 'Hand, David J., and Robert J. Till. "A simple generalization of the area under the ROC curve for multiple class classification problems." Machine learning 45, no. 2 (2001): 171-186'. It is widely used as the original paper currently has 1435 citation. Multi-class AUC reduces to the standard AUC in case of two classes.

Will this change the current api? How?
No. It will be just an additional function which can be used as a metric.

Who will benefit with this feature?
Machine learning practitioners and researchers, especially those in the medical field, for whom class separability and false positives are very important.

Any Other info.

[seanpmorgan]

Thanks for the issue. Based on citations and general applicability I think this would be a nice addition. Related is #126

[ychervonyi]

Thank you for the quick reply. I have a working implementation and can prepare a PR. By the way moved from here.

[ychervonyi]

@seanpmorgan In my PR should I create a new directory tensorflow_addons/metrics/?

[seanpmorgan]

Yes, it'll require a few different things (added to global __init__ and BUILD) but if you just start a WIP pull request we can advise on the details

[seanpmorgan]

You might want to coordinate with @AakashKumarNain who was planning on starting the subpackage as well.

[ychervonyi]

It seems like it is non-trivial to rewrite my implementation in TF 2.0. I posted several questions on Stackoverflow. Apparently, even standard metrics from tf.metrics don't work in eager mode (see [here])(https://stackoverflow.com/questions/55928569/tf-metrics-x-is-not-supported-when-eager-execution-is-enabled-tensorflow-2-0). I think once at least one metric from tf.metrics will be rewritten in 2.0 I will be able to look at the necessary changes and update my code too.

[facaiy]

@ychervonyi Hi, Yuri. As discussed in https://github.com/tensorflow/addons/issues/126#issuecomment-495103599, I think keras can make sure that tf.metric runs in the graph mode automatically, so we don't have to wait for the time when tf.metrics supports eager mode. What do you think?

[ychervonyi]

@facaiy Hi Yan. Thank you for bringing my attention to your comment. Looks like you are right. I will take a closer look at the Keras metrics early next week and see whether I can port my code. Do you think tf.metrics will be abandoned in favor of Keras metrics?

[facaiy]

Hi, Yuri, tf.metrics is an alias for tf.keras.metrics in tf 2.0.

[facaiy]

Yes, you're right, the old tf.metric will be abandoned totally.

[Squadrick]

Tracking in #265