Autogluon: [tabular-metrics] add confusion matrix + derived metrics

[tlienart]

As a follow up to #379 the suggestion would be to

• add some code to compute a C x C confusion matrix where C is the number of classes
• add some code to compute metrics on said confusion matrix (e.g. balanced accuracy)

I'm happy to eventually open a PR for this but would welcome suggestions as to what metrics are most relevant beyond the binary case.

Possibly related #381

PS: one thing that is unclear to me is whether it may be worth having implementation of these metrics here or whether we only want to wrap sklearn's metrics, at the moment it seems that the code has a bit of both with a fair bit of sklearn)

[Innixma]

A lot of the current metrics code is based off of auto-sklearn's metrics, which is based off the sklearn metrics.

The benefit of this approach would be that you could compute multiple metrics more efficiently if you re-used the C x C confusion matrix between them correct? I think the metrics we would care about in this instance would be the metrics which are computationally expensive to calculate. Which ones those are is unknown to me.

Note: This is not related to #381

[tlienart]

What I meant is that sklearn already does this, as far as I'm aware when you call balanced_accuracy_score with sklearn it forms the confusion matrix. The only advantage in doing this is to reduce the dependency on sklearn and that we could have more flexible implementations.

I could rephrase as follows, either one of the following two options makes sense IMO but not a mix of the two:

• reduce dependency on sklearn, bring here essential code for metrics we care about, all metrics are computed by this package directly or via mxnet
• compute all metrics via sklearn (possibly apart from the ones that are directly supported by mxnet), only have code here for metrics that are not supported by both sklearn and mxnet

At the moment the situation is a bit of a mix of the two; for instance balanced_accuracy is computed here where one could use scikitlearn to get it.

[Innixma]

I think we should stick with thinking of sklearn as a hard-dependency (even more-so than mxnet). It is so widely used and useful that I don't think we should consider rewriting our own versions if the sklearn ones satisfy our needs.

As to why some of the metrics don't use sklearn, that is hard to tell. Those decisions were made a long time ago and I don't remember exactly what was simply re-used from auto-sklearn, slightly edited, or newly introduced. If we can move to the sklearn version of a metric without losing functionality or performance, I would encourage it (Although you showed we do lose functionality for sklearn multi-class balanced_accuracy in your PR).

It is also possible some of the metrics are only available in later versions of sklearn, whereas AutoGluon supports sklearn as old as 0.20.0 (I think balanced_accuracy was not in this version of sklearn, but I could be wrong). I'd be ok with increasing the minimum required version of sklearn to 0.21 or maybe 0.22 to simplify our metrics.

[tlienart]

that makes sense, but so I suggest closing this for now and maybe leaving the balanced accuracy code as it is for now; if there's later work on metrics it might be good to have another think.