Pytorch-lightning: [Metrics] AUC reorder is unstable

Created on 19 Oct 2020 · 6Comments · Source: PyTorchLightning/pytorch-lightning

🐛 Bug

When auc(x, y, reorder=True) is called (True is the default) it reorders the xs and ys. It uses torch.argsort on x internally, which is unstable and does not take the second key y into account when sorting. In some cases this leads to reordering that changes the order of ys in an undesirable way resulting in incorrect metric computation.

fpr = torch.tensor([0.4167, 0.5000, 0.5000,
        0.5000, 0.5833, 0.6667, 0.6667, 0.6667, 0.7500, 0.7500, 0.7500, 0.8333,
        0.9167, 0.9167, 0.9167, 1.0000])
tpr = torch.tensor([0.1111, 0.1111, 0.2222,
        0.3333, 0.3333, 0.3333, 0.4444, 0.5556, 0.5556, 0.6667, 0.7778, 0.7778,
        0.7778, 0.8889, 1.0000, 1.0000])

auc_sk = torch.tensor(sk_auc(fpr, tpr)).float()
auc_reorder = auc(fpr, tpr, reorder=True)
auc_no_reorder = auc(fpr, tpr, reorder=False)

print("Sklearn", auc_sk.item())
print("PTL no reorder", auc_no_reorder.item(), "SK==PTL:", torch.allclose(auc_sk, auc_no_reorder))
print("PTL with reorder", auc_reorder.item(), "SK==PTL:", torch.allclose(auc_sk, auc_reorder))
print("unstable argsort: ", torch.argsort(fpr))

>>> Sklearn 0.324056476354599
>>> PTL no reorder 0.324056476354599 SK==PTL: True
>>> PTL with reorder 0.3240675926208496 SK==PTL: False
>>> unstable argsort:  tensor([ 0,  1,  2,  3,  4,  7,  6,  5,  8,  9, 10, 11, 12, 13, 14, 15])

See an example colab. I'm sure there is a more minimal example possible, this is what I encountered when implementing a different issue.