Ignite: Documentation for ConfusionMatrix is thin on details, and error messages not informative

[nsinenian]

🐛 Bug description

Confusion matrix requires that y_pred and y are to have different dimensions (batch,num_classes) and (batch,), respectively. Getting this wrong results in the following error message, which makes no sense (batch size is 5 here, and both y_pred and y have shape (5,1):

y_pred must have shape (batch_size, num_categories, ...) and y must have shape of (batch_size, ...), but given torch.Size([5, 1]) vs torch.Size([5, 1])

I believe that the torch.Size([5, 1]) vs torch.Size([5, 1]) message should probably read torch.Size([5, 1]) vs torch.Size([5,]) to illustrate the problem.

Separately, it would be good to mention in the ConfusionMatrix docs that one-hot encoded vectors are required, and that this can be implemented using from ignite.utils.to_onehot (I stumbled upon it after realizing I had to one-hot encode, since most people use BCEWithLogitsLoss() anyway, and hence their model predictions are not one-hot encoded).

Environment

• PyTorch Version (e.g., 1.4): 1.4.0
• Ignite Version (e.g., 0.3.0): 0.4.2
• OS (e.g., Linux): MacOS
• How you installed Ignite (conda, pip, source): pip
• Python version: 3.8
• Any other relevant information:

[vfdev-5]

Thanks for the report @nsinenian !
Context : https://discuss.pytorch.org/t/ignite-confusionmatrix-metric-error/99079

[afzal442]

@vfdev-5 Hi, Could you take me through this code? Any link? Thanks.

[vfdev-5]

@Afzal-Ind please take a look here: https://github.com/pytorch/ignite/blob/master/ignite/metrics/confusion_matrix.py

[vfdev-5]

Separately, it would be good to mention in the ConfusionMatrix docs that one-hot encoded vectors are required, and that this can be implemented using from ignite.utils.to_onehot (I stumbled upon it after realizing I had to one-hot encode, since most people use BCEWithLogitsLoss() anyway, and hence their model predictions are not one-hot encoded).

@nsinenian actually, this is not exactly true. Predictions are required to have shape (batch_size, num_categories, ...) (logits or after softmax), one-hot encoded. And i'd say, this something we directly have as network output for classification/segmentation tasks. As for targets, they should not be one-hot encoded: (batch_size, ...)
https://github.com/pytorch/ignite/blob/dca273fc73cdeefe0bf6858de64c69f23cbffe8b/ignite/metrics/confusion_matrix.py#L17-L19

[nsinenian]

@vfdev-5 The comment was with respect to predictions, not targets. For the predictions, I have (relevant excerpt):
````
__NUM_CLASSES__ = 2

def binary_one_hot_output_transform(output):
""" Ouptuts a one-hot encoded vector with 0 or 1 values """
[y_pred,y] = output
[y_pred,y] = [torch.sigmoid(y_pred).round().long(),y]
return (to_onehot(y_pred,__NUM_CLASSES__), y.long())

valuation_metrics = {
'confusion_matrix' : ConfusionMatrix(num_classes=__NUM_CLASSES__,output_transform=binary_one_hot_output_transform),
}

evaluator = create_supervised_evaluator(model,metrics=valuation_metrics,output_transform=lambda x,y,y_pred: (y_pred,y))
trainer = create_supervised_trainer(model,optimizer,criterion)

My model output (forward function) is:
def forward(self,features):

    y = self.fc1(features)
    y = self.fcd(y)
    y = self.fc2(F.relu(y))

    return y

````

For binary classification, I use the BCEWithLogitsLoss() loss function which does not use one-hot encoded vectors (and hence the model output being what it is above). Perhaps my use case is atypical.

[vfdev-5]

@nsinenian thanks for the details. Yes, that's true that you have binary classification use-case and yes, it definitely makes sense to mention that how to deal with that in the docs !