Pytorch-lightning: Precision/Recall/Fbeta error out for floating tensor input of preds but OK for long tensor of preds

[junwen-austin]

🐛 Bug

The current implementation of Precision/Recall/Fbeta used _input_format function to format input shapes and types. There appears to be a bug how this function deals with essential the same input of preds but different data types (long vs float)

Please reproduce using the BoringModel and post here

To Reproduce

Use the following notebook to replicate:

https://colab.research.google.com/drive/1GOzy9urgRmAud-Sadtva3c_GwO1nk5BD?usp=sharing

Expected behavior

The same output should be expected regardless of the input data types (long vs float) for preds

Environment

• CUDA:
  
  
  
  • GPU:
  
  
  • available: False
  
  
  • version: 10.1
  
  
• Packages:
  
  
  
  • numpy: 1.18.5
  
  
  • pyTorch_debug: True
  
  
  • pyTorch_version: 1.7.0+cu101
  
  
  • pytorch-lightning: 1.0.7
  
  
  • tqdm: 4.41.1
  
  
• System:
  
  
  
  • OS: Linux
  
  
  • architecture:
    
    
    
    
    
    • 64bit
    
    
    -
    
    
  
  
  • processor: x86_64
  
  
  • python: 3.6.9
  
  
  • version: #1 SMP Thu Jul 23 08:00:38 PDT 2020
  
  

[awaelchli]

@teddykoker do we need to convert to onehot vectors? it seems the reshape operation at the end requires that.

https://github.com/PyTorchLightning/pytorch-lightning/blob/09c2020a9325850bc159d2053b30c0bb627e5bbb/pytorch_lightning/metrics/classification/precision_recall.py#L42

[rohitgr7]

preds = torch.tensor([1.0, 1.0, 1.0])

I think the reason is if preds is float then it is taken to be as multi-label case, so both preds and targets should be of shape [Batch size, ..., C] in that case.

[SkafteNicki]

As @rohitgr7 says it is because we assume the problem is multi-label when preds and target are the same shape and preds is float. It is mentioned in the docs, but probably not clear enough:
https://github.com/PyTorchLightning/pytorch-lightning/blob/09c2020a9325850bc159d2053b30c0bb627e5bbb/pytorch_lightning/metrics/classification/precision_recall.py#L67-L68

[awaelchli]

ok, I understand. The docs are clear, the first thing I tried was comparing to sklearn that's why I was confused.

[SkafteNicki]

After thinking about it, I don´t think we should support this.
The case where preds and target are the same shape and preds are float in sklearn, is when you want to evaluate a decision function. As we do not really work with the concept of decision functions in lightning, I think it is up to the user to explicitly convert in this case.

[awaelchli]

Yes I noticed that too that the interpretation is different here.
So no further actions required? Let's ask rest of metric team @teddykoker @ananyahjha93 @justusschock before we close :)

[justusschock]

I would also opt for not supporting this!