Pytorch-lightning: Metric names when using multiple dataloader for validation

[kantholtz]

Hi guys,

I am using multiple dataloaders for validation. This works great so far, but I have some questions regarding the logged metrics:

As far as I understand, lightning will automatically assign a metric suffix (/dataloader_idx_X). Is there any way for me to control that behaviour? Wandb assumes groups with that slash, and it means, that it groups by metric name and not validation set. However, there are scenarios where it makes much more sense to group by validation set, as they might look at completely different metrics. On the bottom, there is a screenshot on how it looks without any further wanbd configuration.

Apart from my question, this is more of a feature proposal: Can't we supply a (ordered) dictionary instead of a list via val_dataloader() in DataModules? So that it looks like this:

DataModule:

def val_dataloader(self) -> Union[Dict[str, DataLoader], List[Dataloader], DataLoader]:
   return {
      'scenario1': DataLoader('...'),
      'scenario2': DataLoader('...'),
  }

LightningModule

def validation_step(self, batch, batch_idx: int, loader: Union[str, int]):
   # do something
   self.log('my_metric', x)

Which then logs scenarioX/my_metric in case of the example or dataloader_idx_X/my_metric when a list is provided.

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[tchaton]

Hey @kantholtz,

Thanks for sharing your ideas ! It shouldn't be too hard to add.
Let me see if I have some time for it this week.

Best regards,
T.C

[kantholtz]

If not, no stress: I wouldn't mind doing it myself and open up a PR :) But unfortunately I will only have time to do that from mid December on.