Pytorch-lightning: TensorBoardLogger should be able to add metric names in hparams

[tstumm]

🚀 Feature

TensorBoard allows investigating the effect of hyperparameters in the hparams tab. Unfortunately, the log_hyperparams function in TensorBoardLogger cannot add any information about which of the logged metrics is actually a "metric" which can be used for such a comparison.

Motivation

I would like to use the built-in hparams module of TensorBoard to evaluate my trainings.

Pitch

PyTorch-Lightning should give me the possibility to define the metrics of my model in some way such that any logger is able to derive which metric may be used for hyperparameter validation, as well as other possible characteristics which may be defined for those.

Additional context

The hparams method of a summary takes the following parameters:

def hparams(hparam_dict=None, metric_dict=None):

metric_dict is basically a dictionary mapping metric names to values, whereas the values are omitted in the function itself.

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[awaelchli]

Since this is specific to tensorboard and other loggers handle hparams and metrics differently, it is better to use the SummaryWriter object directly. You can always do that with
self.logger.experiment.add_hparams(hparam_dict, metric_dict) within your LightningModule.

[tstumm]

I think if Lightning offers such a logger mechanism, it should offer an abstraction to enable this functionality. I'd be fine with having a register_metric function in TensorBoardLogger, but I don't want to rely on implementation details of the underlying logging mechanism.

[Borda]

@tstumm that sounds good to me, would you mind to send a PR?
cc: @PyTorchLightning/core-contributors

[Borda]

@tstumm with logger you can access directly to the base TensorBoard so whatever is allowed there you could be able to do also here... May point some example of this Tensofoard functionality/use-case?

[awaelchli]

It was introduced here recently: #1630
Feel free to reopen if issues remain.

[mateuszpieniak]

@awaelchli Is there a plan to automatically log all metrics for hparams tab in TensorBoard? I mean all metrics returned in log key inside methods like validation_step using newly merged TensorBoardLogger().log_hyperparams()?

[awaelchli]

I'm not up to date with the logger features atm. Will reopen to keep track of your suggestion and also because I just saw that there is still a bugfix in the works here: #1647

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.