Pytorch-lightning: Access metrics in custom callbacks

[pbmstrk]

❓ Questions and Help

I have found it useful/helpful to sometimes access metrics in custom callbacks. In v0.9.0 this works using something like this:

def training_step(self, batch, batch_idx):
    return {"loss": self._step(batch)}

def validation_step(self, batch, batch_idx):
    return {"val_loss": self._step(batch)}

def training_epoch_end(self, outputs):
    # ...
    return {"interesting_key_train": interesting_value}

def validation_epoch_end(self, outputs):
    # ...
    return {"interesting_key_val": interesting_value}

The setup allows for the values returned in the _epoch_end methods to be accessed via trainer.callback_metrics. As such, a callback could use these values, e.g.

class CustomCallback(Callback):

    def on_validation_end(self, trainer, pl_module):
        metrics = trainer.callback_metrics
        interesting_value = metrics["interesting_key_train"]

When using the current master branch, the above approach is possible for values returned in validation_epoch_end but no longer possible for training_epoch_end as setting a return value in training_epoch_end raises the exception,

MisconfigurationException: training_epoch_end expects a return of None. HINT: remove the return statement in training_epoch_end

Additionally the values stored in trainer.callback_metrics have changed. Using the example above, in v0.9.0, it is {"loss": ..., "interesting_key_train": ..., "interesting_key_val": ...} and on master it is simply {"interesting_key_val": ...}.

What is the intended way to access metrics (in particular from the training loop) in callbacks?

[ananthsub]

When using the current master branch, the above approach is possible for values returned in validation_epoch_end but no longer possible for training_epoch_end as setting a return value in training_epoch_end raises the exception,

Can you use self.log("interesting_key_train", interesting_value)?

Though there does seem to be an issue with accessing metrics on epoch end on master @williamFalcon

[pbmstrk]

self.log works on master, I had tried it earlier today and run into issues but these seem to be resolved now.