Pytorch-lightning: How to access validation logs data inside the on_validation_batch_end callback

[DomainFlag]

❓ Questions and Help

What is your question?

How can I have access to validation logs data inside the on_validation_batch_end callback?

For the on_batch_endcallback, I have access to the train metrics by using the class property callback_metricsof the Trainerbut when querying the same property inside the on_validation_batch_endcallback I still have the same train logs data.

I've tried to loop all properties of the Trainer but couldn't find any reference for any validation log data.

Code

Processing batch:

def validation_step(self, image_ray_batch, batch_idx):
    ...
    output = {
        "val_loss": loss,
        "validation/loss": loss,
        "validation/coarse_loss": coarse_loss,
        "validation/psnr": psnr,
    }

    if rgb_fine is not None:
        output["validation/fine_loss"] = fine_loss

    return output

Custom Logging:

def on_validation_batch_end(self, trainer, pl_module):
    print("on_validation_batch_end")
    # Gather trainer metrics
    metrics = trainer.callback_metrics
    print(metrics)

What's your environment?

Using version [0.7.6]

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[ananyahjha93]

@DomainFlag hey, first of all we would prefer that you update to the latest version. I know there are some changes from 0.7.6 in the API, but now it is being kept stable and should remain unchanged for the most part.

We don't currently have this and discussed about making this available within hooks.

[DomainFlag]

@ananyahjha93 Hey, thanks for the reply, I've upgraded to 0.9.0 and currently I am returning like so when overriding the validation_step:

output = {
    "loss": loss,
    "log": {
        "val_loss": loss,
        "validation/loss": loss,
        "validation/coarse_loss": coarse_loss,
        "validation/psnr": psnr
    },
}

return output

The issue is that in on_validation_batch_end by accessing the callback_metrics property of the trainer I can have access to the validation logs but they aren't updated for each individual processed batch, thus I have access only to the last batch results. Is that part related to the "making this available within hook"?

[DomainFlag]

Alright, so with some research done, I've found out the callback_metricsproperty gets updated only with the logs from processing the last val batch, thus the only way to have access to all val batch logs is to override the validation_epoch_end and store the list of batch results as dict property and access it by overriding the on_validation_epoch_end for pytorch_lightning.callbacks Callback. Is not the same as on_validation_batch_end but it's still something.

With some digging, I noticed the trainer has the property evaluation_loop that has subsequently the outputs property but there isn't any way to access it directly from the callback? I mean I am working with time-consuming val batch processing, and having logs for each step will be very helpful...

[EspenHa]

You could implement a custom logger. It should receive the logged data.
Example:

class MyLogger(pytorch_lightning.loggers.LightningLoggerBase):
    def log_metrics(self, metrics, step=None):
        print(metrics)

    ... # implement other methods as well

[DomainFlag]

@EspenHa How would I go about logging for each step, instead of each epoch as it's now?

[EspenHa]

I guess you could do a hack like: self.trainer.logger.log_metrics(...) in validation_step? (Not sure :P)

[ananyahjha93]

@DomainFlag you should checkout the results object API in the new version: https://pytorch-lightning.readthedocs.io/en/stable/results.html

With the Results API, you should have access to both the step wise and epoch wise logs for validation.

Is this all you require? 'having logs for each step will be very helpful', if yes then you don't have to worry about hooks and having logs available within the hooks. If you do require the logs for a different purpose, then we'll have to specifically make it available to the hooks.

[DomainFlag]

@ananyahjha93 How about custom logging, as it's referred from the thread, I need full control over the logging...

[antoinebrl]

I'm wondering if get_batch_log_metrics could help you. See Result.get_batch_log_metrics documentation

[rohitgr7]

It's available on 1.0.2 now.