How do I access the measured time per epoch shown in progress bar?
Epoch 8: : 150it [00:03, 46.84it/s, loss=1.625, v_num=0]
It clearly says 00:03 i.e. 3 seconds and I could parse the logs as a hack, but I was wondering if there was any clean way to access the measured time elapsed per training epoch, as I would like to write it to a file.
And if not, how would you recommend timing each epoch? I could put a time.time() in training_step but then would need to add it all up in training_epoch_end, would this work? Thank you!
Just an idea, if you use tensorboard as the logger, it has timestamps for everything, you could open the log (in python) and find the stamps.
I would use tensorboard logs.
Otherwise just build your own callback and overwrite epoch_start and epoch_end
https://pytorch-lightning.readthedocs.io/en/latest/callbacks.html
Most helpful comment
Just an idea, if you use tensorboard as the logger, it has timestamps for everything, you could open the log (in python) and find the stamps.