Pytorch-lightning: Flush Events for Tensorboard Logger

[sauhaardac]

🚀 Feature

Add an option to automatically flush events every epoch for TensorBoardLogger

Motivation

TensorboardX and Tensorboard only refresh once their "buffer" is filled. Often, the user would like to see an update before the buffer is filled (say, after every epoch). In tensorboardX/tensorflow, this is accomplished by calling "flush_events."

Alternatives

If you could share how to manually call flush_events so I can insert it into my training loop, that would be great. This feature is not currently described anywhere in the documentation.

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[awaelchli]

Did you try self.logger.experiment.flush()?

[williamFalcon]

@awaelchli i thought we did this automatically?
@tullie

[tullie]

SummaryWriter writes asynchronously every 120 seconds or when the max_queue_size is 10. These defaults can be changed with the SummaryWriter args: max_queue_size, flush_secs.

For our use case at work we can't write every iteration (I think it's too slow because we write to an nfs). Are users fine with explicitly flushing themself with self.logger.experiment.flush()? Otherwise we could make it an arg for the logger interface?

[awaelchli]

We could generally just add a kwargs argument to the wrapper and pass them down to the SummaryWriter. I don't see a reason why to hide these args.

[williamFalcon]

maybe we can tie it to the flag about intervals of saving to disk? that flag already exists today. If we enable -1 it will use the default tensorboard interval or the default logger interval

[sauhaardac]

I'll do self.logger.experiment.flush() for now. Thanks @awaelchli

I think flushing every epoch should be default behavior, with less frequent flushing an optional parameter you can pass (similar to how validation is handled).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.