Tensorboard: Chart domains should update as training progresses

[GalOshri]

TensorBoard version: https://pypi.org/project/tb-nightly/1.14.0a20190301/

Repro steps:

1. Train with Keras.fit() using the TensorBoard callback for 5 epochs
2. Open TensorBoard and look at the Scalars dashboard. See the loss and accuracy charts for epochs 1-5.
3. Continue training with Keras.fit() using the TensorBoard callback for another 5 epochs

Expected: As long as I didn't manually change the zoom of the charts, the chart should rescale to show me the new data (of the new epochs).

Actual: The data from the continued training gets added to the chart but I can see that it goes off the chart.

Relevant code (assuming notebook environment):

model.fit(x=x_train, 
          y=y_train, 
          epochs=5, 
          validation_data=(x_test, y_test), 
          callbacks=[tensorboard_callback])

```
%tensorboard --logdir logs

```py
model.fit(x=x_train, 
          y=y_train, 
          epochs=10,
          initial_epoch=5,
          validation_data=(x_test, y_test), 
          callbacks=[tensorboard_callback])

Image of current behavior:

Note: this is not specific to Keras.fit().

[nfelt]

If you did manually change the zoom of the charts, would you expect it to stay fixed, or for it to scroll as the run progresses to the right? The latter was requested in #1897.

[GalOshri]

That is more ambiguous :). If I zoomed in on some data, I would not want things to suddenly shift without me being able to stop it. We could add some form of "auto-scrolling" toggle, but I would prioritize this lower than the case where I didn't change any zoom settings.

What do others think?

[jarlva]

@GalOshri suggestion for "auto-scrolling" toggle makes sense. Another option is if the "fit domain to data" button was pressed then it would be considered as "ok to auto-shift left".

[makslevental]

yea the fact that domain doesn't refit is definitely confusing to me.

[pesapower]

This issue has been fixed in Tensorboard 2.1.1.

[jacobaustin123]

This issue still isn't fixed for me in Tensorboard 2.2.1. I'd love to be able to watch my results live like I can in Visdom, instead of being forced to double click the chart every time I reload.

[asorie]

This problem still exists in version 2.3.0. Is there any update on this?

[thavlik]

It's kind of difficult to believe more people aren't after this! Hitting half a dozen buttons to window the data every few minutes while monitoring training is annoying. Providing both "refit on update" and "pan horizontally on update" would be superb.

[nfelt]

Thanks for your continued interest, folks. I'm happy to announce that our experimental new Time Series dashboard in TensorBoard 2.4.0 (most recent release) addresses the "refit on update" part of the issue. The new dashboard lets you view scalars, histograms, and images side-by-side in a combined view, and the scalar charts in the Time Series dashboard should auto-adjust the chart domain when new data is loaded as long as you haven't changed the pan or zoom settings.

Please give that a try if you can and let us know if you have feedback. We recognize the "pan horizontally on update" part of the FR is not addressed by these changes, but as discussed above, it's a little less clear how to automatically do the right thing in that case. For example, if the new data shifts in the Y axis such that none of it would be within the zoomed view, showing a blank chart could be confusing, but panning the chart to show the data could also be confusing.

We can consider at least showing a UI indication of new data to display in that case; it would also be helpful to know in more detail about exactly the behavior you would prefer to see there, and what is leading you to zoom in on the chart in the first place, since there might be other ways to approach this.