Tensorboardx: Detected out of order event.step when adding text after scalars with global_step

[timbmg]

When logging text after logging scalar values with the global_step parameter set, tensorboard gives a warning (see below).

This is a MWE:

from tensorboard import SummaryWriter
writer = SummaryWriter('runs/test')
writer.add_text("T", "42")
writer.add_scalar("S", 0, 0)
writer.add_scalar("S", 0, 1)
writer.add_scalar("S", 0, 2)
writer.add_text("T", "42")

WARNING:tensorflow:Detected out of order event.step likely caused by a TensorFlow restart. Purging expired events from Tensorboard display between the previous step: 2 (timestamp: 1501673169.092804) and current step: 0 (timestamp: 1501673169.0928545). Removing 3 scalars, 0 histograms, 0 compressed histograms, 0 images, and 0 audio.

As can be seen from the warning log, this actually removes logged data.

Is this a problem just on my side or tensorboard related or from the pytorch wrapper?

[lanpa]

By default, tensorboard removes out of order events.
So you can add a third parameter niter for add_text to solve it.

writer.add_text("T", "42", 0)
writer.add_scalar("S", 0, 0)
writer.add_scalar("S", 0, 1)
writer.add_scalar("S", 0, 2)
writer.add_text("T", "42", 2) #<- not less than previous one

or pass --nopurge_orphaned_data when invoke tensorboard server.

[mjasonong]

By default, tensorboard removes out of order events.
So you can add a third parameter niter for add_text to solve it.

writer.add_text("T", "42", 0)
writer.add_scalar("S", 0, 0)
writer.add_scalar("S", 0, 1)
writer.add_scalar("S", 0, 2)
writer.add_text("T", "42", 2) #<- not less than previous one

or pass --nopurge_orphaned_data when invoke tensorboard server.

Hi, I also have this problem right now, may I ask where to add this codes?