Tensorboardx: Incompatible with asynchronous event writing

[hrtang]

The following code breaks if several processes spawned by multiprocessing attempt to write events asynchronously.
https://github.com/lanpa/tensorboard-pytorch/blob/594a01b73e6849a9c8e2504e17b5d8d8544aa88a/tensorboardX/event_file_writer.py#L118

A simple fix is to replace six.moves.queue.Queue by multiprocessing.JoinableQueue.

[lanpa]

I am not familiar with async programming. I googled around and found that popped object from multiprocessing.JoinableQueue may be out of order. so we need to test tensorboard's behavior under this. Can you show the error message or code to reproduce it? I am pretty interested, thanks.

[hrtang]

Sorry for the late reply. I have been using multiprocessing.JoinableQueue for my code base on asynchronous methods for reinforcement learning, and it has not caused any trouble so far. But there are probably better ways to fix it.

Here is a simple example to reproduce the bug. It is a tricky one, since it runs perfectly if n = 10.

import multiprocessing as mp
from tensorboardX import SummaryWriter

writer = SummaryWriter(log_dir="/tmp/test")


def run(i):
    n = 100
    for t in range(n):
        writer.add_scalar(str(i), i, t)


processes = [
    mp.Process(
        target=run,
        args=(i, )
    )
    for i in range(2)
]

for p in processes:
    p.start()

for p in processes:
    p.join()

The processes will run into dead locks. If you kill them with KeyboardInterrupt, you will see the following.

^CProcess Process-2:
Process Process-1:
Traceback (most recent call last):
  File "test_tensorboardX.py", line 24, in <module>
    p.join()
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/process.py", line 124, in join
    res = self._popen.wait(timeout)
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/popen_fork.py", line 57, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/popen_fork.py", line 35, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test_tensorboardX.py", line 9, in run
    writer.add_scalar(str(i), i, t)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/writer.py", line 272, in add_scalar
    self.file_writer.add_summary(scalar(tag, scalar_value), global_step)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/writer.py", line 96, in add_summary
    self._add_event(event, global_step)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/writer.py", line 126, in _add_event
    self.event_writer.add_event(event)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/event_file_writer.py", line 148, in add_event
    self._event_queue.put(event)
  File "/home/main/anaconda3/lib/python3.6/queue.py", line 133, in put
    self.not_full.wait()
  File "/home/main/anaconda3/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/main/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "test_tensorboardX.py", line 9, in run
    writer.add_scalar(str(i), i, t)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/writer.py", line 272, in add_scalar
    self.file_writer.add_summary(scalar(tag, scalar_value), global_step)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/writer.py", line 96, in add_summary
    self._add_event(event, global_step)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/writer.py", line 126, in _add_event
    self.event_writer.add_event(event)
  File "/home/main/anaconda3/lib/python3.6/site-packages/tensorboardX/event_file_writer.py", line 148, in add_event
    self._event_queue.put(event)
  File "/home/main/anaconda3/lib/python3.6/queue.py", line 133, in put
    self.not_full.wait()
  File "/home/main/anaconda3/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
KeyboardInterrupt

[darkrush]

The following code breaks if several processes spawned by multiprocessing attempt to write events asynchronously.
https://github.com/lanpa/tensorboard-pytorch/blob/594a01b73e6849a9c8e2504e17b5d8d8544aa88a/tensorboardX/event_file_writer.py#L118

A simple fix is to replace six.moves.queue.Queue by multiprocessing.JoinableQueue.

Just simply replace, and this work for me. Thank you!!!!! : )

[weicheng113]

Thanks for the great library. I am having the same issue. Can this be fixed? multiprocessing is a very common requirement. Thanks.