Tensorboardx: writer is never closed

[loic001]

Hi,
https://github.com/lanpa/tensorboard-pytorch/blob/master/tensorboardX/record_writer.py
self._writer = open(path, 'wb') is never closed

[mrshu]

@loic001 It is being flushed every time something is being written though:

https://github.com/lanpa/tensorboard-pytorch/blob/8a296a17e35217f9f9632ca0fe2ab6e6c5ded02d/tensorboardX/record_writer.py#L31

[loic001]

Yes but...

from tensorboardX import SummaryWriter
for i in range(10000):
  writer = SummaryWriter()
  writer.close()
----
OSError: [Errno 24] Too many open files...

It depends on the limit of the maximum number of opened files on the linux system but there shouldn't be this error anyway. That's why I said you should close it when SummaryWriter is closed.

[mrshu]

@loic001 Ah, you are indeed right. I've proposed a fix that should deal with this.

[lanpa]

the test fails on mac.

[mrshu]

@lanpa interesting. Does it fail with OSError: [Errno 24] Too many open files? I sadly do not have a way of easily testing it so more information would be very helpful.

Thanks!

[lanpa]

@mrshu It's RuntimeError: can't start new thread. I also confirmed that it fails on ubuntu if we use: for i in range(1000000): (a bigger number). Should be other issue than this one.

[mrshu]

@lanpa Thanks. Looking at this (http://python.6.x6.nabble.com/maximum-number-of-threads-td1124579.html) it seems that the number is very much dependent on the memory address space of the box you run the test on. In other words, it is not related to the SummaryWriter but rather _EventLoggerThread.

In other words, I'd say that the test is working, but using such a high number does not make too much sense. Using something like a 100 would probably be sufficient.

[NilsRethmeier]

@lanpa
IMHO this should be fixed asap. This is a very serious issue. Opening 10 SummaryWriter creates close to 3k open files. Even after updating to tensorboardX==1.2.

def test_summary_writer_close():
    # Opening and closing SummaryWriter a lot should not run into
    # OSError: [Errno 24] Too many open files
    passed = True
    try:
        for i in range(10):
            writer = SummaryWriter()
            writer.file_writer.flush()
            writer.file_writer.close()
            writer._record_writer
            del writer
            gc.collect()
    except OSError:
        passed = False

    assert passed
test_summary_writer_close()

lsof | grep ' yourusername ' | awk '{print $NF}' | sort | wc -l

the .close() method has no effect.

[Robin-des-Bois]

I also find myself in a situation where I need to have many summary writers and hence run into
RuntimeError: can't start new thread even though I close the summary writers.

What about adding a flag terminate to the close method as a quick fix: When set to True the _EventLoggerThread gets killed.

Ideally close would kill the thread and reopen would create a new one.

[hangg7]

Any updates on this?

[veugene]

An additional problem is that the SummaryWriter doesn't close the file at program exit and actually hangs. This can make some events not flush, if the flusher thread hasn't yet drained the queue when it hangs upon having its stop() method called. Proposing a fix in #498

[jeandut]

Would love to have a fix for this issue ! Can we get #498 accepted now that it passes tests ?