Tensorboard: TypeError with GetNext() for TF 1.8+ or 1.7 source

[advaza]

Hi,

I upgraded my tensorflow library to 1.7, and now I get an error when executing tensorboard.

System description:

• TensorBoard version: 1.7
• TensorFlow version: 1.7 (compiled from source)
• OS Platform and version: Linux Ubuntu 16.04.2
• Python version: 3.5.3

Error:

(tensorflow)/home/username$ tensorboard --logdir=/mydir/log
2018-04-07 01:56:45.493235: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-04-07 01:56:45.493638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:02:00.0
totalMemory: 11.91GiB freeMemory: 11.74GiB
2018-04-07 01:56:45.493651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-04-07 01:56:45.637711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-04-07 01:56:45.637737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-04-07 01:56:45.637741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
Exception in thread Reloader:
Traceback (most recent call last):
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/threading.py", line 862, in run
self._target(self._args, *self._kwargs)
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/application.py", line 360, in _reload_forever
reload_multiplexer(multiplexer, path_to_run)
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/application.py", line 334, in reload_multiplexer
multiplexer.Reload()
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 195, in Reload
accumulator.Reload()
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 185, in Reload
for event in self._generator.Load():
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 88, in Load
for event in self._LoadInternal():
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 112, in _LoadInternal
for event in self._loader.Load():
File "/home/username/anaconda2/envs/tensorflow/lib/python3.5/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 53, in Load
self._reader.GetNext(status)
TypeError: GetNext() takes 1 positional argument but 2 were given

TensorBoard 1.7.0 at http://my-computer:6006 (Press CTRL+C to quit)

Thanks!
Adva

[chihuahua]

You noted compiling TensorBoard from source. Could you make sure that the source you are compiling from includes https://github.com/tensorflow/tensorboard/pull/1086?

[advaza]

Sorry it was a typo, I compiled tensorflow from source and tensorboard is compiled too. I used the latest master commit (commit 8b5212011c7b67b7f8c2ea1b641aa0a7151c82d0).

Also I just reinstalled (again from source) in a new virtual environment and got the same error.

[chihuahua]

TensorBoard's 1.7 branch excludes #1086, so would you be open to installing TensorBoard's nightly build via pip install tb-nightly? The nightly version of TensorBoard includes bleeding edge features and is compatible with the nightly version of TensorFlow (built the night before).

[advaza]

It solves the problem, thanks!

[chihuahua]

yay

[rani-pinchuk]

Thanks advaza for reporting this, and thanks chihuahua for guiding how to solve it. I faced the same problem and indeed the night built solves it.

[aefuimn]

Thanks advaza for reporting this, and thanks chihuahua for guiding how to solve it. I faced the same problem and indeed the night built solves it.

[samueljosephk]

pip install tb-nightly did not solve it for me but pip install tf-nightly did.

[venik]

for those who build from the source, tag: v1.8.0 commit 93bc2e doesn't have this problem, tag v1.8.0-rc1 has this problem

[nfelt]

TensorBoard 1.8.0 fixes this issue. If you're still seeing this issue, try pip install tensorboard==1.8.0 (or if you have tensorflow installed via pip, update tensorflow via pip install tensorflow==1.8.0 which will also update tensorboard).

[sophistcxf]

pip install tb-nightly solves my problem. By the way, what's the funciton of tb-nightly?

[martin-gorner]

I am using the latest version of tensorflow and tensorboard, installed via pip: 1.8.0 and 1.8.0
The problem still exists. Is there a solution outside of nightly builds or compiling from source ?

[jart]

@martin-gorner Could you provide stack traces and logs? Does pip install tb-nightly help?

[sachinbijalwan]

I also faced this issue. Nothing mentioned above was working for me. It was then I noticed that when I launched tensorboard it's version was 1.5 even though after installing tensorboard >=1.8 . I ran pip freeze and found that their was another package called tensorflow-tensorboard which was launching itself whenever I ran tensorboard. So, I uninstalled it and everything worked fine.

[martin-gorner]

same here, I had to uninstall tensorflow-tensorboard manually

[hadikazemi]

Only the last solution worked for me:
sudo pip uninstall tensorflow-tensorboard

[relh]

If you are running different versions of tensorflow/tensorboard make sure you are running the right version to monitor the right experiments.

conda environments help for this.

I got this error running tensorboard 1.8.0 trying to monitor tensorflow 1.4.1 experiments.

[GeorgSatyros]

Got this error when using tensorflow and tensorboard 1.7.0.Upgrading to tensorboard 1.8.0 fixed the issue, as @nfelt mentioned above.

[martianvenusian]

@chihuahua thank you.
sudo pip3 install tb-nightly solved my problem.