Tensorboard: Tensorboard not starting in notebook

[kettenfett]

I ran the notebooks get_started.ipynb and hyperparameter_tuning_with_hparams.

Each time, after %tensorboard --logdir "logs", I'm getting this under the notebook cell:
ERROR: Timed out waiting for TensorBoard to start. It may still be running as pid 10864.

And then this:

WARNING: Logging before flag parsing goes to stderr.
W0326 09:10:24.666816 1004 manager.py:322] invalid info file: 'C:\Temp\.tensorboard-info\pid-6196.info'
Traceback (most recent call last):
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 317, in get_all
info = _info_from_string(contents)
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 158, in _info_from_string
raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJyb290X2xvZ2RpciJdLCJjb25maWd1cmVfa3dhcmdzIjp7fSwid29ya2luZ19kaXJlY3RvcnkiOiJDOlxccHl0aG9uX2NvZGUifQ==', 'db': '', 'logdir': 'root_logdir', 'path_prefix': '', 'pid': 6196, 'port': 6006, 'start_time': 1553171458, 'version': '1.13.1'}
W0326 09:10:24.666816 1004 manager.py:322] invalid info file: 'C:\Temp\.tensorboard-info\pid-6224.info'
Traceback (most recent call last):
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 317, in get_all
info = _info_from_string(contents)
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 158, in _info_from_string
raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXI9bG9ncy9ocGFyYW1fdHVuaW5nIl0sImNvbmZpZ3VyZV9rd2FyZ3MiOnt9LCJ3b3JraW5nX2RpcmVjdG9yeSI6IkM6XFxweXRob25fY29kZVxcdGVuc29yYm9hcmRfbm90ZWJvb2tzIn0=', 'db': '', 'logdir': 'logs/hparam_tuning', 'path_prefix': '', 'pid': 6224, 'port': 6006, 'start_time': 1553256806, 'version': '1.13.1'}
W0326 09:10:24.682441 1004 manager.py:322] invalid info file: 'C:\Temp\.tensorboard-info\pid-6420.info'
Traceback (most recent call last):
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 317, in get_all
info = _info_from_string(contents)
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 158, in _info_from_string
raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJsb2dzL2hwYXJhbV90dW5pbmciXSwiY29uZmlndXJlX2t3YXJncyI6e30sIndvcmtpbmdfZGlyZWN0b3J5IjoiQzpcXHB5dGhvbl9jb2RlXFx0ZW5zb3Jib2FyZF9ub3RlYm9va3MifQ==', 'db': '', 'logdir': 'logs/hparam_tuning', 'path_prefix': '', 'pid': 6420, 'port': 6006, 'start_time': 1553256443, 'version': '1.13.1'}
W0326 09:10:24.682441 1004 manager.py:322] invalid info file: 'C:\Temp\.tensorboard-info\pid-9488.info'
Traceback (most recent call last):
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 317, in get_all
info = _info_from_string(contents)
File "C:\anaconda\envs\tf2course\lib\site-packages\tensorboard\manager.py", line 158, in _info_from_string
raise ValueError("incompatible version: %r" % (json_value,))
ValueError: incompatible version: {'cache_key': 'eyJhcmd1bWVudHMiOlsiLS1sb2dkaXIiLCJsb2dzIl0sImNvbmZpZ3VyZV9rd2FyZ3MiOnt9LCJ3b3JraW5nX2RpcmVjdG9yeSI6IkM6XFxweXRob25fY29kZVxcdGVuc29yYm9hcmRfbm90ZWJvb2tzIn0=', 'db': '', 'logdir': 'logs', 'path_prefix': '', 'pid': 9488, 'port': 6006, 'start_time': 1553242957, 'version': '1.13.1'}

This goes on in a long list, over and over again.

After a few attempts to start TB like this, I notice several tensorboard.exe entries in my taskmanager processes tab.

Opening TB via console worked fine.
How can I get TB working in notebook?

These are the relevant packages installed in the conda env I use as kernel in jupyter notebook:

tb-nightly                1.14.0a20190319          pypi_0    pypi
tensorflow-estimator-2-0-preview 1.14.0.dev2019031400          pypi_0    pypi
tf-nightly-2-0-preview    2.0.0.dev20190325          pypi_0    pypi

on Windows10 x64, conda 4.6.8, Python 3.6.8

[stephanwlee]

@wchargin probably can answer better but if I may do the first triage, based on your error message, there are TensorBoard==1.13.1 (contrary to the snippet on conda env) on running with pid 9488, 6196, 6224, and 6420. Though you should do a better job than the timeout, you can probably work around it by killing these processes manually first.

[wchargin]

What Stephan says is correct. Can you check whether you also have the
tensorboard package (not tb-nightly) installed in your environment?

If you run !tensorboard --logdir /tmp/whatever from a Jupyter cell,
what version does TensorBoard print out once it starts its server?
(After checking, you can press the “stop” button in Jupyter to kill the
subprocess.)

[kettenfett]

There was no Tensorboard 1.13.1 in that env. I guess it detected the pid.info files from previous runs, when it was still installed. But I uninstalled tensorboard earlier and reinstalled tb-nightly (as *wchargin recommended in #2038).

I also noticed, although conda list did not show 'tensorboard' anymore, there still was a tensorboard folder in site-packages folder. I deleted that, but it didn't help. I decided to setup a fresh Env and the error I originally posted here was gone.

But I'm still having issues starting Tensorboard. I keep getting either timeouts like

"ERROR: Timed out waiting for TensorBoard to start. It may still be running as pid 2020."

or it says

"Reusing TensorBoard on port 6006 (pid 10284), started 0:01:42 ago. (Use '!kill 10284' to kill it.)"

and a browser iframe is shown with a failed connection error.

then suddenly it works after many tries, and i don't know why,

Reusing TensorBoard on port 6006 (pid 11232), started 0:06:12 ago. (Use '!kill 11232' to kill it.)

Next time I start another notebook, or reboot my pc it doesn't start with what I wrote down as working workflow from last time. It's very very confusing.

[wchargin]

there still was a tensorboard folder in site-packages folder. I deleted that

Directories in site-packages are named by the Python module name, not
the Pip distribution name. The tensorboard entry in your site-packages
corresponded to tb-nightly’s code. Deleting it will surely corrupt
your TensorBoard installation:

$ cd "$(mktemp -d)"
$ virtualenv -q ./ve/
$ . ./ve/bin/activate
(ve) $ pip install -q tb-nightly
(ve) $ python -c 'import tensorboard; print(tensorboard)'
<module 'tensorboard' from '/tmp/tmp.XJuvTxHNBs/ve/local/lib/python2.7/site-packages/tensorboard/__init__.pyc'>
(ve) $ rm -r ./ve/local/lib/python2.7/site-packages/tensorboard/
(ve) $ python -c 'import tensorboard; print(tensorboard)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/buildtools/current/sitecustomize/sitecustomize.py", line 152, in SetupPathsAndImport
    return real_import(name, globals, locals, fromlist, level)
ImportError: No module named tensorboard

It sounds like this conda environment is in pretty bad shape
(conflicting versions, manual changes to the internal structures, …).
Can you just blow it away and create a new one?

Please also make sure that you do not have TensorFlow or TensorBoard
installed at the system level: python -c "import tensorflow" should
fail outside of a virtualenv, and so should tensorboard.

[Harshini-Gadige]

@philippHRO Any update on this ?

[Harshini-Gadige]

Closing this issue as it is in "awaiting response" status for more than 3 days. Please post your comments(if any) and we will reopen. Thanks!

[SahilDahiya]

I am having the same issue, I have tried creating the environment again also, tried deleting .tensorboard-info folder also

[wchargin]

Hi @SahilDahiya—the implementation has changed since this issue was
written (incl. #2470), so please open a new issue and provide all the
requested environment information.