Tensorboard: Development Environment: ValueError: Protocol message has no non-repeated submessage field "metadata"

[weberxie]

Hi, I cloned the latest tensorboard code, and followed development doc then I run:
bazel run //tensorboard/plugins/scalar:scalars_demo
to generate fake data, then I run:
bazel run //tensorboard -- --logdir /tmp/scalars_demo

Unexpectedly, the terminal show:

INFO: Found 1 target...
Target //tensorboard:tensorboard up-to-date:
bazel-bin/tensorboard/tensorboard
INFO: Elapsed time: 4.033s, Critical Path: 0.44s

INFO: Running command line: bazel-bin/tensorboard/tensorboard --logdir /tmp/scalars_demo
Exception in thread Reloader:
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(self.__args, *self.__kwargs)
File "/private/var/tmp/_bazel_xxx/3148572edcd824f06744282741008230/execroot/org_tensorflow_tensorboard/bazel-out/darwin_x86_64-fastbuild/bin/tensorboard/tensorboard.runfiles/org_tensorflow_tensorboard/tensorboard/backend/application.py", line 316, in _reload_forever
reload_multiplexer(multiplexer, path_to_run)
File "/private/var/tmp/_bazel_xietingwen/3148572edcd824f06744282741008230/execroot/org_tensorflow_tensorboard/bazel-out/darwin_x86_64-fastbuild/bin/tensorboard/tensorboard.runfiles/org_tensorflow_tensorboard/tensorboard/backend/application.py", line 290, in reload_multiplexer
multiplexer.Reload()
File "/private/var/tmp/_bazel_xxx/3148572edcd824f06744282741008230/execroot/org_tensorflow_tensorboard/bazel-out/darwin_x86_64-fastbuild/bin/tensorboard/tensorboard.runfiles/org_tensorflow_tensorboard/tensorboard/backend/event_processing/event_multiplexer.py", line 189, in Reload
accumulator.Reload()
File "/private/var/tmp/_bazel_xxx/3148572edcd824f06744282741008230/execroot/org_tensorflow_tensorboard/bazel-out/darwin_x86_64-fastbuild/bin/tensorboard/tensorboard.runfiles/org_tensorflow_tensorboard/tensorboard/backend/event_processing/event_accumulator.py", line 239, in Reload
self._ProcessEvent(event)
File "/private/var/tmp/_bazel_xietingwen/3148572edcd824f06744282741008230/execroot/org_tensorflow_tensorboard/bazel-out/darwin_x86_64-fastbuild/bin/tensorboard/tensorboard.runfiles/org_tensorflow_tensorboard/tensorboard/backend/event_processing/event_accumulator.py", line 371, in _ProcessEvent
if value.HasField('metadata'):
File "/private/var/tmp/_bazel_xxx/3148572edcd824f06744282741008230/execroot/org_tensorflow_tensorboard/bazel-out/darwin_x86_64-fastbuild/bin/tensorboard/tensorboard.runfiles/protobuf/python/google/protobuf/internal/python_message.py", line 810, in HasField
raise ValueError(error_msg % field_name)
ValueError: Protocol message has no non-repeated submessage field "metadata"

I wonder if it is inconsistent event format between the latest tensorboar and fake data generator.
So I run bazel run //tensorboard -- --logdir /tmp/tf_1.2_event_data again, which event data was produced by tensorflow r.12, then it raised the same error.

I want to build a development environment for do some some customization needs, such as multi-tenant, Any one can help me resolve this problem?

By the way, why the DEVELOPMENT.md doc of independent Tensorboard is so simple compared with the Tensorboard of Tensorflow r1.2?

[wchargin]

TensorBoard depends on the latest nightyl version of TensorFlow; it will not work on TensorFlow r1.2. What version of TensorFlow do you have installed in your virtualenv? (Based on the error, i strongly suspect that installing the latest nightly will fix your problem.)

[chihuahua]

Per @wchargin, the metadata field had only been recently added to the Value proto within TensorFlow:
https://github.com/tensorflow/tensorflow/blob/de0904e90669d345e0feaa89d1e99b7618fb6677/tensorflow/core/framework/summary.proto#L113

pip installing TensorFlow nightly or one of these wheels would likely fix this issue.

[weberxie]

@wchargin @chihuahua Thanks for your reply. I solved this problem under your advice.

I have another question may not suitable to mention in this issue, could you give me some advice for which branch I should choose to do some customized features?

I'm trying to build a development environment for TensorBoard, the latest development doc is too simple, on the contrary, the Tensorboard development doc of branch r1.2 in the TensorFlow project is very datailed, such as branch r1.2 introduced how to Launching a Development Instance, why the latest TensorBoard have removed this doc? Should I follow the old doc to build the latest version's development environment, or it is possible to provide an up-to-date development doc?

Look forward to your reply.😁

[BinhangYuan]

Hi @weberxie , I asked the same question in #272. I hope it also helps you.

[weberxie]

Hi @BinhangYuan , thanks for your reply, I solved my first problem after install latest TensorFlow. I didn't meet the problem you pointed out.

[BinhangYuan]

I'm trying to build a development environment for TensorBoard, the latest development doc is too simple, on the contrary, the Tensorboard development doc of branch r1.2 in the TensorFlow project is very datailed, such as branch r1.2 introduced how to Launching a Development Instance, why the latest TensorBoard have removed this doc? Should I follow the old doc to build the latest version's development environment, or it is possible to provide an up-to-date development doc?

I think it will address this issue.

Instead of forking the repository, you'll want to follow the instructions in tensorflow/custom-tensorboard-example. That repository has a fully self-contained example plugin, and also has reasonably detailed instructions about how plugin development works.

[wchargin]

I solved my first problem after install latest TensorFlow.

Great!

[wchargin]

branch r1.2 introduced how to Launching a Development Instance, why the latest TensorBoard have removed this doc?

Because it doesn't work! We don't support this feature. :-)

[chrisranderson]

@wchargin Since TensorBoard relies on a nightly build of TensorFlow, that means that people using 3rd party plugins will have to reinstall TensorFlow (and cuDNN 6 it appears?) on top of the other requirements. Using a different version of TensorFlow is a show stopper for some folks I would guess; easier for them not to use a plugin than to install a new version of TensorFlow and potentially port their code over.

Would it be better to rely on the most recent stable version of TensorFlow instead that lots of people already use? Just trying to lower the barrier to entry as much as possible. :) Right now you can use Beholder, but not see other summaries because of this issue.

[wchargin]

people using 3rd party plugins will have to reinstall TensorFlow

Correct. I'm not familiar with cuDNN, but maybe?

Once tf1.3 is released (which is soon—1.3rc1 is out), TensorBoard will be able to depend on just "latest TensorFlow." What we really use is tf.SummaryMetadata, which was introduced in 1.3 and is available in nightlies only (plus 1.3 release candidates) right now.

Would it be better to rely on the most recent stable version of TensorFlow instead that lots of people already use? Just trying to lower the barrier to entry as much as possible.

It should always be the case that the latest TensorBoard on pypi (pip install tensorflow-tensorboard) only depends on the latest version of TensorFlow. It's just that GitHub TensorBoard master contains lots of things that you as a plugin author need (as you know), and we just haven't released a new pip package yet.

Does this allay your fears somewhat?