Keras: Assertion error when using histogram_freq>0 in TensorBoard callback

[llach]

Hey guys,
I'm trying to get Keras to work with TensorBoard, but when I set histogram_freq to something different then 0, I get the following AssertionError:

File "/home/projects/keras/keras/callbacks.py", line 925, in on_epoch_end
    assert len(val_data) == len(tensors)

from here.
The objects whose lengths are compared are created a few lines above like so:

val_data = self.validation_data
tensors = (self.model.inputs +
                  self.model.targets +
                  self.model.sample_weights)

Honestly, I don't really see why this assertion makes sense, at least not with weights involved in the tensors. Might be obvious, but its my first time using Keras and I'm happy to learn.

The implementation can be found here. The version online works if you check it out, but changing histogram_freq in this line makes the code crash after the first epoch.

I'm using the master branch, tested with the (currently) newest commit, but this error is present longer, at least since last weekend.

TensorFlow is version 1.11.0 from pip.

Also, as I'm new to Keras, general comments / advice about my code is very welcome.

Edit: #11044 seems to have the same problem.

[gabrieldemarmiesse]

I don't know what is going on, but if it's a user error, we should at least have a proper error message. If it's a bug, we should fix it.

[chencjiajy]

I met this problem too, I'm training a variational autoencoder, my code is silimar to the offitial example。 but I use Tensorboard in fit's callback:
TensorBoard(log_dir='./logs', histogram_freq=1, write_grads=True, write_images=True)
the error is same as llach
but when I set histogram_freq=0, the error disappeare.

[tik0]

It seems that the tensors list is full of None entries in some cases (for me it was the Keras vanilla VAE example as well).
However, removing all None entries before the assertion fixed the bug for me:

Content of _keras/callbacks.py_:

...
tensors = [t for t in tensors if t is not None] # Add this line
assert len(val_data) == len(tensors)
...

[llach]

The fix from @tik0 works fine.

I switched to writing my own callback when using VAEs though. Either way, the issue is resolved for me.

[marcosterland]

@llach Would you please like to share your solution?

[gabrieldemarmiesse]

Reopening this issue until the fix lands in master.

[llach]

@marcosterland as stated above, my solution did not involve any specific fix, I ended up implementing my own callback.

From my point of view, TensorboardCallback is a generic class that works for many models, but not for VAEs. Therefore implementing my own callback class where I choose which parameters to monitor seemed more reasonable and straightforward to me then tracing the error back to Model creation in keras.

If one wants to investigate this issue further, this is a good place to start.

Also #11044 is still open, which is most likely referring to the same issue. I'd suggest closing either that issue or this one so things won't be discussed twice.

[kwaegel]

Bumping this to indicate continued interest. This is the fifth still-open, 9-month-old Keras bug I've encountered in the past two weeks.

[lovellbrian]

Just encountered the same bug porting the Keras VAE example. Surely this should be fixed by now after another 9 months. I can't see how to edit Keras/callbacks.py as I am running in a Colab notebook.