Keras: Tensorboard callback only supports logging Embeddings layer weights

[gewoonrik]

Currently the Tensorboard embeddings logger (for visualizing embeddings) only logs the weights of an Embedding layer.
It would be interesting if it could log the output of every layer, instead of only the weights.
This way you can visualize how the output changes after every layer.

[gokceneraslan]

tf.train.Saver only serializes variables, activations on the other hand are only tensors. So it's tricky, but it might work with a workaround like saving activations to a variable and passing it to the Saver function.

[gewoonrik]

I tried replacing:
weights[0] with output on this line: https://github.com/fchollet/keras/blob/master/keras/callbacks.py#L715
but that didn't work, Tensorboard seemed to crash because output has no static shape.

Or maybe it did not save anything at all, I don't know

[tomgankema]

I second this request. At the moment the embeddings part of the TensorBoard callback logs weights, which are entirely different from embeddings. Embeddings are typically the output of a layer given an input. Then you can attach metadata to see how your input data translates to some latent space. I would argue the current implementation is just incorrect.

I did look into the problem, however the tensorboard documentation on embeddings is quite minimal and there is no good example code. So I eventually gave up on making it work with the current keras callback. The typical usage pattern however is like below (in psydo-python just to bring the idea accross), the following you only have to call once.

sess = K.get_session()

# define an embedding
embedding_input = model.get_layer(embeddings_layer_name).output
embedding_size = int(embedding_input.shape[-1])
embedding = tf.Variable(tf.zeros([validation_data_size, embedding_size]), name='embedding')

# assign the embedding
assignment = embedding.assign(embedding_input)

# define a saver and writer
saver = tf.train.Saver()
writer = tf.summary.FileWriter(log_dir)
writer.add_graph(sess.graph)

# define a projector config
config = projector.ProjectorConfig()
embedding_config = config.embeddings.add()
embedding_config.tensor_name = embedding.name
embedding_config.metadata_path = metadata_path

tensorflow.contrib.tensorboard.plugins.projector.visualize_embeddings(writer, config)

Then every x batches/epochs you call the following:

sess.run(assignment, feed_dict={model.input: validation_data})
saver.save(sess, os.path.join(log_dir, 'keras_embedding.ckpt'), epoch)

(inspired by: https://github.com/dandelionmane/tf-dev-summit-tensorboard-tutorial)

If anyone could integrate this properly into the current callback then I think that would be really nice.

[mhajiaghayi]

I'm trying to visualize the embedding layer output for the imdb sentiments in Keras. But I don't get the expected result. Specifically, I don't see the expected relation among words, e.g. word kingdom is not near king or queen. can this issue be related to what you have mentioned above? I get a good accuracy for the model. so I suppose the embedding should be fine too. I have attached the code here.

from __future__ import print_function
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import LSTM
from keras.datasets import imdb
from keras.callbacks import TensorBoard
from keras.callbacks import ModelCheckpoint
import pdb,shutil
import tensorflow as tf 
max_features = 1000
maxlen = 80  # cut texts after this number of words (among top max_features most common words)
batch_size = 32


def saveActionMetaData(_map, fileName,max_features):
    row = 0
    with open(fileName, 'w') as f:
        f.write("Word\tFrequency\n")
        for action, _id in _map.items():
            try:
                if _id <= max_features:
                    if row < max_features:
                        f.write(action + "\t" +str(_id) + "\n")
                    else: 
                        f.write(action + "\t" +str(_id))
                    row = row + 1 
            except:
                print("something is wrong")
                print(action)
                continue 
    return

with tf.device('cpu:0'): 
    ROOT_DIR = 'tfboard'
    shutil.rmtree(ROOT_DIR, ignore_errors=True)
    print('Loading data...')
    (x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
    _map = imdb.get_word_index()
    saveActionMetaData(_map, 'metadata.tsv',max_features)
    print(len(x_train), 'train sequences')
    print(len(x_test), 'test sequences')
    print('Pad sequences (samples x time)')
    x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
    x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
    print('x_train shape:', x_train.shape)
    print('x_test shape:', x_test.shape)

    print('Build model...')
    model = Sequential()
    model.add(Embedding(max_features, 40, name = 'embedding'))
    model.add(LSTM(40, dropout=0.2, recurrent_dropout=0.2))
    model.add(Dense(1, activation='sigmoid'))

    # try using different optimizers and different optimizer configs
    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=['accuracy'])

    print('Train...')


    embeddingsMetadata = {'embedding': 'metadata.tsv'}
    tbCallback = TensorBoard( log_dir= ROOT_DIR,
                            histogram_freq= 10, 
                            write_graph = False,
                            embeddings_freq=10, 
                            batch_size=32,
                            embeddings_layer_names = ['embedding'],
                            embeddings_metadata= embeddingsMetadata
                             )
    model.fit(x_train, y_train,
              batch_size=batch_size,
              epochs=40,
              validation_data=(x_test, y_test),
              callbacks = [tbCallback],
              verbose = 1 )
    score, acc = model.evaluate(x_test, y_test,
                                batch_size=batch_size)
    print('Test score:', score)
    print('Test accuracy:', acc)

[raghavgurbaxani]

@mhajiaghayi I tried running your code, but I get the following error after the first epoch -
ValueError: To visualize embeddings, embeddings_data must be provided.

[phiyodr]

I am receiving the same error as @raghavgurbaxani:
ValueError: To visualize embeddings, embeddings_data must be provided.

You can fix it with following updates of @mhajiaghayi's code:

1) Add embeddings_data=x_test, so that:
tbCallback = TensorBoard(log_dir=ROOT_DIR, histogram_freq=10, write_graph=False, embeddings_freq=10, batch_size=32, embeddings_layer_names=['embedding'], embeddings_metadata=embeddingsMetadata, embeddings_data=x_test)

2) Add input_length=maxlen, so that:
model.add(Embedding(max_features, 40, name='embedding', input_length=maxlen))
(otherwise you will receive: TypeError: __int__ returned non-int (type NoneType))

Now you can start TensorBoard. During loading the Projector, the Error

Number of tensors (25000) do not match the number of lines in metadata (1000).

shows up.

Actually, the 1000 words should be embedded (as in this Word2Vec-Example) and not the 25000 reviews.
Nevertheless, I replaced metadata.tsv with (following tensorboard_embeddings_mnist.py):

import os
import numpy as np

if not os.path.exists(ROOT_DIR):
    os.makedirs(ROOT_DIR)

with open(os.path.join(ROOT_DIR, 'metadata.tsv'), 'w') as f:
    np.savetxt(f, y_test)

Now the Projector is working. The shown embedding does not show the 1000 words, but the 25000 reviews, what I don't want.

My questions:

• How to extract the 80d embedding for the words?
• Does anyone know where to find the code from the Word2Vec example from http://projector.tensorflow.org/?