Keras: 'You must feed a value for placeholder tensor' when using Tensorboard callback with submodels

[ophiry]

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [ ] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

• [X] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [ ] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Trying to train a model that is composed of submodels, and using the Tesnorboard callback.
If histogram_freq is set, there's an error about missing input to the submodel

using tensorflow 1.8 (same error also in older versions)

sample code that reproduces the error:

import tensorflow
keras = tensorflow.keras
import numpy


INPUT_LEN = 10
N = 100


def get_sub_model(in_dim, out_dim):
    input_layer = keras.layers.Input(shape=(in_dim,))
    dense=keras.layers.Dense(out_dim)
    return keras.models.Model(inputs=input_layer, outputs=dense(input_layer))


input_layer = keras.layers.Input(shape=(INPUT_LEN,), name='model_input')
sub_model1 = get_sub_model(INPUT_LEN, 5)
sub_model2 = get_sub_model(5, 1)
m = keras.models.Model(inputs=[input_layer], outputs=sub_model2(sub_model1(input_layer)))
m.compile(loss='binary_crossentropy', optimizer='adam')

x = numpy.random.rand(N, INPUT_LEN)
y = numpy.random.randint(0, 2, N)

m.fit(x, y, callbacks=[keras.callbacks.TensorBoard(log_dir='/tmp', histogram_freq=1)], epochs=10, validation_data=(x,y))

error:

Traceback (most recent call last):
  File "/Users/ophir/dev/ophir/tensorboard_issue.py", line 25, in <module>
    m.fit(x, y, callbacks=[keras.callbacks.TensorBoard(log_dir='/tmp', histogram_freq=1)], epochs=10, validation_data=(x,y))
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/engine/training.py", line 1216, in fit
    validation_steps=validation_steps)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/engine/training_arrays.py", line 269, in fit_loop
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/callbacks.py", line 95, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/callbacks.py", line 799, in on_epoch_end
    result = self.sess.run([self.merged], feed_dict=feed_dict)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,10]
     [[Node: input_1 = Placeholder[dtype=DT_FLOAT, shape=[?,10], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

The reason of using submodels is the need to also run them separately

[Dref360]

Your model has 2 inputs and you feed only one.

[ophiry]

@Dref360 it's not 2 inputs - one input to the model, and inputs to the sub models.

also, calling fit with histogram_freq=0, and then calling predict works well

m.fit(x, y, callbacks=[keras.callbacks.TensorBoard(log_dir='/tmp', histogram_freq=0)], epochs=10, validation_data=(x, y))

print(m.predict(x))

[sleighsoft]

I have a very similar issue:

i1 = Input()
m1 = Model(inputs=i1, ...)
o = m1(i1)
i2 = Input()
i3 = Input()
m2 = Model(inputs=[i2,i3],...)
o2 = m2([i2, o])

final = Model(inputs=[i1, i2], outputs=o2)

Throws an error that i3 placeholder is not specified even though that input is set by o from o=m1(i1) when calling final.predict([x1,x2]).

[wangqin4377]

same here

[pdpino]

I had the same problem, that only appeared when using TensorBoard callback and histogram_freq other than 0. Clearing tensorflow session right before creating the model fixed it:

import keras.backend as K
K.clear_session()

From this issue

[igorgad]

I am having the same problem here. Calling K.clear_session() is not helping on my case. Any workaround?

[dimimal]

I am having the same problem here. Calling K.clear_session() is not helping on my case. Any workaround?

Did you try to call K.clear_session() before you build the model?

[lunaticcoding]

I have the same issue. I call K.clear_session() right before creating the model. :O Any news?

[darteaga]

Same issue here. Calling K.clear_session() before creating the model is not helping on my case. Any news?

[edwardCao]

I have the same issue. the clear_session call does not work.

[icyblade]

Just FYI, for tf.data.Dataset users, call tf.keras.backend.clear_session() before creating Dataset will do the trick.

I can't imagine I encountered the same issue after 3 years later https://github.com/keras-team/keras/issues/4417.

[ISimplifyComplexity]

I get AssertionError: Do not use tf.reset_default_graph() to clear nested graphs. If you need a cleared graph, exit the nesting and create a new graph. when using clear session

[renatosc]

Same problem here. Clearing the session as proposed by @pdpino fixed for me.

[datistiquo]

Same! Clearing before AND before creating tf.dataset with tf.keras not working and same error.

[tiantengfei]

I have the same issue. Below is my solution.
Why do we get this error when there are submodel in our model?
for exmaple, below is a model which has a submodel.

def shared_model(): 
    input = keras.Input(shape=(self.input_dim), dtype=tf.float32, name="shared_input")`
    # model layers
    ......
    ......
    share_out = .....
    shared_model = keras.Model([input], share_out, name="out_model")
    return shared_model

input1 = keras.Input(shape=(self.input_dim), dtype=tf.float32, name="input1")
input2= keras.Input(shape=(self.input_dim), dtype=tf.float32, name="input2")
sub_model = shared_model()
out1 =  sub_model([input1])
out2 = sub_model([input2])
out = keras.layers.concatenate([out1, out2], axis=1)
out = .....
model = Model(.....)
model.compile(.....)
tensorboard =  keras.callbacks.TensorBoard(histogram_freq=1,........)
model.fit(...., callbacks=[ tensorboard])```

When training the model with the above code, get error:
'You must feed a value for placeholder tensor shared_input'.
why shared_input ? Let's see keras.callbacks.TensorBoard source code:

 if self.histogram_freq and self.merged is None:
      for layer in self.model.layers:
        for weight in layer.weights:
          mapped_weight_name = weight.name.replace(':', '_')
          tf_summary.histogram(mapped_weight_name, weight)
          if self.write_images:
            w_img = array_ops.squeeze(weight)
            shape = K.int_shape(w_img)
            if len(shape) == 2:  # dense layer kernel case
              if shape[0] > shape[1]:
                w_img = array_ops.transpose(w_img)
                shape = K.int_shape(w_img)
              w_img = array_ops.reshape(w_img, [1, shape[0], shape[1], 1])
            elif len(shape) == 3:  # convnet case
              if K.image_data_format() == 'channels_last':
                # switch to channels_first to display
                # every kernel as a separate image
                w_img = array_ops.transpose(w_img, perm=[2, 0, 1])
                shape = K.int_shape(w_img)
              w_img = array_ops.reshape(w_img,
                                        [shape[0], shape[1], shape[2], 1])
            elif len(shape) == 1:  # bias case
              w_img = array_ops.reshape(w_img, [1, shape[0], 1, 1])
            else:
              # not possible to handle 3D convnets etc.
              continue

            shape = K.int_shape(w_img)
            assert len(shape) == 4 and shape[-1] in [1, 3, 4]
            tf_summary.image(mapped_weight_name, w_img)

when histogram_freq=1, layer 'out_model' in self.model.layers is a submodel which has its own input 'shared_input'. tf_summary.histogram(mapped_weight_name, weight) add this op to tf.Session() to run.
but submodel is a independent model , not in the tensorflow default graph. So we need feed value for shared_input. When modifying the code:

  for layer in self.model.layers:
        if layer.name == 'out_model':
              continue
        for weight in layer.weights:
          mapped_weight_name = weight.name.replace(':', '_')
          tf_summary.histogram(mapped_weight_name, weight)

With above code, we don't add submodel to session.run. It works fine. But it does't add histograms for weights and bias.
How can we add histograms for these ? It‘s clear now. It works if we get weights, bias and so on from tensorflow default graph， not keras's layers. In tensorflow, tf.trainable_variables() get all trainable variables besides weights and bias. When modifying the code:
```
if self.histogram_freq and self.merged is None:
for weight in tf.trainable_variables():
mapped_weight_name = weight.name.replace(':', '_')
tf_summary.histogram(mapped_weight_name, weight)
if self.write_images:
w_img = array_ops.squeeze(weight)
shape = K.backend.int_shape(w_img)
if len(shape) == 2: # dense layer kernel case
if shape[0] > shape[1]:
w_img = array_ops.transpose(w_img)
shape = K.backend.int_shape(w_img)
w_img = array_ops.reshape(w_img, [1, shape[0], shape[1], 1])
elif len(shape) == 3: # convnet case
if K.image_data_format() == 'channels_last':
# switch to channels_first to display
# every kernel as a separate image
w_img = array_ops.transpose(w_img, perm=[2, 0, 1])
shape = K.backend.int_shape(w_img)
w_img = array_ops.reshape(w_img,
[shape[0], shape[1], shape[2], 1])
elif len(shape) == 1: # bias case
w_img = array_ops.reshape(w_img, [1, shape[0], 1, 1])
else:
# not possible to handle 3D convnets etc.
continue

                shape = K.backend.int_shape(w_img)
                assert len(shape) == 4 and shape[-1] in [1, 3, 4]
                tf_summary.image(mapped_weight_name, w_img)

```
Using the above code will not report error any more.
You can write yourself custom callback with a little different with keras.callback.Tensorboard in set_model method. Alternately, you can inherit keras.callback.Tensorboard and rewrite set_model method.

[sabyUWO]

@ophiry , were u able to resolve this issue?

[beasteers]

@tiantengfei you'd probably want to be using model.trainable_variables instead of tf.trainable_variables() so that it is scoped by the model, right? This looks like a simple enough fix. It looks like it's still like that in tfv2 as well

[fantasyjie]

sess=tf.Session()
saver = tf.train.import_meta_graph('/content/checkpoint_dir/MyModel_all.meta')
saver.restore(sess, tf.train.latest_checkpoint('./checkpoint_dir'))
w1 = graph.get_tensor_by_name('x1_all:0')

w1 shape is
x = np.linspace(0,3,100).reshape(-1,1)
feed_dict ={w1:x)

but when the feed_dict shape is <100,1>，turns a error?

the error is
You must feed a value for placeholder tensor 'x1' with dtype float and shape [?,1]

[ISimplifyComplexity]

I think it's row by column so your second value is the X which is 1.. the
first value is Y which is your variable

On Thu, 19 Dec 2019, 1:52 pm fantasyjie, [e-mail] wrote:

sess=tf.Session()
saver =
tf.train.import_meta_graph('/content/checkpoint_dir/MyModel_all.meta')
saver.restore(sess, tf.train.latest_checkpoint('./checkpoint_dir'))
w1 = graph.get_tensor_by_name('x1_all:0')

w1 shape is
x = np.linspace(0,3,100).reshape(-1,1)
feed_dict ={w1:x)

but when the feed_dict shape is <100,1>，turns a error?

the error is
You must feed a value for placeholder tensor 'x1' with dtype float and
shape [?,1]

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/keras-team/keras/issues/10074?email_source=notifications&email_token=AA6XBOMPHEFHEPN6OVLE3CLQZN4BLA5CNFSM4E5T24KKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJVLDA#issuecomment-567498124,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA6XBOLFT4QNTJQ3JVESSSDQZN4BLANCNFSM4E5T24KA
.

[fantasyjie]

I think it's row by column so your second value is the X which is 1.. the first value is Y which is your variable
…
On Thu, 19 Dec 2019, 1:52 pm fantasyjie, @.*> wrote: sess=tf.Session() saver = tf.train.import_meta_graph('/content/checkpoint_dir/MyModel_all.meta') saver.restore(sess, tf.train.latest_checkpoint('./checkpoint_dir')) w1 = graph.get_tensor_by_name('x1_all:0') w1 shape is x = np.linspace(0,3,100).reshape(-1,1) feed_dict ={w1:x) but when the feed_dict shape is <100,1>，turns a error? the error is You must feed a value for placeholder tensor 'x1' with dtype float and shape [?,1] — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10074?email_source=notifications&email_token=AA6XBOMPHEFHEPN6OVLE3CLQZN4BLA5CNFSM4E5T24KKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJVLDA#issuecomment-567498124>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6XBOLFT4QNTJQ3JVESSSDQZN4BLANCNFSM4E5T24KA .

thank you very much,you are right ,i had a stupid mistake,thanks a lot.

[ISimplifyComplexity]

Fantastic, Glad I could help.

I’ve made that mistake myself a few times. I wasn’t entirely sure if my answer was right so I’m glad it helped you 😊