Keras: Error when using BatchNormalization Layer in GPU mode

Created on 7 Dec 2016 · 9Comments · Source: keras-team/keras

I tried to build a LSTM model, attached the code:
model = Sequential()
model.add(BatchNormalization(batch_input_shape = (None,None,513),mode=0, axis=2))
model.add(LSTM(input_dim=513, input_length=None, output_dim=1026, return_sequences=True))
model.add(TimeDistributed(Dense(output_dim=1026)))
model.add(Activation("sigmoid"))
model.compile(optimizer='RMSprop', loss='binary_crossentropy')

The code works fine in CPU, when I turned to use GPU. It caused an error:
Failed precondition: Attempting to use un
initialized value batchnormalization_1_running_mean/biased
[[Node: batchnormalization_1_running_mean/biased/read = IdentityT=DT_FLOAT, _
class=["loc:@batchnormalization_1_running_mean"], _device="/job:localhost/replica:0/tas
k:0/gpu:0"]]
Traceback (most recent call last):
File "first_ideal_amplitude_binary_crossentropy.py", line 70, in
hist = model.fit(keras_inputs, keras_targets, nb_epoch=nb_epoch, batch_size= batch_
size, verbose=2, shuffle=True)
File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 620, in fit
sample_weight=sample_weight)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1106, in
fit
callback_metrics=callback_metrics)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 824, in
_fit_loop
outs = f(ins_batch)
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", li
ne 1013, in __call__
updated = session.run(self.outputs + [self.updates_op], feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", li
ne 766, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value batchnormalization_1_running_mean/biased
[[Node: batchnormalization_1_running_mean/biased/read = Identity[T=DT_FLOAT, _class=["loc:@batchnormalization_1_running_mean"], _device="/job:localhost/replica:0/task:0/gpu:0"](batchnormalization_1_running_mean/biased)]]

Do any one meet this problem? Or do any one know how to use keras in CPU mode?

Source

nateanl

👍1

Most helpful comment

using

Keras==2.0.6
tensorflow==1.3.0
tensorflow-gpu==1.3.0

imports are

import tensorflow.contrib.keras as keras
import tensorflow as tf

from keras.engine import Layer, InputSpec
from keras.models import Model, Sequential
from keras.layers import Dropout, Embedding
from keras.layers import Conv1D, MaxPooling1D, Conv2D, MaxPooling2D
from keras.layers import Dense, Input, Flatten
from keras.layers import Concatenate, concatenate, Merge
from keras.layers.core import Reshape, Activation
from keras.optimizers import Adam

I have a custom layer

class KMaxPooling(Layer):
    """
    K-max pooling layer that extracts the k-highest activations from a sequence (2nd dimension).
    TensorFlow backend.
    """
    def __init__(self, k=1, axis=0, **kwargs):
        super(KMaxPooling, self).__init__(**kwargs)
        self.input_spec = InputSpec(ndim=3)
        self.k = k

        assert axis > 0 and axis < 3,  'expected dimensions (samples, filters, convolved_values),\
                                       cannot fold along samples dimension or axis not in list [1,2]'
        self.axis = axis

        # need to switch the axis with the last elemnet 
        # to perform transpose for tok k elements since top_k works in last axis
        self.transpose_perm = [0,1,2] #default
        self.transpose_perm[self.axis] = 2
        self.transpose_perm[2] = self.axis

    def compute_output_shape(self, input_shape):
        input_shape_list = list(input_shape)
        input_shape_list[self.axis] = self.k
        return tuple(input_shape_list)

    def call(self, x):
        # swap sequence dimension to get top k elements along axis=1
        transposed_for_topk = tf.transpose(x, perm=self.transpose_perm)

        # extract top_k, returns two tensors [values, indices]
        top_k = tf.nn.top_k(transposed_for_topk, k=self.k, sorted=True, name=None)[0]

        # return back to normal dimension but now sequence dimension has only k elements
        # performing another transpose will get the tensor back to its original shape
        # but will have k as its axis_1 size
        transposed_back = tf.transpose(top_k, perm=self.transpose_perm)

        return transposed_back

got this error

FailedPreconditionError: Attempting to use uninitialized value conv1d_13/kernel
     [[Node: conv1d_13/kernel/read = Identity[T=DT_FLOAT, _class=["loc:@conv1d_13/kernel"], _device="/job:localhost/replica:0/task:0/gpu:0"](conv1d_13/kernel)]]
     [[Node: k_max_pooling_14/transpose/_117 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_906_k_max_pooling_14/transpose", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'conv1d_13/kernel/read', defined at:
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
    if self.run_code(code, result):
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-40-643b064d16a1>", line 15, in <module>
    Dense(1, activation="sigmoid")
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/models.py", line 401, in __init__
    self.add(layer)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/models.py", line 469, in add
    output_tensor = layer(self.outputs[0])
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/engine/topology.py", line 569, in __call__
    self.build(input_shapes[0])
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/layers/convolutional.py", line 134, in build
    constraint=self.kernel_constraint)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/engine/topology.py", line 391, in add_weight
    weight = K.variable(initializer(shape), dtype=dtype, name=name)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 321, in variable
    v = tf.Variable(value, dtype=_convert_string_dtype(dtype), name=name)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 199, in __init__
    expected_shape=expected_shape)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 330, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1400, in identity
    result = _op_def_lib.apply_op("Identity", input=input, name=name)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/bicepjai/Programs/anaconda2/envs/kagglec2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value conv1d_13/kernel
     [[Node: conv1d_13/kernel/read = Identity[T=DT_FLOAT, _class=["loc:@conv1d_13/kernel"], _device="/job:localhost/replica:0/task:0/gpu:0"](conv1d_13/kernel)]]
     [[Node: k_max_pooling_14/transpose/_117 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_906_k_max_pooling_14/transpose", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

the only solution is

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    model.fit(x_train, y_train, 
          validation_data=(x_test, y_test),
          epochs=4, batch_size=512,
          callbacks=[tb_callback])

when using notebooks this is super annoying

bicepjai on 29 Aug 2017

👍9 ❤5

All 9 comments

It seems to be a bug, add this before fitting, works for me.
keras.backend.get_session().run(tf.initialize_all_variables())

minogame on 9 Dec 2016

👍10

Thank you. It works for me too :)

nateanl on 10 Dec 2016

Seems like Tensorflow is deprecating initialize_all_variables so instead you should use:
keras.backend.get_session().run(tf.global_variables_initializer())

burgalon on 25 Dec 2016

👍10 🎉2 ❤1

Has this bug been fixed yet? I still have this issue.

256lie on 7 Feb 2017

Thanks for the workaround, please fix this bug

bicepjai on 16 Feb 2017

I also encountered a similar issue while following the Keras + Tensorflow tutorial (https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html).

This workaround fixed it - thanks!

dumkar on 17 Feb 2017

Could my issue be related? Batch Normalization doesn't work with mode = 0, but does for mode =2. The proposed solutions above unfortunately don't help

mjs-wpi on 18 Feb 2017