Keras: clipnorm doesn't work with Embedding

[Vladimir-Yashin]

I'm getting a Traceback every time "clipnorm" is used in NN with Embedding layer.
Here is a simple script where the problem is obvious:

import numpy as np
from keras.layers import Input, Embedding
from keras.optimizers import Adam
from keras.models import Model

input_layer = Input(shape = (1,) )

embedding = Embedding(input_dim = 1,
                      output_dim = 1)(input_layer)

model = Model(input = input_layer, output = embedding)

model.compile(optimizer = Adam(clipnorm = 1.0), loss = 'mse')

X = np.array([[1]])
Y = np.array([[[0.5]]])
model.fit(X, Y, nb_epoch = 1)

Failure:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0)
Traceback (most recent call last):
  File "./clipnorm-bug.py", line 20, in <module>
    model.fit(X, Y, nb_epoch = 1)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1079, in fit
    self._make_train_function()
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 696, in _make_train_function
    self.total_loss)
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 379, in get_updates
    grads = self.get_gradients(loss, params)
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in get_gradients
    grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in <listcomp>
    grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 9, in clip_norm
    g = K.switch(n >= c, g * c / n, g)
TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'float'

Keras version is 1.1.0, TensorFlow is 0.10rc

clipvalue on the other hand works fine.

[kracwarlock]

I have the same issue

[kracwarlock]

The way tensorflow handles scalar multiplication with IndexedSlices is: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/math_ops.py#L458-L467

@fchollet Should we convert the scalar to tensor

• here or,
• move clip_norm to the backend from https://github.com/fchollet/keras/blob/master/keras/optimizers.py and convert it there

I can make a PR on the desired behavior

[MechCoder]

I got bit by this as well. Would be great if someone could fix it!

[jphalip]

I too just ran into this issue with clipnorm and embeddings. I also confirm that clipvalue does work.

[PiranjaF]

@fchollet - This seems like a serious issue regarding clipnorm as it's very useful for training RNN's, which often employs an Embedding layer. Could you give some guidance on how to fix this issue?

[PiranjaF]

@kracwarlock Did you manage to find a solution?

[kracwarlock]

I think I switched back to keras/tf versions where this was working
my project does not need this anymore but will try to write a fix if I get time

[kracwarlock]

@PiranjaF you can try out the branch I have PR'd above (https://github.com/kracwarlock/keras/tree/fix-3859)
it works for me

[PiranjaF]

Thanks, it works!

[kracwarlock]

I made some changes to the PR. Can you re-check? The latest one also works for me locally.

[nelson-liu]

+1, I also got bit by this as well. @kracwarlock i'll try out your patch in a bit and let you know if it works on my setup, thanks for contributing.

EDIT: it works for me, thanks!

[PiranjaF]

@kracwarlock Sorry about the delay, I didn't have a free GPU for testing. The latest one works for me as well.

[maremoto]

Hi,
When will this patch be included in the main Keras branch and the pip release?
Thanks a lot.

[ViktorM]

What is a current status of this bug? Clip norm is very important for training RNN's and this bug kills the opportunity of using it in models with RNN's and embedded layer.

[nelson-liu]

agreed. has anyone been able to check if the bug is present in tf.contrib.keras (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/keras)?