Keras: Issue with Tensorflow's 'switch' op

[claymcleod]

Hi all,

I have the following bit of code that works in Theano but not tensorflow:

def step(x):
    """Step function"""

    return K.switch(x > 0, 1, 0)

where x is just a matrix. As I said before, this runs perfectly as expected in Theano, but when I run this using tensorflow, I get the following error:

File ".../python2/lib/python2.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 584, in assert_same_rank
    "Shapes %s and %s must have the same rank" % (self, other))
ValueError: Shapes (?, 25, 34, 34) and () must have the same rank

Any thoughts on why this might be occurring? I can see that the dimensions aren't matching, but I'm not sure how to get the desired functionality in Tensorflow. I'm just trying to get a matrix of the same dimension as the input where any value greater than 0 is a 1 in the resulting matrix (otherwise, 0).

[claymcleod]

Looks like this solved my error (sloppy but gets the point across):

def step(x):
    """Step function"""

    if (_BACKEND == 'tensorflow'):
        import tensorflow as tf
        return tf.select(tf.python.math_ops.greater(x, 0), K.ones_like(x), K.zeros_like(x))
    else:
        return K.switch(x > 0, 1, 0)

Maybe @fchollet can comment on what the correct way to do this within Keras without a hack would be?

[NasenSpray]

@fchollet I'd propose to follow Theano's definition of switch and to use tf.select instead of tf.cond in the backend. This expects Theano's behavior, btw.

[fchollet]

Yes, that sounds reasonable.

On 21 March 2016 at 09:51, NasenSpray [email protected] wrote:

@fchollet https://github.com/fchollet I'd propose to follow Theano's
definition of switch and to use tf.select instead of tf.cond in the
backend. This
https://github.com/fchollet/keras/blob/master/keras/layers/advanced_activations.py#L189
expects Theano's behavior, btw.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2014#issuecomment-199374333

[pajotarthur]

Tried this solution, but got the following error message at run time.

argument: Number of batchs of 'then' must match size of 'cond', but saw: 1 vs. 200
     [[Node: Select_1 = Select[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Equal_11, Tanh_16, Tile_17)]]

where 1 in the size of my minibatch.
How could it be fixed ?

[Shuailong]

@fchollet In optimizers.py we still have this code:

def clip_norm(g, c, n):
    if c > 0:
        g = K.switch(n >= c, g * c / n, g)
    return g

If I use tensorflow as backend, I have this error:

/home/xiaolong/.local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Traceback (most recent call last):
File "train.py", line 304, in
main()
File "train.py", line 289, in main
nb_val_samples=len(X_dev)//batch_size*batch_size
File "/home/xiaolong/.local/lib/python2.7/site-packages/keras/models.py", line 924, in fit_generator
pickle_safe=pickle_safe)
File "/home/xiaolong/.local/lib/python2.7/site-packages/keras/engine/training.py", line 1401, in fit_generator
self._make_train_function()
File "/home/xiaolong/.local/lib/python2.7/site-packages/keras/engine/training.py", line 713, in _make_train_function
self.total_loss)
File "/home/xiaolong/.local/lib/python2.7/site-packages/keras/optimizers.py", line 134, in get_updates
grads = self.get_gradients(loss, params)
File "/home/xiaolong/.local/lib/python2.7/site-packages/keras/optimizers.py", line 64, in get_gradients
grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
File "/home/xiaolong/.local/lib/python2.7/site-packages/keras/optimizers.py", line 9, in clip_norm
g = K.switch(n >= c, g * c / n, g)
TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'int'

For tensorflow, shall we use tf.select instead of switch?

[Cogitans]

@fchollet To compound the problems listed above. In #2669 we also are having issues implementing Clockwork RNNs in backend-agnostic Keras because of K.switch's use of tf.cond instead of tf.select. Is there an update to the above suggestion of making this switch?

[patyork]

It is always possible to implement a backend-agnostic function via the usage of keras.backend.backend(), which will return either "theano" or "tensorflow" (at this time). This allows you to branch on the backend, even if this is not the cleanest solution.

[griff4692]

Is there an answer on how to solve this. I've got a 2D tensor and I want to set all values below a threshold to 0 but keep getting a similar error (TF backend).

ValueError: Shape must be rank 0 but is rank 2 for 'lambda_3/cond/Switch' (op: 'Switch') with input shapes: [?,?], [?,?].

[logiccodes]

@griff4692 I have met the similar problem and found this issue.
My version of ValueError: Shape must be rank 0 but is rank 2 for 'lambda_3/cond/Switch' (op: 'Switch') with input shapes: [?,?], [?,?] is caused by pred's shape in tf.cond.
Precisely, pred's shape should be () or a tf.bool constant, but I have defined it as (None). In your error, it might be pred's shape was defined in shape (None,None)?

[rex-yue-wu]

I solved this problem by writing a custom accuracy function (assuming target labels are {-1,+1} )

def hinge_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.cast( y_pred > 0, 'float32' ) - K.cast( y_pred <= 0, 'float32' ) ), axis=-1)

[ffa]

tf.where is similar to switch in theano

[mkocabas]

In tensorflow > v1.0 tf.select replaced with tf.where and logical operations can be accessed directly like tf.equal. A sample loss function can be:

def loss(y_true, y_pred):
    import tensorflow as tf
    pt = tf.where(tf.equal(y_true, 1), y_pred, 1 - y_pred)
    return -K.sum(alpha * K.pow(1. - pt, gamma) * K.log(pt))