Keras: k_gradients question for layer is model

[fr407041]

library(keras)
conv_base <- application_vgg16(
  weights = "imagenet",
  include_top = FALSE,
  input_shape = c(150, 150, 3)
)

model <- keras_model_sequential() %>% 
  conv_base %>% 
  layer_flatten() %>% 
  layer_dense(units = 256, activation = "relu") %>% 
  layer_dense(units = 8  , activation = "softmax")

model_output <- model$output[, 1]
last_conv_layer <- model$layers[[1]] %>% get_layer("block5_conv3")
k_gradients( model_output, last_conv_layer$output)

return is below :
[[1]]
NULL

My question how can I make k_gradient link model_output & last_conv_layer$output without return NULL?

I have tried another way to test how k_gradient work, plz see below

model_output <- model$output[, 1]
last_conv_layer <- model$layers[[3]] 
k_gradients( model_output, last_conv_layer$output)

return is below :
[[1]]
Tensor("gradients_42/dense_20/MatMul_grad/MatMul:0", shape=(?, 256), dtype=float32)

I guess "model$layers[[1]] %>% get_layer("block5_conv3")" maybe not belong the same graph with "model$output[, 1]".

How can I achieve my goal to gradient "model$layers[[1]] %>% get_layer("block5_conv3")" by "model$output[, 1]"?

[skeydan]

Not sure exactly, but I'd suspect it has to do with the first "layer" here really being a model:

> model
Model
________________________________________________________________
Layer (type)                Output Shape              Param #   
================================================================
vgg16 (Model)               (None, 4, 4, 512)         14714688  
________________________________________________________________
flatten_2 (Flatten)         (None, 8192)              0         
________________________________________________________________
dense_3 (Dense)             (None, 256)               2097408   
________________________________________________________________
dense_4 (Dense)             (None, 8)                 2056      
================================================================
Total params: 16,814,152
Trainable params: 16,814,152
Non-trainable params: 0

It would be very interesting to know if the same happens when using Python...

As a workaround, would it help you to use trainable_weights instead of outputs?

> last_conv_layer$trainable_weights
[[1]]
Variable(shape=(3, 3, 512, 512), dtype=float32_ref)


[[2]]
Variable(shape=(512,), dtype=float32_ref)

The difference would be that in one case, you'd be computing the gradient w.r.t. the relu, in the other case, the conv filter (and bias).

[fr407041]

Hi skeydan , thank for your reply. But the issue is the output of last_conv_layer$output is not connected to model_output in the k_gradient. I don't realize why use trainable_weights instead can solve this issue?

[skeydan]

How do you mean not connected? There is just one graph there...

I suspect it may be a Keras or TensorFlow bug. See

https://github.com/keras-team/keras/issues/9992

https://stackoverflow.com/questions/49834380/k-gradientsloss-input-img0-return-none-keras-cnn-visualization-with-ten

which pretty much look like the same thing - you can get gradients w.r.t. layers you've added on but not to layers belonging to the pretrained model imported...

[skeydan]

OK trying to get to the root of this: Opened

https://github.com/keras-team/keras/issues/10716

in Keras. To find out if this is expected behavior or a bug.

[fr407041]

skeydan Thanks!!! . It seems to still have no solution currently. Thank you for your reply.

[skeydan]

OK, so if you switch to the functional API this should work :-)

You can see 2 different working examples in

https://github.com/keras-team/keras/issues/10716

because I wrote up the workaround I had found in about the same minute as the issue was reopened & answered from Keras side ;-)

Let me know if you have any problems.

[fr407041]

It works!!!!! Thank you for your reply!!!

Below is my working R code and I hope it would help somebody like me before.

img_input      <- layer_input(shape = list(150, 150, 3), name = "input")

conv_base <- application_vgg16(
  weights = "imagenet",
  include_top = FALSE,
  input_tensor = img_input
)

pred <- conv_base$output %>% 
  layer_flatten() %>% 
  layer_dense(units = 256, activation = "relu") %>% 
  layer_dense(units = 8  , activation = "softmax")

model <- keras_model( img_input , pred )

model_output <- model$output[, 1]
last_conv_layer <- model %>% get_layer("block5_conv3")
k_gradients( model_output, last_conv_layer$output)

skeydan , thank for your great help again.

[skeydan]

Great, happy to hear that! And thanks for posting the code!