Keras: How to get trainable weights?

[ritchieng]

Because I'm manually running a session, I can't seem to collect the trainable weights of a specific layer.

# Keras layers can be called on TensorFlow tensors:
        x = Convolution2D(16, 3, 3, init='he_normal', border_mode='same')(img)

        for i in range(0, self.blocks_per_group):
            nb_filters = 16 * self.widening_factor
            x = residual_block(x, nb_filters=nb_filters, subsample_factor=1)

        for i in range(0, self.blocks_per_group):
            nb_filters = 32 * self.widening_factor
            if i == 0:
                subsample_factor = 2
            else:
                subsample_factor = 1
            x = residual_block(x, nb_filters=nb_filters, subsample_factor=subsample_factor)

        for i in range(0, self.blocks_per_group):
            nb_filters = 64 * self.widening_factor
            if i == 0:
                subsample_factor = 2
            else:
                subsample_factor = 1
            x = residual_block(x, nb_filters=nb_filters, subsample_factor=subsample_factor)

        x = BatchNormalization(axis=3)(x)
        x = Activation('relu')(x)
        x = AveragePooling2D(pool_size=(8, 8), strides=None, border_mode='valid')(x)
        x = tf.reshape(x, [-1, np.prod(x.get_shape()[1:].as_list())])

        # Readout layer
        preds = Dense(self.nb_classes, activation='softmax')(x)

       loss = tf.reduce_mean(categorical_crossentropy(labels, preds))

        optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

        with sess.as_default():

            for i in range(10):

                batch = self.next_batch(self.batch_num)
                _, l = sess.run([optimizer, loss],
                                feed_dict={img: batch[0], labels: batch[1]})
                print(l)
                print(type(weights))

I'm trying to get the weights of the last convolution layer.

I tried get_trainable_weights(layer) and layer.get_weights()but I did not manage to get anywhere.

The error

AttributeError: 'Tensor' object has no attribute 'trainable_weights'

[fchollet]

model.trainable_weights is the list of trainable weights of a model. Of course you should first define a model in that case.

You can also retrieve that attribute separately on every layer (layer.trainable_weights).

[ritchieng]

@fchollet If I were to follow your guide and integrate with my TensorFlow workflow (https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html) as with others, you cannot access the weight variable because we won't be building the model as shown in your guide. We're merely using the layers. There is no need to compile when we use it as a simplified interface to TensorFlow. How then do we access the weights?

Because if we use with TensorFlow like the guide, we do not call Model or Compile but merely use the layers to build.

[asanakoy]

@ritchieng
Have you found the way?
Seems like the API used in the Blog post is not working any more

[cbdbdd]

you can get weights of a layer by model.layer[i].W at run time. this is tested under theano backend.

[Faur]

+1, I have the same issue as OP (using TF)

[ltrottier]

Did you try this ?

import keras.backend as K
K.get_session().run(model.trainable_weights)

[KFDing]

Do remember that layer.get_weights() is for a layer object in keras, if you add input for that layer you would get one Tensor(number not a object). To change that, use one temporary variable to store layer, like this:
tmp = Dense()
tmp.get_weights()
output = tmp(x)

It works for me. Hope I clear it.

[nbro]

@fchollet If I attempt to access the attribute trainable_weights of a layer of a model

x_in = Input(shape=(...))
h1 = MyLayer1(...)(x_in)
...
print(h1.trainable_weights)

I get the error

AttributeError: 'Tensor' object has no attribute 'trainable_weights'

with TensorFlow 2.0.0 and Keras 2.3.1.

However, you can access the weights of the layers through the model

print(model.layers[0].trainable_weights)
print(model.layers[1].trainable_weights)
...

[Kh4n]

@nbro The code you wrote is slightly incorrect I believe. Try this:

x_in = Input(shape=(...))
h1 = MyLayer1(...)
x = h1(x_in)
...
print(h1.trainable_weights)

In your code, h1 was not the actual layer, but its output, which is a Tensor. That is why you recieved the error

[nbro]

@Kh4n No, my code is correct.

x_in = Input(shape=(...))
h1 = MyLayer1(...)(x_in)
...
print(h1.trainable_weights)

is the usual way of creating a model in Keras with the functional API. Have a look at the documentation.

[Kh4n]

@nbro I believe there is a misunderstanding. I did not mean your code is incorrect insofar as the model layers are concerned. I agree that you showed the typical way in which you create a keras functional model. However, if you need to get the trainable weights for a specific layer, you cannot build the model in the usual way.

x_in = Input(...)
h1 = MyLayer1(...)(x_in)
h2 = Dense(...)(h1)
...

x_in = Input(...)
h1 = MyLayer1(...)
h1_out = h1(x_in)
h2 = Dense(...)(h1_out)
...



md5-4f413ecef05d8364a57ba86004227a86



h1 = MyLayer(...)
h2 = MyLayer(...)(...)



md5-b4bbb62038530f871fcd6a31bd3abbdd



h1_out = h1(...)

to actually get the output of the layer. I suggest you try my code and see whether or not it works correctly. I have confirmed that constructing the model this way does indeed let you get the weights via .trainable_weights and get_weights() (the latter of which is preferred as it will avoid pitfalls of modifying weights within tf.function or compiled graph code).

[Ghlano]

Do remember that layer.get_weights() is for a layer object in keras, if you add input for that layer you would get one Tensor(number not a object). To change that, use one temporary variable to store layer, like this:
tmp = Dense()
tmp.get_weights()
output = tmp(x)

It works for me. Hope I clear it.

hi there, I change my code to this:
dna_tmp = keras.layers.Attention('softmax',name='dna_attention_layer')
att_weights = dna_tmp.get_weights()
dna_attention_layer = dna_tmp([dna_bigru, dna_bigru])
np_array = np.array(att_weights).reshape(1, 20)
but when I run the code , it says that: Can not be reshaped because att_weights' length is 0. is it means that I do not get the weights?