Keras: Is it possible to keep some weights fixed during training

Created on 2 Jun 2016 · 20Comments · Source: keras-team/keras

I want to keep some weights fixed during training the neural network, which means not updating these weights since they are initialized.

''Some weights'' means some values in weight matrices, not specific rows or columns or weight matrix of a specific layer. They can be any element in weight matrices.

Is there a way to do this in Keras? I know Caffe can do this by setting a mask to the weight matrix so the masked weight will not affect the output.

stale

Source

tuming1990

👍9

Most helpful comment

Is it solved? I have the same problem_

SunTongtongtong on 4 May 2018

👍8

All 20 comments

self.non_trainable_weights ?

xingdi-eric-yuan on 2 Jun 2016

I think this doesn't work according to https://github.com/fchollet/keras/issues/2395

tuming1990 on 2 Jun 2016

I think that you can use a custom constraint

sileod on 2 Jun 2016

http://keras.io/getting-started/faq/#how-can-i-freeze-keras-layers may be helpful.

henry0312 on 2 Jun 2016

👍1

            Node(outbound_layer=self,
                 inbound_layers=[],
                 node_indices=[],
                 tensor_indices=[],
                 input_tensors=self.inputs,
                 output_tensors=self.outputs,
                 # no model-level masking for now
                 input_masks=[None for _ in self.inputs],
                 output_masks=[None],
                 input_shapes=[x._keras_shape for x in self.inputs],
                 output_shapes=[self.outputs[0]._keras_shape])

Seems model-level masking is a future plan, right? @fchollet

tuming1990 on 4 Jun 2016

This seems to be a limitation with TensorFlow. The limitation I keep running into is that when the variable is added to the graph, you must define a shape that works for an entire layer. Something like this:

initial = tf.truncated_normal(shape, stddev=0.1, name='W_conv2')
W_conv2 = tf.Variable(initial)

where W_conv2 may have a shape something like [3, 3, 32, 32]. Freezing one of those 3x3 kernels (or one pixel of one of those kernels) while keeping the others trainable is not straightforward. I've got a similar question posted on stackoverflow.

http://stackoverflow.com/questions/42517926/how-to-freeze-lock-weights-of-one-tensorflow-variable-e-g-one-cnn-kernel-of-o

harchankoj on 16 Feb 2017

I think this feature is important. Does anyone find out how to do so?

YuanhaoGong on 22 Mar 2017

👍2

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

stale[bot] on 21 Jun 2017

👎15

Still having a similar issue. Any ideas?

caugusta on 16 Jan 2018

This is one reason that implementation of neural networks within hardware or firmware continues to be an issue: network pruning is very limited. It's unfortunate stale bot intervened while the real issue remains open. Or perhaps it's been solved and not shared.

harchankoj on 17 Jan 2018

👍5 😄1

Maybe the tf.boolean_mask could do this trick.

maxtomCMU on 22 Mar 2018

Is it solved? I have the same problem_

SunTongtongtong on 4 May 2018

👍8

how to realize by caffe

theforthforgotten on 6 Nov 2018

@tuming1990

theforthforgotten on 6 Nov 2018

http://keras.io/getting-started/faq/#how-can-i-freeze-keras-layers may be helpful.

This may well be exactly what you need.

It certainly is exactly what I needed.

otdewiljes on 7 Jun 2019

👎1

http://keras.io/getting-started/faq/#how-can-i-freeze-keras-layers may be helpful.

This may well be exactly what you need.

It certainly is exactly what I needed.

Actually @tuming1990 asked for fine grained pruning, the link provided by @henry0312 is about coarse grained pruning. It is only possible to freeze the layers with the trainable parameter. However the author clearly asked for a way to freeze any weight in the tensors.

Does anybody have an update about element-wise weight freezing ?

N0ciple on 26 Jul 2019

👍2

Is this solved? It is very handy to keep some weights (not necessarily the whole layer) fixed.

fqassemi on 15 Feb 2020

👍1

Any update? I am looking to do this as well. At the very least, I'd like to set certain weights in a layer to be what I want; not all the weights of a layer or in a model.

ryanmaxwell96 on 8 Jul 2020

👍1

If anyone is still watching this, I have two classes (an initializer and a constraint) which may solve this problem. I have only tested them on my use case so take them with a grain of salt. They allow for initializing any slice of a given weight tensor to chosen values (FixSlice) and freezing any slice of a given weight tensor to chosen values (FreezeSlice).

from keras.initializers import Initializer

class FixSlice(Initializer):
    """
    Initializer which forces a certain slice to be chosen values

    INPUTS:

    values - An object which can be converted into a numpy ndarray. These are
             the pre-chosen values. When using this initializer, the user should
             ensure that the dtype of values can be converted to the desired
             dtype of the weight tensor.

    slice - A slice or tuple of slices (it is recommended to use numpy.s_ to
            specify this parameter). This specifies which entries should be
            filled with the pre-chosen values. When using this initializer,
            the user should ensure that the slice object "fits inside" the shape
            of the tensor to be initialized, and that the resulting slice of the
            tensor has the same shape as the values ndarray.

    backup - An initializer instance. The remaining values are filled using this
             initializer.
    """
    def __init__(self, values, slice, backup="glorot_uniform"):
        if hasattr(values, "numpy"):
            self.values = values.numpy()
        elif isinstance(values, np.ndarray):
            self.values = values
        else:
            try:
                self.values = values.to_numpy()
            except:
                self.values = np.array(values)

        self.values = values
        self.slice = slice
        self.backup = initializers.get(backup)

    def __call__(self, shape, dtype=None):
        result = self.backup(shape, dtype=dtype).numpy()
        result[self.slice] = self.values
        return tf.Variable(result)

from keras.constraints import Constraint

class FreezeSlice(Constraint):
    """
    Constraint which keeps a certain slice frozen at chosen values

    INPUTS:

    values - An object which can be converted into a numpy ndarray. These are
             the pre-chosen values. When using this constraint, the user should
             ensure that the dtype of values can be converted to the desired
             dtype of the weight tensor.

    slice - A slice or tuple of slices (it is recommended to use numpy.s_ to
            specify this parameter). This specifies which entries should be
            filled with the pre-chosen values. When using this initializer,
            the user should ensure that the slice object "fits inside" the shape
            of the tensor to be initialized, and that the resulting slice of the
            tensor has the same shape as the values ndarray.
    """
    def __init__(self, values, slice):
        if hasattr(values, "numpy"):
            self.values = values.numpy()
        elif isinstance(values, np.ndarray):
            self.values = values
        else:
            try:
                self.values = values.to_numpy()
            except:
                self.values = np.array(values)

        self.values = values
        self.slice = slice

    def __call__(self, w):
        zs = np.zeros(w.shape)
        zs[self.slice] = self.values
        os = np.ones(w.shape)
        os[self.slice] = 0
        return w * os + zs

Doekeb on 1 Oct 2020

from keras.initializers import Initializer

class FixSlice(Initializer):
    """
    Initializer which forces a certain slice to be chosen values

    INPUTS:

    values - An object which can be converted into a numpy ndarray. These are
             the pre-chosen values. When using this initializer, the user should
             ensure that the dtype of values can be converted to the desired
             dtype of the weight tensor.

    slice - A slice or tuple of slices (it is recommended to use numpy.s_ to
            specify this parameter). This specifies which entries should be
            filled with the pre-chosen values. When using this initializer,
            the user should ensure that the slice object "fits inside" the shape
            of the tensor to be initialized, and that the resulting slice of the
            tensor has the same shape as the values ndarray.

    backup - An initializer instance. The remaining values are filled using this
             initializer.
    """
    def __init__(self, values, slice, backup="glorot_uniform"):
        if hasattr(values, "numpy"):
            self.values = values.numpy()
        elif isinstance(values, np.ndarray):
            self.values = values
        else:
            try:
                self.values = values.to_numpy()
            except:
                self.values = np.array(values)

        self.values = values
        self.slice = slice
        self.backup = initializers.get(backup)

    def __call__(self, shape, dtype=None):
        result = self.backup(shape, dtype=dtype).numpy()
        result[self.slice] = self.values
        return tf.Variable(result)

from keras.constraints import Constraint

class FreezeSlice(Constraint):
    """
    Constraint which keeps a certain slice frozen at chosen values

    INPUTS:

    values - An object which can be converted into a numpy ndarray. These are
             the pre-chosen values. When using this constraint, the user should
             ensure that the dtype of values can be converted to the desired
             dtype of the weight tensor.

    slice - A slice or tuple of slices (it is recommended to use numpy.s_ to
            specify this parameter). This specifies which entries should be
            filled with the pre-chosen values. When using this initializer,
            the user should ensure that the slice object "fits inside" the shape
            of the tensor to be initialized, and that the resulting slice of the
            tensor has the same shape as the values ndarray.
    """
    def __init__(self, values, slice):
        if hasattr(values, "numpy"):
            self.values = values.numpy()
        elif isinstance(values, np.ndarray):
            self.values = values
        else:
            try:
                self.values = values.to_numpy()
            except:
                self.values = np.array(values)

        self.values = values
        self.slice = slice

    def __call__(self, w):
        zs = np.zeros(w.shape)
        zs[self.slice] = self.values
        os = np.ones(w.shape)
        os[self.slice] = 0
        return w * os + zs

Can you provide specific example of it's application?