Keras: Freezing layers

[anjishnu]

There are lots of papers that train neural nets by training a layer, freezing the weights, stacking a layer on top, training etc. How does one implement this with Keras?

e.g.
http://ufldl.stanford.edu/wiki/index.php/Stacked_Autoencoders

[amitbeka]

+1 for this feature

[fchollet]

This is trivial _before compilation_ by setting layer.params and layer.updates to []. This has to be done after layer instantiation but before adding the layer to the model. You could develop a shorthand layer constructor argument: freeze=True for convenience.

It is not possible after compilation.

So what you want to do is create a _new_ network reusing the weights of the previous network (set_weights and get_weights), stack something on top, then freeze the layers you want to freeze via the method above.

PS: do not waste your time reproducing really outdated research like stacked autoencoders... freezing + stacking can have some interesting applications though.

[amitbeka]

Thanks for the tip, I really looked into a way to do this without changing the core keras code.

[jonilaserson]

In caffe you can define a "learning rate multiplier" per layer.
Maybe this can be done in keras? That way if you want to freeze a layer
you can just set its learning rate to 0.

On Thu, Sep 3, 2015 at 1:12 PM, Amit Beka [email protected] wrote:

Thanks for the tip, I really looked into a way to do this without changing
the core keras code.

—
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/622#issuecomment-137400792.

[wuaalb]

I kind of liked Lasagne's way of doing this where each parameter can have a number of binary flags set, e.g. trainable=True. That way you don't have to remove the parameters from the list completely and stuff like saving weights still works.

[mmmikael]

I also need to freeze a few layers in order to fine-tune a network in a multi-stage manner (fine-tune last layers first).

Following previous suggestions, I have added a boolean trainable to the base Layer class which is then checked when adding a layer to a container. Also, to avoid redefining the whole network every time a layer becomes trainable or not, I have added a rebuild function:

class Sequential(Layer):
...
    def add(self, layer):
        self.layers.append(layer)
        if len(self.layers) > 1:
            self.layers[-1].set_previous(self.layers[-2])
            if not hasattr(self.layers[0], 'input'):
                self.set_input()
        layer.init_updates()

        params, regularizers, constraints, updates = layer.get_params()
        if layer.trainable:
            self.params += params
            self.updates += updates

        self.regularizers += regularizers
        self.constraints += constraints

    def rebuild(self):
        layers = self.layers
        self.layers = []
        self.params = []
        self.regularizers = []
        self.constraints = []
        self.updates = []

        for layer in layers:
            self.add(layer)

I made a couple tests on the mnist example and it seems to work.
Do you see any possible issue?

[fchollet]

@mmmikael I believe this should work (at first glance). But you'll need to recompile your network after rebuild every time.

[mmmikael]

@mmmikael I believe this should work (at first glance). But you'll need to recompile your network after rebuild every time.

Yes, I have actually added some code at the end of rebuild to invalidate the compiled Theano functions of the model:

    if hasattr(self, '_train'):
        del self._train
    if hasattr(self, '_train_with_acc'):
        del self._train_with_acc
    if hasattr(self, '_predict'):
        del self._predict
    if hasattr(self, '_test'):
        del self._test
    if hasattr(self, '_test_with_acc'):
        del self._test_with_acc

[mathDR]

Is it possible to only freeze a subset of weights during training?

[mmmikael]

@mathDR you can freeze the weights of specific layers with the trainable argument http://keras.io/layers/core/.