Keras: Adding regularizer to an existing layer of a trained model without resetting weights?

[Moondra]

Let's say I'm transfer learning via Inception. I add a few layers and train it for a while.

Here is what my model topology looks like:

base_model = InceptionV3(weights='imagenet', include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu', name = 'Dense_1')(x)
predictions = Dense(12, activation='softmax', name = 'Predictions')(x)
model = Model(input=base_model.input, output=predictions)

I train this model for a while, save it and load it again for retraining; this time I want to add l2-regularizer to the 'Dense_1' without resetting the weights? Is this possible?

path = .\model.hdf5
from keras.models import load_model
model = load_model(path)

The docs show only show the that regularizer can be added as parameter when you initialize a layer:

from keras import regularizers
model.add(Dense(64, input_dim=64,
                kernel_regularizer=regularizers.l2(0.01),
                activity_regularizer=regularizers.l1(0.01)))

[rex-yue-wu]

1. adding a regularizer to a layer does not add extra parameters to this layer
2. adding a regularizer to a layer adds a new loss

Thus, you can't do it in the way you described, but I think you can do it as follows

from keras.applications.inception_v3 import InceptionV3
from keras.layers import GlobalAveragePooling2D, Dense
from keras import regularizers
from keras.models import Model

def create_my_model( use_regularizer ) :
    base_model = InceptionV3(weights='imagenet', include_top=False)
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    if not use_regularizer :
        x = Dense(1024, activation='relu', name = 'Dense_1')(x)
    else :
        x = Dense(1024, activation='relu', name = 'Dense_1',
                  kernel_regularizer=regularizers.l2(0.01),
                  activity_regularizer=regularizers.l1(0.01))(x)
    predictions = Dense(12, activation='softmax', name = 'Predictions')(x)
    model = Model(input=base_model.input, output=predictions)
    return model

# define models w/o and w/ regularizer
model_no_reg = create_my_model( False )
model_reg = create_my_model( True )

# verify these two models have the same number of parameters
print model_no_reg.summary()
print model_reg.summary()

You should see both models have the following parameter stats

Total params: 23,913,260
Trainable params: 23,878,828
Non-trainable params: 34,432

This means you can use one model's weights for the other. For example,

model_reg.set_weights( model_no_reg.get_weights() )

or

model_no_reg_weights = 'tmp.h5'
model_no_reg.save_weights( model_no_reg_weights )
model_reg.load_weights( model_no_reg_weights )

[Moondra]

@rex-yue-wu Thank you! Someone also posted a solution on StackOverflow that seems to work if you save and then load the model.

https://stackoverflow.com/questions/48330137/adding-regularizer-to-an-existing-layer-of-a-trained-model-without-resetting-wei/48330834#48330834

[Pyrestone]

Just because I keep forgetting how it's done and finding this issue on my search, I'm gonna post my solution here:

1. Set the regularizer via layer.kernel_regularizer=... (that doesn't change anything yet)
2. Save the model via model.save("temp_path.h5")
3. load the model via model=load_model("temp_path.h5")

Note: you can also set layer.kernel_regularizer=None in Step 1 to remove a regularizer

This works because the get_config() and from_config() methods are used internally when saving and loading models.