Keras: How to figure out if Keras is loading the weights "correctly": model.load_weights(filename, by_name=True)

[dc-ai]

I know that if we use the by_name=True option in model.load_weights, it load the weights for the layers with matching name as the saved model. Obviously, my current architecture and saved architecture is not quite the same but share similar layers.

The issue I am running into, is that I can't figure out which layers have their weights reloaded, and I also can't figure out an easy manner to "check" that the loaded weights are correct. This is important because if I accidentally made a typo in my new architecture, an internal check that tells me which layers have reloaded weights, would be a very helpful sanity check.

I know how to print out all of the weights:

for layer in model.layers:
    weights = layer.get_weights()

But that prints out everything. I also tried summing the entire list of weights, but that doesn't always work (depends on the layer type).

[k0suke-murakami]

I have the exact same question here.

[Chandrak1907]

Does this issue pertains only to by_name=True?
I found that even model.load_weights(filename) does not give an error, if new model is different from saved model. gist file of my working is here.

[fchollet]

In general I would recommend against use by_name, it is error prone (same
as TF checkpoints, which are name-based). Load your load model, then
transfer its weights to the new model layer by layer using
get_weights/set_weights

I found that even for model.load_weights(filename) does not give an
error, if new model is different from saved model.

Weights loading skips layers that have no weights, so many different models
can be compatible with the same set of weights. If any weight could not be
loaded, or if the target model has weights that no match in the weights
set, an error is always raised.

On 12 March 2017 at 19:32, Chandrak1907 [e-mail] wrote:

Does this issue pertains only to by_name=True?
I found that even for model.load_weights(filename) does not give an error,
if new model is different from saved model. gist file of my working is
here.
https://gist.github.com/Chandrak1907/2e1ec3c2c07d734802d28a22f2c16b7a

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/5397#issuecomment-286001761,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArWb-Tua1Ti0Xg-3RkPphVeA8yrWQW2ks5rlKrCgaJpZM4MA94v
.

[farahanams]

how to get_weights from previous built model?
autoencoder = Model.get_weights('autoencoder_model.h5')

it gives an error;
AttributeError: 'str' object has no attribute 'layers'.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[tjdurant]

can someone give an example of fchollet's answer here? I have same question as OP and I just want to make sure I am understanding the answer correctly.

[ljames1]

@tjdurant I believe fchollet is reccomending somthing like this:

new_layers = ['flatten_1 ', 'fc6', 'dropout_1', 'fc7', 'dropout_2', 'fc8']
for i in range(len(model.layers)):
  if model.layers[i].name not in new_layers:
  model.layers[i].set_weights(model_pretrained.layers[i].get_weights())

The problem that I have with this is that tensorboard shows remnants of the original model in a weird jumbled mess. When displaying the model with model.summary(), the model always looks as intended. Unfortunately, this is an ongoing issue #6229 #2371

[sagnibak]

@ljames1 @fchollet If we have a weights file instead of a model file, how can we load the weights layer-by-layer as shown by ljames1 and suggested by fchollet?

[ljames1]

@sagnibak If you are generating the weights file, then you can always just save the full model instead of just the weights. If you are using a weights file generated by someone else, you can load their model and weights into a clone of their model, and then save the model that way. Saving the full model adds so much convenience and makes little difference in the total amount of storage consumed, so I would always recommend going that route.

[kangskii]

Hello,

Is there a formal method to solve the issue from @dc-ai ?

[kvnptl]

@tjdurant I believe fchollet is reccomending somthing like this:

new_layers = ['flatten_1 ', 'fc6', 'dropout_1', 'fc7', 'dropout_2', 'fc8']
for i in range(len(model.layers)):
  if model.layers[i].name not in new_layers:
  model.layers[i].set_weights(model_pretrained.layers[i].get_weights())

The problem that I have with this is that tensorboard shows remnants of the original model in a weird jumbled mess. When displaying the model with model.summary(), the model always looks as intended. Unfortunately, this is an ongoing issue #6229 #2371

It worked for me, thanks @fchollet @ljames1

[cuevas1208]

another way to do it:

    # store weights before loading pre-trained weights
    preloaded_layers = self.model.layers.copy()
    preloaded_weights = []
    for pre in preloaded_layers:
        preloaded_weights.append(pre.get_weights())

    # load pre-trained weights
    self.model.load_weights(filepath, by_name=True)

    # compare previews weights vs loaded weights
    for layer, pre in zip(self.model.layers, preloaded_weights):
        weights = layer.get_weights()

        if weights:
            if np.array_equal(weights, pre):
                print('not loaded', layer.name)
            else:
                print('loaded', layer.name)

[Danmou]

A correction to @cuevas1208's solution: Layer.get_weights() returns a list of weight matrices, but np.array_equal only works for individual arrays (on lists of arrays it always returns False). This should work in general:

initial_weights = [layer.get_weights() for layer in model.layers]
model.load_weights(checkpoint, by_name=True)
for layer, initial in zip(model.layers, initial_weights):
    weights = layer.get_weights()
    if weights and all(tf.nest.map_structure(np.array_equal, weights, initial)):
        print(f'Checkpoint contained no weights for layer {layer.name}!')