Keras: "trainable" attribute of a submodel used as layer in a supermodel not preserved by supermodel.save()

[KiraMichiru]

Using Python3, Keras 2.1.5, Tensorflow 1.6.0:

I build a supermodel that has some submodel as layer, and I only set submodel.trainable=False, leaving the trainable flags of the layers contained in submodel unchanged. Then I compile the model using Adam optimizer, train it and save it afterwards with supermodel.save(filename) .
When loading the model with supermodel=load_model(filename), the trainable attribute of the submodel is not preserved (the submodel is shown as trainable in supermodel.summary()). As a consequence, also the Adam optimizer state cannot be restored, because it is expecting a different shape of weights than at saving time.

A workaround is to explicitly set all the layers of the submodel to trainable=False, but I still consider this a bug.

Additionally, if the trainable attribute of the underlying layers changes after model compilation (common in GAN training), not the trainable value at compilation time but the trainable value at saving time will be recorded for the saved model. Again, this behavior is highly unintuitive.

Something similar was already reported in this issue

• [x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[mburaksayici]

Does this has a relation between https://github.com/keras-team/keras/issues/9588 ? Im finetuning and I get very different accuracies for validation and training accuracy, and Im using exactly the same dataset for training and validation.

[KiraMichiru]

Can you elaborate why you think the two issues would be related?

[pbfman]

Ran into the same problem! +1 for this bug

[thoward27]

Can anyone post a solution specific to GANs for this issue?

[ismailsimsek]

solved it by upgrading Keras to 2.2.4 and using pickle. i was having issue with pix2pix model

[SmallStrawHat]

Question added:
If there is only one model file and there is no original model definition code, you can avoid the above error in the following way.

Keras to 2.2.4

mod = load_model('mod.h5', compile=False)

[qmilangowin]

solved it by upgrading Keras to 2.2.4 and using pickle. i was having issue with pix2pix model

I'm still getting this on Keras 2.2.4 (both with load_model or using pickle).
I'm trying to train my model using a pretrained convnet on an AWS EC2 instance with 8 GPUs. Obviously, due to cost I don't want more than I need to and I'd rather be able to dump and save the model for using on my laptop after. But don't seem to be able to do it using the model.save method or pickle as I get the above error when loading it.

Any work-arounds???

[beskyfil]

Was this fixed in latest tf2?