Keras: Make a copy of a model

[snakeztc]

Is there a way to make a deep copy of model? I want to replicate the DQN paper in which you freeze the target model and only after several step of iterations, you copy the temporal model to the target model. Thanks!

[jnlaia]

Early January, I can tell you it was possible to deep copy a keras model with theano backend. However, it was not possible to deep copy one with tensorflow backend. #1466
Is this the problem you are having?

[fchollet]

Why not just transfer the weights to an identical model?

On 22 February 2016 at 06:26, João N. Laia [email protected] wrote:

Early January, I can tell you it was possible to deep copy a keras model
with theano backend. However, it was not possible to deep copy one with
tensorflow backend. #1466 https://github.com/fchollet/keras/issues/1466
Is this the problem you are having?

—
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/1765#issuecomment-187202679.

[snakeztc]

@jnlaia Yeah deep_copy a model give me an error message. @fchollet Thanks. This is the solution I end up with. I created a new instance and copy over the weights.

[VladMironov]

Model's copy would be such a nice feature.

Could save a lot of time on initialization in case of similar models.

[viksit]

@fchollet @snakeztc in the case of creating a new instance from the json file - does this mean model.compile() is run each time, and then you get to a model.load_weights?

[angadgill]

@viksit No need to compile if you're loading from the JSON file. These save and load functions might help: https://github.com/sharathrao13/deep-learning/blob/master/utils.py#L25 and https://github.com/sharathrao13/deep-learning/blob/master/utils.py#L56

[viksit]

@angadgill ah no, I mean that internally, the model.load_weights() function invokes model.compile() [1]. If your model is a complex one, this compile time can be pretty high. Being able to deep copy would make this compile() process only happen on initial load and you can replicate, as @VladMironov mentioned.

@VladMironov have you found any ways to quickly initialize the same model over multiple weight files?

[1] https://github.com/fchollet/keras/blob/master/keras/models.py#L211

[VladMironov]

@viksit I've tried python out-of-box copy.deepcopy with no luck (stopped with an exception).
Thought copy.copy could work, I haven't checked that properly.

Right now I'm simply initializing models (it takes ~7 minutes but I need it only once in a session) and then, if I need to, — I reset weights with model.load_weights

[bzamecnik]

In Keras 2.0.7 there's a new function keras.models.clone_model(model, input_tensors) for that purpose.

[SimonEnsemble]

@bzamecnik how can I get the appropriate format for the input_tensors?
e.g.
model2 = keras.models.clone_model(model, model.get_weights())
does not work.
i.e. I want model2 to be an exact copy of model. Thanks.

[bzamecnik]

@SimonEnsemble input_tensors are tensors for external inputs, not weights (internal model parameters).

[tu-curious]

@bzamecnik I am using clone_model() to create 3 identical branches of a network. The problem is when I concatenate their output layers and build the final model using a terminal dense layer, I get an error saying all the layer names should be unique.
The last layer of 3 clones I made, has the same name. Is there a way to get unique layer names while making model clones?
PS: I am new here and hope that I am not making a mistake by continuing this thread. If I am, please let me know and I will open a new thread for the question.

[stavBodik]

@SimonEnsemble
model_copy = keras.models.clone_model(model)
model_copy.set_weights(model.get_weights())

[TristanJM]

@stavBodik I'm guessing this doesn't clone optimizer state, how would one achieve that?

[stavBodik]

@TristanJM

I don't remmber but copying the model means copying the structure ( layers ) of the model, and the last layer should include the optimizer and its state, so please just test it.

I don't remmber that I had problems with remmbering the optimizer state after copying and setting the weights.

[sachag678]

@stavBodik @TristanJM
I used the following code:
model_copy = keras.models.clone_model(model)
model_copy.set_weights(model.get_weights())
mode_copy.save('model_copy.h5')
keras.models.load_model('model_copy.h5')
I get the following when I load the model -

UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file:

So I don't think the clone_model clones the optimizer, the loss or the optimizer state. Still trying to figure out how to do that.

[mariushegele]

@sachag678 If you save and load the model, you have to recompile. Because compilation resets the models' weights, save them along with the model and load them after compilation:

# save
model_copy.save('model_copy.h5')
model_copy.save_weights('model_copy-weights.h5')

# load
model = keras.models.load_model('model_copy.h5')
model.compile(
   optimizer='rmsprop', 
   loss='sparse_categorical_crossentropy', 
   metrics=['accuracy']
)
model.load_weights('model_copy-weights.h5')

I'm not a hundred percent sure, but I think if you don't save and load the model, re-compilation is not necessary.

[sachag678]

@mariushegele
I agree with you. My issue is I want to save the status of the optimizer, the current loss. i.e if I'm using Adam then the alpha would have been changing over time. I don't want to start from the default value if I load the model and continue training it. I want it to start from the value of alpha when I saved it. My current thought is if I can access those values of the optimizer then I can save them separately and load them with the model and set them myself after recompiling.

[mariushegele]

@sachag678
https://stackoverflow.com/questions/49503748/save-and-load-model-optimizer-state
This might solve your problem.

[nlp4whp]

@sachag678
clone_model() doesn't work for me. I think you might try this like me.

first, write a function to return your Model structure, like model_build()

then, in main(), you could get the basic model by model_build()

when you need a copy model, just create a copy_one by model_build()

and recover weights by copy_one.set_weights(model.get_weights())

hope works for you ~~

[carMartinez]

@mariushegele

Compilation does not reset layer weights.

[mikhail-barg]

@sachag678 thanks for pointing to set_weights.

I had to do this:

model_m1 = keras.models.clone_model(model)
#modify model here if needed
model_m1.build()
model_m1.set_weights(model.get_weights())