Keras: Can not save model using model.save following multi_gpu_model

[SimonWalsh1000]

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• [x] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

This is a short and simple issue. Following upgrading to Keras 2.0.9 I have been using the multi_gpu_model utility but I can seem to save my models or best weights using model.save.

The error I get is

TypeError: can’t pickle module objects

I suspect there is some problem gaining access to the model object. Is there a work around this issue?

[datumbox]

Are you trying to save directly the model which is parallelized? That can be the problem. Could you provide a snippet that shows the steps you are following?

[SimonWalsh1000]

Yes - thats exactly what I am doing - I'm building the model normally, then calling the multi gpu model method function on it and after training, trying to save it with model.save. Does this need to be converted back to a non-parallelized version of the model? If so, how is this achieved?

[fchollet]

For now we recommend saving the original (template) model instead of the parallel model. I.e. call save on the model you passed to multi_gpu_model, not the model returned by it.

Both models share the same weights.

[SimonWalsh1000]

That's fantastic, and straightforward advice.

Best
Simon

SLFWalsh MD MRCP FFRRCSI
Consultant Radiologist
Kings College Hospital Foundation Trust

On 10 Nov 2017, at 19:09, François Chollet [e-mail] wrote:

For now we recommend saving the original (template) model instead of the parallel model. I.e. call save on the model you passed to multi_gpu_model, not the model returned by it.

Both models share the same weights.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[SimonWalsh1000]

Just to clarify - do you mean call model.save or model.save_weights on the template model at the end of training?

[fchollet]

Yes. Either method should work fine.

[DingkunLiu]

But how can I save the optimizer state by just saving the template model?

[s-zk]

I am on keras 2.1.2 and encountered same problem. I tried follow this answer on StackOverflow, and it works for me. Hope it helps.

[wt-huang]

Closing as this is resolved

[taalbrecht]

Closing as this is resolved

Has this been resolved by a commit and can models and weights be saved as expected when using multi_gpu_model?

[Heisenberg0391]

So, as mentioned above, i should train with parallel_model but save the origin model. But what if i want save weights on every epoch as checkpoints using a callback, what should i do?

[jashshah]

Are there any updates regarding this either on keras or on tf.keras?