Baselines: Model Save/Load implementation

[huvar]

Hello,
What is the purpose of changing the implementation of this from the native tf Saver (save_state) to save_variables?
Answering the TODO comment, some things are left out... Optimizer parameters, layer normalization... I guess it would be better to rollback and use the save_state/load_state instead.

[hellandhansen]

I'm also wondering about the save format. A normal TensorFlow model would have 4 components:
".meta" files: containing the graph structure
".data" files: containing the values of variables
".index" files: identifying the checkpoint
"checkpoint" file: a protocol buffer with a list of recent checkpoints

However, the Baselines save is only one file. As I'm interested in testing the trained network on new data, and perhaps use it, it would be great to find out how to save the TF file the "ordinary" way, or to "decode" the Baselines save, so that I can use the finished, trained NN independent of OpenAI code, EG to control a robot.

Perhaps it isn't that hard? The Saver class provided by the TensorFlow library is the recommended way for saving the graph's structure and variables. As any Baselines algo basically sets up at TF NN network, where in the code could I insert and call an instance of Saver?

Thanks to all the committed people who contribute to the codebase!

[simoninithomas]

Hi @hellandhansen

Personally I changed a little bit the PPO2 code and I use instead tf.train.Saver.

I modified the save and load function in the Model object

```
def save(save_path):
"""
Save the model
"""
saver = tf.train.Saver()
saver.save(sess, save_path)

def load(load_path):
"""
Load the model
"""
saver = tf.train.Saver()
print('Loading ' + load_path)
saver.restore(sess, load_path)

And the training part of the learn function

savepath = "./models/" + str(update) + "/model.ckpt" model.save(savepath) print('Saving to', savepath)
```
If you need the complete implementation check here https://github.com/simoninithomas/Deep_reinforcement_learning_Course/tree/master/PPO%20with%20Sonic%20the%20Hedgehog
Hope it helps,

[Atcold]

Oh, LOL. I've just figured that by default it does not save any model.
Weird-ish default, no? Wouldn't it make sense to save the trained model with the monitor.csv, progress.csv, and log.txt?

[pzhokhov]

in baselines.common.tf_util there are (in a sense interchangeable) two pairs of serialization methods:
save_state / load_state, and save_variables / load_variables. The first pair, save_state / load_state is a wrapper for tf.Saver and does basically what @simoninithomas has outlined; storing all the usual tensorflow checkpoint files.
The second pair (save_variables / load_variables) serializes variables by creating dictionary of variable name -> numpy array with variable value, then joblib-dumps that (@hellandhansen, I hope that answers your question). @huvar thanks for pointing out the issue with non-trainable variables (hence the bug label)!
Ideally, we'd like to move to saving only variables and their values for the reasons of interoperability between model versions - if you have changed the tf graph ever so slightly, trying to load an existing checkpoint in it can become a nightmare; loading values from a dict is at most a matter of renaming. Also, this way models trained in tensorflow can be used, at least in principle, in other frameworks (like pytorch). This ties into @Atcold point - I think the core serializer should save only the part that's relevant to the reuse of the model (i.e. weights), in a format as simple as possible; however, it makes sense to have a way to save the model also with all the training info and other things. In fact, it would be pretty cool to have a loader that automatically creates the object of the right class (ppo2.Model, a2c.Model or deepq.Model, for instance), remembers all the training info and can either resume training, or do test/render, or return weights. One day we'll have that too :) (hence the enhancement label)

[srivatsankrishnan]

@simoninithomas
are you able to load the model in *.ckpt format?
saver = tf.train.Saver()
_File "/home/trinity/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__ self.build() File "/home/trinity/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1347, in build self._build(self._filename, build_save=True, build_restore=True) File "/home/trinity/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1372, in _build raise ValueError("No variables to save") ValueError: No variables to save_
I am getting an error when I load the model saved in *.ckpt format. Any help is appreciated.

[ThomasZav]

I'm also wondering about the save format. A normal TensorFlow model would have 4 components:
".meta" files: containing the graph structure
".data" files: containing the values of variables
".index" files: identifying the checkpoint
"checkpoint" file: a protocol buffer with a list of recent checkpoints

However, the Baselines save is only one file. As I'm interested in testing the trained network on new data, and perhaps use it, it would be great to find out how to save the TF file the "ordinary" way, or to "decode" the Baselines save, so that I can use the finished, trained NN independent of OpenAI code, EG to control a robot.

Perhaps it isn't that hard? The Saver class provided by the TensorFlow library is the recommended way for saving the graph's structure and variables. As any Baselines algo basically sets up at TF NN network, where in the code could I insert and call an instance of Saver?

Thanks to all the committed people who contribute to the codebase!

Hello,

Have you figured it out how to output as a normal tensorflow model?