Baselines: save_interval in PPO2

[yoojinoh]

Hi!
I am trying to train the atari environment in PPO2, but whenever I activate the save_interval by setting a certain number in learn() function in ppo2.py I encounter an error like below.

'Pickling an AuthenticationString object is ' TypeError: Pickling an AuthenticationString object is disallowed for security reasons

Has anyone encountered this kind of error when saving PPO2 models?

Here is the full error

Traceback (most recent call last):
File "/home/yoojinoh/anaconda3/lib/python3.5/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/yoojinoh/anaconda3/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/yoojinoh/libraries/openai-baselines/baselines/run.py", line 243, in
main()
File "/home/yoojinoh/libraries/openai-baselines/baselines/run.py", line 220, in main
model, _ = train(args, extra_args)
File "/home/yoojinoh/libraries/openai-baselines/baselines/run.py", line 73, in train
alg_kwargs
**File "/home/yoojinoh/libraries/openai-baselines/baselines/ppo2/ppo2.py", line 233, in learn
fh.write(cloudpickle.dumps(make_model))
File "/home/yoojinoh/venvs/gym/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
cp.dump(obj)
File "/home/yoojinoh/venvs/gym/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 408, in dump
self.save(obj)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/venvs/gym/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 408, in save_function
self.save_function_tuple(obj)
File "/home/yoojinoh/venvs/gym/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 559, in save_function_tuple
save(state)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 814, in save_dict
self._batch_setitems(obj.items())
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 840, in _batch_setitems
save(v)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 774, in save_list
self._batch_appends(obj)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 798, in _batch_appends
save(x)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/venvs/gym/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 413, in save_function
self.save_function_tuple(obj)
File "/home/yoojinoh/venvs/gym/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 559, in save_function_tuple
save(state)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 814, in save_dict
self._batch_setitems(obj.items())
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 840, in _batch_setitems
save(v)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 774, in save_list
self._batch_appends(obj)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 798, in _batch_appends
save(x)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 627, in save_reduce
save(state)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 814, in save_dict
self._batch_setitems(obj.items())
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 840, in _batch_setitems
save(v)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 774, in save_list
self._batch_appends(obj)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 798, in _batch_appends
save(x)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 627, in save_reduce
save(state)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 814, in save_dict
self._batch_setitems(obj.items())
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 840, in _batch_setitems
save(v)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 814, in save_dict
self._batch_setitems(obj.items())
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 840, in _batch_setitems
save(v)
File "/home/yoojinoh/anaconda3/lib/python3.5/pickle.py", line 495, in save
rv = reduce(self.proto)
File "/home/yoojinoh/anaconda3/lib/python3.5/multiprocessing/process.py", line 285, in __reduce__
'Pickling an AuthenticationString object is '
TypeError: Pickling an AuthenticationString object is disallowed for security reasons

[simoninithomas]

Hi,

Personally I changed a little bit the PPO2 code and I use instead tf.train.Saver. I don't understand why they use pickle.

I modified the save and load function in the Model object

 def save(save_path):
            """
            Save the model
            """
            saver = tf.train.Saver()
            saver.save(sess, save_path)

def load(load_path):
            """
            Load the model
            """
            saver = tf.train.Saver()
            print('Loading ' + load_path)
            saver.restore(sess, load_path)

And the training part of the learn function

`savepath = "./models/" + str(update) + "/model.ckpt"
model.save(savepath)
print('Saving to', savepath)`

Hope it helps,

[gregorgebhardt]

This is an issue if SubprocVecEnv is used, which uses the multiprocessing package to distribute the sampling on multiple processes. I fixed this and filed a merge request which you can find here: #539

@simoninithomas I think the advantage of using pickle over just saving the parameters is that you will get the whole object back from the pickle and do not have to recreate it.

[wookayin]

Still, pickling for saving parameters is a bad idea.