Baselines: Unable to reproduce HER results from Plappert et al., 2018

[sanjeevanahilan]

I ran the code baselines.her.experiment.train for five different seeds and generated plots with some minor changes to the plotting code (see issue #311) for HandManipulateBlockRotateXYZ-v0 across 200 epochs. I verified from the .json file that the 'scope' was 'DDPG' and that the rewards were sparse.

Unfortunately the performance of the algorithm was substantially worse than I found in the paper (above are my results and below are the results from Plappert et al., 2018 https://arxiv.org/abs/1802.09464):

Any idea why the results are so much worse? Further, this issue seems to apply to multiple other cases including 'FetchSlide' and 'HandManipulateBlockRotateZ' although I have only run one seed for these.

In case its useful, I also copy the details of the 'params.json' file:

{"pi_lr": 0.001, "network_class": "baselines.her.actor_critic:ActorCritic", "norm_clip": 5, "polyak": 0.95, "scope": "ddpg", "n_cycles": 50, "random_eps": 0.3, "env_name": "HandManipulateBlockRotateXYZ-v0", "rollout_batch_size": 2, "n_batches": 40, "layers": 3, "buffer_size": 1000000, "action_l2": 1.0, "Q_lr": 0.001, "clip_obs": 200.0, "hidden": 256, "test_with_polyak": false, "batch_size": 256, "noise_eps": 0.2, "n_test_rollouts": 10, "relative_goals": false, "norm_eps": 0.01, "max_u": 1.0, "replay_strategy": "future", "replay_k": 4}

And of course, I'd like to thank OpenAI for making their code and environments available; it's really helpful to independent researchers!

[sanjeevanahilan]

UPDATE:

So I decided to test the code using the optional argument --num_cpu 16, whereas previously I had not used this. With this change I now seem to get plots in line with Plappert et al., which is good to see.

However, the fact that this optional argument changes the results seems strange; using more cpu cores should simply affect the speed at which the result is acquired, not the result itself. The x-axis should represent the agent's total experience with the environment in an unbiased way, one which is independent of num_cpu (I would have thought).

So given this issue are we able to have greater clarity as to what constitutes an epoch, and whether this should really depend on num_cpu?

[matthiasplappert]

So to clear this up:

We define an epoch in terms of updates to the actor/critic and not in terms of simulation steps. You could also define it in terms of simulation steps, but that's just not how we decided to do it.

So you are completely right that you need to either use more CPUs to get comparable results (we used --num_cpu 19 for all experiments on a 20 core machine) or you need to let the algorithm run for much longer. However, I would expect that just letting it run for longer still gives suboptimal results since we found that having a much larger batch size helps a lot (the batch size scales linear with num_cpu in our implementation since each MPI worker computes the gradient for a batch size of 256 and we then average those together, making the effective batch size num_cpu * 256).

I agree, however, that this is confusing if you want to reproduce our result. The reason why we use num_cpu = 1 by default is that it would simply be a pretty terrible idea to run it with much more on a less powerful machine. I'll add a warning that informs the user that, to reproduce the result, a different setting should be used.

[matthiasplappert]

Addressed in https://github.com/openai/baselines/pull/319

[sanjeevanahilan]

Great, thanks for clarifying this.

[JoyChopra1298]

@matthiasplappert Does that mean if we double the batch_size on half as much cores than we might get similar results?

[matthiasplappert]

Possibly but you will still have less experience being generated. You can also increase the number of parallel rollouts per MPI worker, so I'd look into that.

[DanielTakeshi]

@sanjeevanahilan When you ran with 16 CPUs, what kind of computer did you run it on? Specifically, did you have more than one GPU, now many CPUs on the machine, and how many processors? Thanks.

[sanjeevanahilan]

@DanielTakeshi I ran it using the cpu version of tensorflow on an AMD Threadripper 1950x with 16 cores and 32 threads. Although I do have gpus I don't think they are meant to be used with this code given no mention in the paper (and also when I ran tensorflow-gpu in conjunction with num_cpu = 16 it threw an error but I didn't investigate further).

[DanielTakeshi]

@sanjeevanahilan Ah, thanks, maybe that helps to explain. I wonder if OpenAI used GPUs for this code as well.

[sanjeevanahilan]

@DanielTakeshi I think ultimately many applications will use hybrid CPU/GPU implementations for deep reinforcement learning. There has been some work in this direction:

https://openreview.net/pdf?id=r1VGvBcxl
https://github.com/NVlabs/GA3C

But for this particular implementation OpenAI are not using large deep nets (unlike say for visual processing of pixels) and so I might speculate that the speedup provided by a gpu would be limited.

[matthiasplappert]

We do not use any GPUs for our experiments since, as mentioned, our networks are relatively small and getting more experience with parallel simulations proves more important.

[ruizhaogit]

Hi guys @matthiasplappert @sanjeevanahilan ,

I tried to run the script with --num_cpu 19, but it still cannot reproduce the results from Plappert et al., 2018, in the HandManipulateBlockRotateXYZ-v0 environment. While the results in HandManipulateEggFull-v0 get in line with the paper Plappert et al.

The command I use is: python baselines/her/experiment/train.py --num_cpu 19 --env_name HandManipulateBlock-v0 --n_epochs 200 --replay_strategy future

The results are:

Anyone gets similar results or know how to solve the provlem?
Thank you!

Best,
Rui

[ruizhaogit]

Problem is solved using: python baselines/her/experiment/train.py --num_cpu 2 --env_name HandManipulateBlockRotateXYZ-v0 --n_epochs 200 --replay_strategy future

[DanielTakeshi]

@matthiasplappert Sorry if this seems like a dumb question but:

The reason why we use num_cpu = 1 by default is that it would simply be a pretty terrible idea to run it with much more on a less powerful machine.

Why would this be a "terrible idea"? Is it mostly that with a machine that has very few cores, the algorithm is dominated by communication among the different processes?

[matthiasplappert]

"Terrible idea" might have been an exaggeration but at least on Mac OS X it hits your CPUs so hard that the rest of the system starves.

[ghost]

@sanjeevanahilan would you mind providing the version of mujoco, gym, tensorflow, and baselines? I am unable to get any signal of learning. And I have tried many different version of settings. Thank you!

[GbengaOdesanmi]

@all please I am unable to reproduce the result, using this command python -m baselines.her.experiment.plot /tmp/openai-2018-11-30-16-53-40-674939/policy_best.pkl

python -m baselines.her.experiment.plot /tmp/openai-2018-11-30-16-53-40-674939/policy_best.pkl
/home/gbenga/Downloads/abiona1008/envs/tensorflow/lib/python3.6/site-packages/matplotlib/__init__.py:886: MatplotlibDeprecationWarning:
examples.directory is deprecated; in the future, examples will be found relative to the 'datapath' directory.
"found relative to the 'datapath' directory.".format(key))