Baselines: What's the best parameters configuration to run humanoid-v2 env?

[RebornHugo]

I have tried Humanoid-v2 env using ppo2 algorithms with bash command in README.md.

1. python -m baselines.run --alg=ppo2 --env=Humanoid-v2 --network=mlp --num_timesteps=2e7
   The first one ran normally, but the result is not good (not as good as the result in PPO paper).
2. python -m baselines.run --alg=ppo2 --env=Humanoid-v2 --network=mlp --num_timesteps=2e7 --ent_coef=0.1 --num_hidden=32 --num_layers=3 --value_network=copy
   The second one crashed, like issue #587 .
   And in MUJOCO benchmark page, I can't find a humanoid benchmark. Could you please provide a default configuration for Humanoid-v2 and a comparable benchmark result for it?

[pzhokhov]

Very reasonable request... I don't have data for running ppo2 with humanoid handy, but we'll produce it. I went through this exercise some time in the past with ppo1 (hence ppo1/run_humanoid.py); I think it can be a good starting point for trying ppo2 (i.e. hyperparameters for ppo2 should be approximately the same).

[RebornHugo]

@pzhokhov Thank you for your answer, I have tried to run multiple humanoid env with multiple subprocesses, it does increase performance. And there are some paper states that with larger batch size, the performance will increase too.

[Michielskilian]

Hi @RebornHugo,

After a long search I managed to increase the performance to a reasonable point for the Humanoid-v2 with the following parameters:

• nsteps: 4096
• nminibatches: 64
• lam: 0.95
• gamma: 0.99
• noptepochs: 10
• ent_coef: 0.0
• lr: lambda f: 1e-4 * f
• cliprange: 0.1
• value_network: copy
• num_hidden: 512
• num_layers: 4

With this I get a reward of 3436 after 319 updates. I set num_env to 6 and run with MPI.

However, I can't seem to figure out why there is a sudden performance drop starting around 300 updates since all the other metrics seem to behave quite normal.. I've added the plots below so you can check whether these results match when you try it.