Baselines: Deactivate learning rate scheduling to PPO with Adam

[timmeinhardt]

In contrast to A2C and A2C_ACKTR, PPO already includes a learning rate scheduling performed by Adam. In supervised learning it is debatable if one should use manual scheduling in combination with Adam, but in doubt I would recommend to not interfere with the Adam intrinsic scheduling.

Did you investigate this issue in RL or would it indeed make sense to deactivate the linear scheduling for PPO by default?

[pzhokhov]

I think @olegklimov and @joschu ran some experiments that show that linear schedule of the learning rate helps ppo, even when adam optimizer is used. I don't know of the works that would justify / disprove from any theoretical standpoint, but it sounds like an interesting experiment to run with different algorithms. Leaving open pending comment from @joschu @olegklimov

[joschu]

Adam doesn't have much to do with LR scheduling--it still has a LR parameter.

[scotthuang1989]

@joschu when I train a supervise problem. lr schedulingwith adam helps. Do you try exponential decay, cycling LR scheduling?