Baselines: DDPG Loss function for Actor

[mehdimashayekhi]

Hi, Thanks for sharing. For the actor loss here https://github.com/openai/baselines/blob/28aca637d0f13f4415cc5ebb778144154cff3110/baselines/ddpg/ddpg_learner.py#L172 looks like you are just considering the contribution of critic. But in the original paper, deterministic policy gradient has two components ∇θμJ≈Est∼ρβ ∇θμQ(s,a|θQ)|s=st,a=μ(st|θμ) =E β ∇ Q(s,a|θQ)| ∇ μ(s|θμ)|. In your implementation the contribution of actor is missing. Appreciate your response.

[pzhokhov]

Hi @mehdimashayekhi! We do have both factors of the gradient due to the tensorflow automatic differentiation magic. I have written up a short overleaf doc explaining why does it work (I am sorry, I don't have your patience with greek symbols in github comments :)) Here: https://www.overleaf.com/read/bdhhbwfpcpbr
Note that self.critic_with_actor_tf corresponds to Q(s, mu(s, theta_mu))

[mehdimashayekhi]

@pzhokhov thanks much. Really appreciate taking time explaining it. Now totally make sense,