Baselines: [Question] A2C entropy of the policy

[damienlancry]

Hi all,

I am having a very hard time trying to understand how do we compute the gradient of the entropy with respect to the actor parameters in continuous action domains. This gradient is added to the policy gradient in A3C paper to discourage too quick convergence toward suboptimal deterministic policies.

It would require a integration over the action space so it seems to me that it is intractable in many cases. Is the integral sampled?

I looked at the code in baselines/common/distributions and i am completely lost, I do not understand many of the notations (Pd = policy distribution? , flat = ? , neglogp = negative log probability? ) and why is the name of the class Diag Gaussian? Are we assuming the policy distribution to be Gaussian in every continuous action domains?

Is there a documentation about this code that i missed?

Thanks a lot for any information
Cheers!

[pzhokhov]

Hi @DamienLancry!
The way we usually deal with stochastic policies in policy gradient-like methods is by choosing certain type of probability distributions and making neural network (or some other function approximator) output parameters of such class. The type of the distribution depends on the action space - for discrete action spaces, categorical distribution is a natural choice, for continuous action spaces gaussian distribution is a reasonable choice (although not always the best, if, for some reason, multi-modal distributions are needed).
To answer your question directly:

• computing the gradient of entropy - entropy the distribution over actions can be computed analytically given the distribution parameters (at least, for the distribution types used). Similarly, entropy can be differentiated with respect to distribution parameters, and these gradients can then be back-propagated to compute gradients of entropy with respect to policy (i.e. neural network) parameters. For that, we are relying on tensorflow autodiff magic - the Pd.entropy() methods construct correct analytical expression for entropy, we then add it to the loss; and the tensorflow optimizer does the heavy lifting of the backprop and gradient accumulation.

• in baselines, we are assuming gaussian distributions for continuous action spaces. At some point we tried beta distributions that are nominally better suited for finite-size domains, but that did not seem to improve the results. Hence the name of the class DiagGaussian.
  Not only it is restricted to gaussian, but also covariance matrix of that (in principle multi-variate) gaussian distribution is diagonal. In fact, you may also notice that neural networks output only mean of that gaussian, and the variance is an observation-independent learnable variable.
  As for notations:
  PdType - type of probability distribution
  Pd - probability distribution (PdType + specific set of distribution parameters)
  flat - denotes flattened vector of parameters of probability distribution. For gaussians, thats a concatenation of means and logstds, where logstd is log of standard deviation.
  neglogp - negative log probability of an action; used to compute negative log probability of a trajectory to compute policy gradient.

Hope that helps!

[damienlancry]

HI @pzhokhov !
Yes that actually helps a lot, thank you so much.
However it is still not clear to me what you mean by "observation-independent learnable variable". It sounds to me that it should be an output of the ANN if it is learnable.
Am I wrong?

Cheers!

[pzhokhov]

According to @joschu and @olegklimov's studies, outputting logstd from neural network does not work great (inviting them to comment more). In DiagGaussianPd we are setting logstd to be a tensorflow variable, which means that gradients with respect to it are computed, and logstd is updated with the optimizer, but does not depend on the observation. In other words, logstd is an updateable parameter, and not a function of observation and some other parameters.