Baselines: PPO2 sample action strategy?

[tyzaizl]

This is not a bug report.
I am confused about the PPO2 sample action strategy:
def sample(self):
u = tf.random_uniform(tf.shape(self.logits))
return tf.argmax(self.logits - tf.log(-tf.log(u)), axis=-1)

Why it is log(-log())?
Why use action_model policy output minus the random numbers: [pi - log(-log(u))]?
The pi is the action_model policy output.
Is there any references for the action sample strategy?
Thanks a lot.

[unixpickle]

That is called the Gumbel-Softmax, it is simply a mathematical trick for sampling from a categorical distribution. This is equivalent to applying the softmax to logits and sampling according to the resulting probabilities.

Relevant reading: https://arxiv.org/abs/1611.01144

[tyzaizl]

@unixpickle Thanks so much.
Also found some references. May be useful.
https://casmls.github.io/general/2017/02/01/GumbelSoftmax.html
https://hips.seas.harvard.edu/blog/2013/04/06/the-gumbel-max-trick-for-discrete-distributions/

[hackershi]

This is black magic. Thanks a lot.