Baselines: A2C Loss Function Value vs Policy

[sccrthlt]

The A2C loss function used in this repository is the following:
` neglogpac = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=train_model.pi, labels=A)

    pg_loss = tf.reduce_mean(ADV * neglogpac)
    entropy = tf.reduce_mean(cat_entropy(train_model.pi))
    vf_loss = tf.reduce_mean(mse(tf.squeeze(train_model.vf), R))
    loss = pg_loss - entropy*ent_coef + vf_loss * vf_coef

Where there is one loss function for updating all weights in the NN (both value and policy function approximators).

Yet, both Reinforcement Learning: An Introduction and the original A3C paper Mnih [16'] [Asynchronous Methods For Deep Reinforcement Learning](https://arxiv.org/pdf/1602.01783.pdf) implement two loss functions, one for the value function and one for the policy function:

w <- w + alpha*discount*advantage*value_gradient
theta <- theta + alpha*discount*advantage*log_policy_gradient

Why does the A2C implementation here only utilize a combined loss function, whereas the literature utilize separate loss functions for the value and policy function approximators?

[pzhokhov]

@sccrthlt I assume the update formulas you mention are the one from Sutton, Barto, p. 274 (the box titled "One-step actor-critic (episodic)", (or with slightly modified notation algorithm S3 from the Mnih et al paper). Note that in their notation parameters w are used for value function, and theta - for policy, but in principle there can be overlap between w and theta parameters.
Consider the loss function

L(w, theta) = value_loss(w) + policy_gradient_loss(theta)

Then (I really wish github started supporting latex :)

dL/dw = dvalue_loss(w)/dw,
dL/dtheta = dpolicy_gradient_loss/dtheta

so the update equations (if the simple SGD is used, without extra fancifications such as adam or momentum) are:

w <-- w - alpha * dvalue_loss / dw
theta <-- theta - alpha * dpolicy_gradient_loss/dtheta = theta + alpha * advantage * d (log pi) / d theta

(because policy_gradient_loss = - advantage * log pi)
which is the same as the update formulas you provided. If some of the parameters are shared, then the corresponding update will be a sum of both value gradient update and gradient of log pi update - whether computed through gradient of the combined loss or as a result of two updates.
I 'll close this issue for now, however, @sccrthlt, please re-open if the explanation above is not clear or does not address your question.