Baselines: Understanding multi iteration update of model in PPO

[mbcel]

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly there are multiple iterations of weight updates done on the model with data that is created from the environment (with the model before the update). Now I think that the updates of the model weights are not correct anymore after the first iteration/optimization step because the model weights changed and therefore the training data is outdated (since the model would now give different actions in the environment and therefore different rewards).

So is this some form of approximation or augmentation on the data by using the data that was created by an older model for multiple iterations or am I missing something here? And if yes where was this idea first introduced or better described? And where is a (empirical) proof that this still leads to a correct weight updating?

I am new to this field so I am sorry if this is an obvious question.

[pisiiki]

You can separate RL algorithms into on-policy and off-policy. Sometimes it is usefull to process a bunch of data before updating, this allows you for example to prepare a batch and use gpu resources better or to train in multiple threads/gpus/machines. Off-policy algorithms process "off-policy" samples, that is, data that has not been generated with your current policy.

[matthiasplappert]

To clarify: PPO is an on-policy algorithm so you are correct that going over the same data multiple times is technically incorrect.

However, we found that PPO is actually quite okay with doing this and we still get stable convergence. This is likely due to the proximal trust region constrained that we enforce, which means that the policy cannot change that much anyway when going over the current set of transitions multiple times, making it still approximately on-policy. You can of course get rid of this but then you'll need more samples.

[mbcel]

Okay I understand. Thank you for the good explaination!

[imcomking]

It's too detailed, but the surrogate objective function of PPO was derived from TRPO and they adopted importance sampling for the sample-based estimation. Typically importance sampling is used to take advantage of past experiences for sample efficiency. If so, PPO is an "off-policy" but their replay buffer is very short?

https://arxiv.org/abs/1502.05477

[scotthuang1989]

I think this is called importance sampling? a standard tools for on-policy RL?

[dbsxdbsx]

@scotthuang1989 ,I think importance sampling is designed for off-policy algorithm. But it happen to be useful in PPO, which is an on-policy algorithm.