Baselines: PPO on Atari Specifics

[Ashboy64]

Thanks for the great resource. I had a question regarding the implementation of PPO for Atari environments. In order for the model to learn the optimal policy, it seems to me that it must be able to detect motion in the game. Thus, feeding one frame to the model each time when an action needs to be taken is inadequate, a collection of frames or a difference frame will need to be fed in to allow the algorithm to 'see' the motion that is occurring. Is this being done in this implementation? Where can I see the specifics of this process?

Thanks in advance.

[DanielTakeshi]

It's standard to use frame stacking of four frames.

Look at baselines.run where the environment is being built (in build_env). You can see parameters such as frame_stack_size. You can tell if you have frame stacking by getting observations from the environment and printing out the shapes of the elements. They should be (batch_size, 84, 84, 4), at least for Atari.

[pzhokhov]

What @DanielTakeshi said :) When working with atari, all algorithms use a stack of frames; it is either inserted via FrameStack environment wrapper:
https://github.com/openai/baselines/blob/d3fed181b57f61013698521f4e940594364253e9/baselines/common/atari_wrappers.py#L153
or via VecFrameStack:
https://github.com/openai/baselines/blob/d3fed181b57f61013698521f4e940594364253e9/baselines/common/vec_env/vec_frame_stack.py#L6
The former operates on a single environment, the latter - on several environments running in parallel (VecEnv).
Hope this helps, please reopen if further clarification is needed.