Baselines: [question] Compute Atari score and not reward
Hello,
It seems that the reward that Atari environments provide is not the score of the atari game. How do you compute it to obtain the benchmark curves?
Related to https://github.com/hill-a/stable-baselines/issues/43
Example: A2C on seaquest in a colab notebook: https://colab.research.google.com/drive/1mBDRxZZu0ajS8tiR1DQSWSAqaUSURbvL
You can download the trained agent here: https://drive.google.com/open?id=1fibRLLmN6YO1h3CX8Jtv4QKhOLBrFbw1
The agent achieves a score of ~800 (with stochastic actions) but this differs from the reward per episode (which is around 10)
araffin
All 3 comments
You can get it from infos when you call env.step(action), if there is an 'episode' keyword, then it hsa finished and you get the unclipped score
DanielTakeshi
on 23 Oct 2018
The environment that agent interacts with may have several layers of wrappers, including ones that modify the reward - that is likely what creates discrepancy between score and reward. For instance, by default ClipRewardEnv is used, which bins rewards at each timestep into -1, 0 or 1 depending on its sign (both in baselines and in stable_baselines, look into common/atari_wrappers.py). To get the score, we use Monitor wrapper (bench/monitor.py) before all other wrappers that saves the raw score of the episode to *.monitor.csv files; and also as @DanielTakeshi pointed out, passes it into info dictionary.
pzhokhov
on 23 Oct 2018
Ok, thanks
araffin
on 23 Oct 2018
Related issues
dkorenkevych
路
11Comments
DanielTakeshi
路
34Comments
gbg141
路
15Comments
DanielTakeshi
路
8Comments
watts4speed
路
7Comments
Most helpful comment
The environment that agent interacts with may have several layers of wrappers, including ones that modify the reward - that is likely what creates discrepancy between score and reward. For instance, by default ClipRewardEnv is used, which bins rewards at each timestep into -1, 0 or 1 depending on its sign (both in baselines and in stable_baselines, look into common/atari_wrappers.py). To get the score, we use Monitor wrapper (bench/monitor.py) before all other wrappers that saves the raw score of the episode to *.monitor.csv files; and also as @DanielTakeshi pointed out, passes it into info dictionary.