Baselines: Where does the ppo2 reset the env?

[HitLyn]

In ppo1, when the env.step() gets done which means the mission has been achieved, the env will be reset by env.set(),then the training continue:
https://github.com/openai/baselines/blob/5b41c926c7a852df3f0928afdf2429f96a3965cb/baselines/ppo1/pposgd_simple.py#L61
But I can't find where ppo2 reset the env, or do I misunderstand something?

[andytwigg]

ppo2 uses vectorized environments, so the reset is done inside the vecenv -
see for example
https://github.com/openai/baselines/blob/master/baselines/common/vec_env/dummy_vec_env.py#L54

On Tue, 19 Feb 2019 at 11:24, Lin Cong [e-mail] wrote:

In ppo1, when the env.step() gets done which means the mission has been
achieved, the env will be reset by env.set(), then the training continue.
But I can't find where ppo2 reset the env.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/openai/baselines/issues/829, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACXttRo4xA-mJoe-vFReBV1FG7RbJfdLks5vPE9jgaJpZM4bDsFK
.

[HitLyn]

@andytwigg Thanks for the explain, problem solved.

[btickell]

@andytwigg sorry to beat a dead horse but it seems like ppo2.runner does not explicitly call env.reset() outside of __init__(). This also isn't done in the ppo2 training code so I am unsure where this is actually being done.

[andytwigg]

The reset is done inside the vectorized env - see eg
https://github.com/openai/baselines/blob/master/baselines/common/vec_env/dummy_vec_env.py#L53

The runner keeps calling vecenv.step() which resets the underlying envs as
necessary

On Tue, 9 Apr 2019 at 13:40, Blake Tickell [e-mail] wrote:

@andytwigg https://github.com/andytwigg sorry to beat a dead horse but
it seems like ppo2.runner does not explicitly call env.reset() outside of
__init__(). This also isn't done in the ppo2 training code so I am unsure
where this is actually being done.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openai/baselines/issues/829#issuecomment-481426961,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACXttZE8492IPdjTptgDS9HizIN__hXtks5vfPrQgaJpZM4bDsFK
.

[btickell]

I see, this seems to require the environment to explicitly set the done flag to True when desiring a reset? For some tasks this may not be desirable but it can be worked around.

[andytwigg]

It uses the reset flag that is returned from the call to env.step

On Tue, 9 Apr 2019 at 13:50, Blake Tickell [e-mail] wrote:

I see, this seems to require the environment to explicitly set the done
flag to True when desiring a reset? For some tasks this may not be
desirable but it can be worked around.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openai/baselines/issues/829#issuecomment-481430243,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACXttU8tQiMX9BuytFNEVpqmTx27jtfRks5vfPz-gaJpZM4bDsFK
.

[btickell]

This seems like an odd requirement for the runner to work correctly since it seems quite a few environments in gym (i.e reacher, pusher, etc.) do not set the done flag to True at any point during training.

[andytwigg]

I'm not that familiar with the mujoco envs. Perhaps they set the flag to
true after a certain number of steps or when a goal is achieved.

On Tue, 9 Apr 2019 at 14:30, Blake Tickell [e-mail] wrote:

This seems like an odd requirement for the runner to work correctly since
even it seems quite a few environments in gym (i.e reacher, pusher, etc.)
do not set the done flag to True at any point during training.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openai/baselines/issues/829#issuecomment-481447896,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACXttWViSNEWM28Rdw8jWkwto0zODGIcks5vfQZ8gaJpZM4bDsFK
.

[btickell]

It's pretty inconsistent between environment since some tasks have a good notion of "doneness" and others don't. As far as I know there is no built in counter for mujoco envs. For some jobs it would make sense to reset the environment after sampling but I can see in some cases that wouldn't be desired