Baselines: VecEnv SubProcEnv

[damienlancry]

Just wondering what is a vectorized environment? Where to get additional information about it?
Cheers!

[pzhokhov]

Vectorized environment is a wrapper for multiple environments running in parallel (so that an algorithm can get updates from multiple trajectories at once - this gives larger update batches and reduces the variance of the updates). SubprocVecEnv uses environments running is separate subprocesses that communicate with head process via pipes. ShmemVecEnv is largely a duplicate of SubprocVecEnv, we'll be cleaning that up at some point.
For theoretical / literature justification of sampling several trajectories at once, let's ask @joschu, @olegklimov or @unixpickle.

[yoojinoh]

Hi @pzhokhov, Thanks for your answer. But I don't understand how SubProcEnv is different from DummyVecEnv.
I am trying to use PPO2 with a customized environment and what I noticed in the run.py the mujoco uses the SubProcVec when there are a number of environment copies being run in parallel or else just a DummyVecEnv.
Do I have to use them together or should the SubProcEnv do the job that the DummyVecEnv does?

[pzhokhov]

Hi @yoojinoh! DummyVecEnv and SubprocVecEnv are doing the same job (in fact, in terms of functionality, one can be replaced with the other - they should have the same API defined by abstract superclass VecEnv), the only difference is that SubprocVecEnv runs steps in multiple environments in subprocesses, i.e. actually in parallel, whereas DummyVecEnv runs steps for multiple environments one after another and is not using any kind of parallelism.
To elaborate on that point, consider a call to venv.step(), where venv is in instance of either DummyVecEnv or SubprocVecEnv. DummyVecEnv interates over multiple environments in a for loop, calling env.step() for the wrapped environments one after another. If using Nenv environments, and each of them takes time Tstep to make a step, DummyVecEnv.step will take Nenv * Tstep. SubprocVecEnv first sends the data to all of the subprocesses, and then waits for all of them to send the data back - this way, computation of environment steps happens in parallel. This way, SubprocVecEnv only takes time max(Tstep_i) to run a step. Natually, when using only one environment, SubprocVecEnv does not give any gain in speed (in fact, it introduces some inter-process communication overhead), that's why one num_envs == 1, we use DummyVecEnv.

[simoninithomas]

@pzhokhov Hi, thanks for this explanation, I finally understood the difference but please put it on the documentation because that can help a lot of users.

Again, thanks

[pzhokhov]

okay, will do

[zuoxingdong]

@pzhokhov Thanks for the detailed explanation in above comments. I have a small additional question, what is the motivation of ShmemVecEnv ? I thought it is because by sharing the same memory to store and add large data it could be more efficient than SubprocVecEnv for sending it through Pipe connection (which has quite small capacity, say 30 - 50 MB).

However, when I read the code, it seems the observation buffer is shared through multiprocessing.Array and things like sending rendered RGB arrays still sending via pipe connections. So I am wondering if I got it right about the goal of ShmemVecEnv over SubprocVecEnv.

[pzhokhov]

ShmemVecEnv is a somewhat optimized version of SubprocVecEnv which also uses subprocesses, but communicates not only via pipes, but also by using shared observation buffers. We use shared buffers only for observations because usually observations are the biggest objects to be sent through the pipes. Rendering is usually less of a bottleneck because normally you would not need to render in a learning loop. On the other hand, we don't have rigorous benchmarks to support the idea that pipes introduce a significant slowdown, so I'd recommend using SubprocVecEnv by default.