Stable-baselines: Communicate between SubProcVecEnv processes

[jviquerat]

Hi,

I am successfully using SubProcVecEnv to launch parallel environments with PPO2.
Is there a way for the parallel environments to communicate between them ?

Thanks in advance

[Miffyli]

This is not part of stable-baselines, rather it is part of the environments.

Off top my head I think you could create multiprocessing objects (e.g. Queue, Manager) to allow communication between environments in different processes. You can feed these to environments during creation of theSubProcVecEnv, e.g.

message_queue = multiprocessing.Queue()
env = SubprocVecEnv([lambda i: make_env(i, message_queue) for i in range(num_cpu)])

I am not sure if this will run into some roadblocks, though.

[araffin]

Hello,
To complete @Miffyli answer, there are also get_attr, set_attr and env_method that you could use to communicate between those (cf doc). However, @Miffyli solution looks cleaner.

[jviquerat]

Thank you very much for these answers.

• Regarding @Miffyli answer: I will document myself on multiprocessing as I am not familiar with it
• Regarding @araffin answer: doesn't that suppose to access the members of the SubprocVecEnv wrapper from inside each environment ? Is that really possible ? To clarify the situation, I instantiate my environments as follows :

def make_env(rank, seed=0):
    def init():
        env = EnvOpt(rank)

        env.seed(seed + rank)
        return env

    set_global_seeds(seed)
    return init

def launch_training():
    ...
    env = SubprocVecEnv([make_env(cpu) for cpu in range(n_cpu)])
    ...

where EnvOpt is my custom environment class.

[araffin]

doesn't that suppose to access the members of the SubprocVecEnv wrapper from inside each environment ?

It depends on who manages the communication. He you manage it from outside, you don't need to access SubprocVecEnv from inside the env. But go for @Miffyli answer, it is much cleaner.

Btw, you should know that DummyVecEnv is usually faster than the subprocess version, and much easier to handler (because this is just a for loop on the envs)

[jviquerat]

Btw, you should know that DummyVecEnv is usually faster than the subprocess version, and much easier to handler (because this is just a for loop on the envs)

If I am not wrong DummyVecEnv is not truly parallel, is it ? In my case, running the environments simultaneously is a true gamechanger, as obtaining my reward is computationally expensive

It depends on who manages the communication. He you manage it from outside, you don't need to access SubprocVecEnv from inside the env. But go for @Miffyli answer, it is much cleaner.

What I am trying to do is share informations between the parallel environments, "from the inside", ie while they are running. I will give a thorough look at multiprocessing and see what I can get from it :)

[jviquerat]

Follow-up: I seem to be able to do what I want using get_attr and set_attr, as suggested by @araffin :)
All the plumbing is managed directly in the callback function, as it suits my needs very well. Might not work for a general-purpose communication scheme though.

Thanks again !

[smorad]

I'm having issues with what @Miffyli suggested. When I pass a Pipe or Queue object, I'd get EOF or invalid argument errors. After many times running the following code

p0, p1 = Pipe()
SubprocVecEnv([MyEnv(p0, True).start, MyEnv(p1, False).start])

class MyEnv:
  def __init__(self, pipe, writer):
    self.pipe = pipe
    self.writer = writer
  ...
  def start(self):
    if self.writer:
      self.pipe.send('foo')
    if not self.writer:
      print('Got ', self.pipe.recv())
  ...

I got lucky and saw
Got (get_spaces, None)
This is a message sent by SubprocVecEnv.__init__ in subproc_vec_env.py

I now suspect all these errors I'm seeing are because there are multiple readers/writers to the same pipe, which is forbidden. I am not well versed in multiprocessing. Why would SubprocVecEnv and MyEnv be writing to the same pipe?

[Miffyli]

Check arguments for vectorized environments. You give in a list of functions, each of which return an environment.