Baselines: TfRunningMeanStd error with mujoco

[inkyusa]

Hello,
According to comment on https://github.com/openai/baselines regarding Mujoco vecnormalization, I changed RunningMeanStd by TfRunningMeanStd as instructed but faced the following error while running "python -m baselines.run --alg=ppo2 --env=Humanoid-v2 --network=mlp --num_timesteps=2e7"

File "/home/testmony/workspace/venv_p3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1659, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shapes must be equal rank, but are 1 and 0 for 'Assign_3' (op: 'Assign') with input shapes: [376], [].

Did anyone experience similar issue?

BTW "python -m baselines.run --alg=ppo2 --env=PongNoFrameskip-v4 --num_timesteps=1e6" seems to work properly with TfRunningMeanStd.

I am using Mujoco200, Ubuntu 16.04, TF 1.13.1 (CPU-only) and baseline master branch.

[Michielskilian]

Hi inkyusa,

It's not quite clear in the description but i think you need to change only one of the RunningMeanStd to TfRunningMeanStd. Since this issue involves the environment, I think you have to change the first one so you have the following code change:

Edit: This should work:

Original:

from baselines.common.running_mean_std import RunningMeanStd
def __init__(self, venv, ob=True, ret=True, clipob=10., cliprew=10., gamma=0.99, epsilon=1e-8):
        VecEnvWrapper.__init__(self, venv)
        self.ob_rms = RunningMeanStd(shape=self.observation_space.shape) if ob else None
        self.ret_rms = RunningMeanStd(shape=()) if ret else None

New:

from baselines.common.running_mean_std import RunningMeanStd, TfRunningMeanStd
def __init__(self, venv, ob=True, ret=True, clipob=10., cliprew=10., gamma=0.99, epsilon=1e-8):
        VecEnvWrapper.__init__(self, venv)
        self.ob_rms = TfRunningMeanStd(shape=self.observation_space.shape, scope='ob_rms') if ob else None
        self.ret_rms = TfRunningMeanStd(shape=(), scope='ret_rms') if ret else None

Hope this helps!

[pzhokhov]

I have updated the description and logic to switch between RunningMeanStd and TfRunningMeanStd in this PR: https://github.com/openai/baselines/pull/886
that being said, the fact that normalization is needed at all is somewhat annoying. I would recommend looking at roboschool continuous control environments that have more regular observations / rewards and do not need normalization.

[inkyusa]

@Michielskilian and @pzhokhov Thanks for pointing out and it works now.