Stable-baselines: Question with HER: AttributeError: 'Box' object has no attribute 'spaces'

[yjc765]

Hello,
I have a problem when I tried to use DDPG + HER.
The problem seems the definition of observation_space.

Traceback (most recent call last):
  File "/home/all-jy/git/jy_gym_stfl/train_ur5.py", line 35, in <module>
    policy_kwargs=dict(layers=[256, 256, 256]))
  File "/home/all-jy/.local/lib/python3.5/site-packages/stable_baselines/her/her.py", line 47, in __init__
    self._create_replay_wrapper(self.env)
  File "/home/all-jy/.local/lib/python3.5/site-packages/stable_baselines/her/her.py", line 62, in _create_replay_wrapper
    env = HERGoalEnvWrapper(env)
  File "/home/all-jy/.local/lib/python3.5/site-packages/stable_baselines/her/utils.py", line 25, in __init__
    self.spaces = list(env.observation_space.spaces.values())
AttributeError: 'Box' object has no attribute 'spaces'

The model is:

model = HER('MlpPolicy', env, DDPG, n_sampled_goal=4,
            goal_selection_strategy='future',
            verbose=1, buffer_size=int(1e6),
            actor_lr=1e-3, critic_lr=1e-3,
            gamma=0.95, batch_size=256,
            policy_kwargs=dict(layers=[256, 256, 256]))

and observation_space I defined is:

        self.action_space = spaces.Box(-np.inf, np.inf, shape=(7,), dtype='float32')
        obs = self._get_observation() # obs is a 1D array with 45 elements.
        self.observation_space = spaces.Box(-np.inf, np.inf, shape=obs.shape, dtype='float32')

According to the error message, I need to add a attribute called 'space' to 'Box'? I'm confused.
Could anyone help me with this problem?

[araffin]

Hello,
The answer is in the documentation:
"
HER requires the environment to inherits from gym.GoalEnv
"

EDIT: cf source code: https://github.com/openai/gym/blob/master/gym/core.py#L156

[PierreExeter]

Hello,

Thanks Araffin. How can I find out whether an environment inherits from gym.GoalEnv?

Obviously BitFlippingEnv does (as shown in the documentation) but most other Gym environments don't (well at least not the ones I tried: CartPole-v1, MountainCar-v0, Acrobot-v1, Pendulum-v0, MountainCarContinuous-v0)

[araffin]

How can I find out whether an environment inherits from gym.GoalEnv?

Because gym does not have a proper documentation, you have to look at the source code.
In practice, only the robotics envs (FetchReach, PickAndPlace, ...) do, but you need a mujoco license for that.
The rule is gym.Env, gym.GoalEnv is the exception.

[kevin5k]

TLDR;
"Difference in observation space type between gym.Env and gym.GoalEnv"