Mujoco-py: Render off- and onscreen

[ChristianLemke]

Hi
I am using Gym and Mujoco to do some Reinforcement Learning. I have a environment with an extra camera "head".
I want to render an image of the camera offscreen so that i can use the image-data in the observation space. For evaluation purpose I also want to render onscreen by using the default MjViewer-window.
I am using Ubuntu 16, python3.5, gym 0.9.7, mujoco-py 1.50.1.33.

This is what i do:

In my Mujoco-Gym-Env:

....
def _get_obs(self):
        qpos = self.sim.data.qpos
        qvel = self.sim.data.qvel
        img = self.sim.render(camera_name='head', width=16, height=16, depth=False)
        gray = color.rgb2gray(img)
        return np.concatenate([qpos.flat[2:], qvel.flat, gray.flat])
...

In my tester:

def test():
    env = gym.make("MyEnv-v1")
    ob = env.reset()
    print("actionspace", env.action_space)
    print("observationspace", env.observation_space)
    step = 0
    while True:
        action = env.action_space.sample()
        ob, reward, done, info = env.step(action)
        env.render()
        step += 1

        if step % 250 == 0:
            env.reset()
if __name__ == __main__:
    test()

Problem:
It is not working together. I can either do env.render() or img = self.sim.render(camera_name='head', width=16, height=16, depth=False) but not both together or i get the following error.

Output error:

actionspace Box(8,)
observationspace Box(276,)
Creating window glfw
X Error of failed request:  BadAccess (attempt to access private resource denied)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  5 (X_GLXMakeCurrent)
  Serial number of failed request:  153
  Current serial number in output stream:  153

Process finished with exit code 1

I spend some time trying MjRenderContextOffscreen(self.sim, 1)-class but with no success.
Please help :-) thx

[ChristianLemke]

Hi
Only a few hours after this issue there was the commit that helped me solve this problem.

I am now using cymj.MjRenderContextOffscreen(self.sim, 0) as an separate viewer.

To render the head camera I am now using:

self.get_head_cam_viewer().render(width, height, device_id)
data = self.get_head_cam_viewer().read_pixels(width, height, depth=False)
return data[::-1, :, :]

It opens some empty full screen windows but ok.

[BolunDai0216]

Can you help me understand the process, I am having trouble to understand how to add an additional camera in the XML file also how to access it in the MuJoCo-py-env class

[mbed92]

Hey. I know that this thread is closed, but I'll leave my post in case someone needs some help.

@BolunDai0216 In order to add an additional camera to you scene, add a tag camera under worldbody in your XML file.
<camera name="main1" mode="targetbody" target="box_link" pos="1.3 -1.3 2.0" fovy="42.5"/>

Then please try the code below:

import mujoco_py
import os
import cv2
import numpy as np

# load scene in MuJoCo
path = os.path.join('your', 'path', 'model.xml')
model = mujoco_py.load_model_from_path(path)
sim = mujoco_py.MjSim(model)

# to speed up computation we need the off screen rendering
viewer = mujoco_py.MjRenderContextOffscreen(sim, 0)
for i in range(3):
    viewer.render(420, 380, 0)
    data = np.asarray(viewer.read_pixels(420, 380, depth=False)[::-1, :, :], dtype=np.uint8)

    # save data
    if data is not None:
        cv2.imwrite("test{0}.png".format(i), data)

    print(i)
    sim.step()

We're using MjRenderContextOffscreen, because it saves us a lot of time comparing to MjSim.render() See this thread. It's probably related.

[Jendker]

@mbed92 But isn't MjSim.render() doing exactly the same thing which you have described here?
https://github.com/openai/mujoco-py/blob/0711ab58777a28aff847adbf05ba246a337908a0/mujoco_py/mjsim.pyx#L131

[mbed92]

Sometimes you want to save images from your simulation, but not necessarily display them during experiments. As far as I remember MjSim.render(...) will always display images on your screen, while MjRenderContextOffscreen(...) will only create an array of pixels (you may e.g. save it to file). It just saved me some time.

EDIT: I get it. I missed the arg 'offscreen' in MjSim.render(), which invokes MjRenderContextOffscreen(). Which is exactly the same as what I've described.

[Jendker]

Thanks for clearing that up!