Open3d: Headless OpenGL rendering

[qianyizh]

This can be useful in scenarios where users want to render a (long) video.
It can also be useful when Open3D is deployed in a remote server and window creation is a problem.

Some information regarding GLFW: https://github.com/glfw/glfw/issues/648

[syncle]

I am looking into this issue and read some documents.
According to this glfw 3.3 support headless rendering using OSMesa.
The glfw version included in Open3D seems to be 3.1. Can we upgrade glfw for this?

[qianyizh]

Yes.

[syncle]

As we discussed today, glfw 3.3 is not available yet. So we need to find the detour.
I tested following code with one of GCP virtual machine

glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);

but got the error.

GLFW Error: X11: The DISPLAY environment variable is missing
Failed to initialize GLFW
Failed creating OpenGL window.

Basically glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE); hides existing window. This method does not working in the environment GLFW cannot make a window.

[qianyizh]

Recently I have the needs for glfw rendering on headless server and have gain some success.
See this:
https://github.com/glfw/glfw/issues/1004

In short, Alexey's email is absolute correct.

First, you need to setup Xorg correctly.
You can check /etc/X11/xorg.conf to see if it is configured headlessly (usually via sudo nvidia-xconfig -a --use-display-device=none).
Stop lightdm and restart Xorg to make sure the correct configuration is running.

Then, if you don't want to specify a GPU, you can just setup the environment variable DISPLAY to :0 and test the OpenGL setup.

export DISPLAY=:0
glxinfo | grep OpenGL

You should see something like:

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: TITAN Xp/PCIe/SSE2
OpenGL core profile version string: 4.5.0 NVIDIA 384.90
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.5.0 NVIDIA 384.90
OpenGL shading language version string: 4.50 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 384.90
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:

If your server has multiple GPUs, you will be able to see multiple copies of the above information.

If you have passed this test, setup the DISPLAY environment variable and try some program with glfwInit(), say

export DISPLAY=:0
./some_program_that_has_glfwinit

There is a risk that you will have this error

glfw.GLFWError: (65544) b'X11: RandR gamma ramp support seems broken'

This is an error of version 3.2 and older. They don't support headless rendering. Since 3.3 (current master) it is supported.
Then everything should be fine.

In addition, if you want to specify GPU, you can use VirtualGL and TurboVNC to mock a display. This is done with the following steps.

First, install VirtualGL and TurboVNC, something like:

sudo dpkg -i virtualgl_xxx.deb
sudo dpkg -i turbovnc_xxx.deb

Then launch vnc server to run glfw.

// These two steps make sure that the right headless Xorg is configured
sudo service lightdm stop
sudo nohup Xorg :0 &

// Launch a vncserver to mock display on :1
// You can change it to :2, :3, ...
/opt/TurboVNC/bin/vncserver :1

// Launch program on :1
export DISPLAY=:1
vglrun -d :0.0 glxinfo
vglrun -d :0.0 <a script that calls glfw>

[qianyizh]

Another option is to mimic libigl's tcpviewer.
See this for reference:
https://github.com/libigl/libigl/blob/master/python/tcpviewer.py

Lower priority. Close for now.

[vkoltun]

This is pretty important. Do we have to close this issue? Can we release
Open3D while this is open? I'm guessing we can. But I don't want this to
fall through the cracks, I want this clearly on the list of issues that
need to be addressed soon. (Perhaps not prior to release, but soon after.)

Vladlen

On Mon, Dec 4, 2017 at 11:18 AM, Qianyi Zhou [e-mail]
wrote:

Another option is to mimic libigl's tcpviewer.
See this for reference:
https://github.com/libigl/libigl/blob/master/python/tcpviewer.py

Lower priority. Close for now.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/IntelVCL/Open3D/issues/17#issuecomment-349074744, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AJra2imGPNUwSeKsdH4b2VLZU35rtm_mks5s9EWcgaJpZM4NXPQW
.

[qianyizh]

Reopen per request.

[bhack]

Other than VNC there is also xpra https://xpra.org/trac/wiki/Usage/OpenGL

[rukie]

As another user, this would be an excellent feature! 👍

[syncle]

Working on this. I am using glfw3.3_dev to avoid hassles.

Edit: my working branch with glfw3.3_dev.

[takanokage]

Another approach is to use Docker to create a headless environment for Open3D.
I already have a docker image based on ubuntu 16.04 with xvfb and x11vnc that works. I've also tested one light window manager (ratpoison) but I'm not too happy with it and want to test others. Next I need to test building/running Open3D inside it.

If we need access to the host GPU (NVIDIA) there are already good solutions to get that: NVIDIA-docker and NVIDIA docker images with CUDA and cuDNN.

I've also been looking into using Alpine as the base if the image size is a concern. The Ubuntu image with xvfb and x1vnc gets to about 450MB while the Alpine image to just about 70MB.

[qianyizh]

The purpose is to make Open3D rendering part running on cloud (AWS/GCloud). @syncle can you help test the solution on GCloud?

[syncle]

Sure. I can use a fresh Ubuntu machine running on GCloud. @takanokage, can you elaborate your steps for me? That will be used for documents too.

[takanokage]

I will post more once I have it working.
I'm at the point where I can build Open3D inside the docker container. However the verification step after the build fails, for this reason I added issue 241.

[takanokage]

I got Open3D built and working inside a Docker container.
However I'm getting the same X11: RandR gamma ramp support seems broken error.
Next: I'll check the VirtualGL suggestion by Qianyi.

$ cd ~/Open3D/build/lib/Tutorial/Basic
$ python3 rgbd_redwood.py
press q

Read Redwood dataset
RGBDImage of size 
Color image : 640x480, with 1 channels.
Depth image : 640x480, with 1 channels.
Use numpy.asarray to access buffer data.
GLFW Error: X11: RandR gamma ramp support seems broken
GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory

[syncle]

From @qianyizh's post

There is a risk that you will have this error

glfw.GLFWError: (65544) b'X11: RandR gamma ramp support seems broken'

This is an error of version 3.2 and older. They don't support headless rendering. Since 3.3 (current master) it is supported. Then everything should be fine.

It might be good to try Open3D + GLFW3.3_dev solution via

git clone -b headless_rendering https://github.com/syncle/Open3D.git

[takanokage]

Jaesik, I just tried your branch with glfw 3.3 inside my docker container.
Somehow I'm still getting the same X11: RandR gamma ramp support seems broken error.
I seem to remember you mentioned I need to build with a particular flag, can you tell me what that was again? tx!

[syncle]

Yes. Try it with

cmake -DOpen3D_USE_NATIVE_DEPENDENCY_BUILD=OFF ../src

Edit:
Check out what cmake says. It should say like building glfw from source.

[takanokage]

Thanks!
So that did something:
GLFW Error: OSMesa: Library not found
Failed to create window
[DrawGeometries] Failed creating OpenGL window.

Did you install a special package for this?

[syncle]

I think this should work

sudo apt-get install libgl1-mesa-dev -y 
sudo apt-get install libosmesa6-dev -y

[takanokage]

This is the output of glxinfo: OpenGL version string: 3.0 Mesa 17.2.8

libgl1-mesa-dev was already installed.
I installed libosmesa6-dev and the previous glfw error dissapeared.

Now I'm getting a segmentation fault when running Tutorial/Basic/rgbd-redwood.py or Tutorial/Advanced/customized_visualization.py. I'll troubleshoot and let you know what I'll find.

$ python3 customized_visualization.py
Reading PLY: [========================================] 100%

1. Customized visualization to mimic DrawGeometry
   Segmentation fault (core dumped)

[takanokage]

After going trough lots of resources it would seem the segmentation fault is likely an issue introduced by the use of docker.
Among the things I have tried:

• use an NVIDIA driver+CUDA based image
• use nvidia-docker to run the containers

Others have seen the same error however there's no consensus on how to address it.

[takanokage]

The Segmentation fault went away. I successfully ran Tutorial/Basic/rgbd_redwood.py and
Tutorial/Advanced/customize_visualization.py.

I cloned and tested using the master branch rather than the headless_rendering branch with glfw3.3.

I'm still getting the X11: RandR gamma ramp support seems broken error but that shows that this error has a different cause/meaning than we previously thought.

[qianyizh]

You are not linking to the dev branch of glfw (3.3).
In dev branch this message has been removed X11: RandR gamma ramp support seems broken. If you are using the correct version, you should not be able to see it.
Read the code of glfw, you will see.

[takanokage]

In other words this is more like a warning instead of an error and will go away once we move to glfw 3.3?

[qianyizh]

https://github.com/glfw/glfw/blame/6e69f63ffa4059d0f16344e6b038285c1ca78c16/src/x11_init.c
https://github.com/glfw/glfw/commit/b0258803c6d6e8b03f194a73457a1e0645196f1a#diff-9c6aefaea564bfbb5f29c9b9026e1b34
https://github.com/glfw/glfw/commit/5fe4dfb511e12b954f9fba11d989df48f5f203fd

[takanokage]

1. If I clone/build/run the headless_rendering branch I get the segmentation fault.
   $ cd ~/Open3D/build/lib/Tutorial/Advanced
$ python3 customized_visualization.py

Reading PLY: [========================================] 100%

1. Customized visualization to mimic DrawGeometry
   Segmentation fault (core dumped)

1. If I clone/build/run the master branch I get the RandR and joystick errors but the tutorials run fine.
   $ cd ~/Open3D/build/lib/Tutorial/Advanced
$ python3 customized_visualization.py

Reading PLY: [========================================] 100%

1. Customized visualization to mimic DrawGeometry
   GLFW Error: X11: RandR gamma ramp support seems broken
   GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
   GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory
2. Customized visualization with a rotating view
   GLFW Error: X11: RandR gamma ramp support seems broken
   GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
   GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory
3. Customized visualization showing normal rendering
   GLFW Error: X11: RandR gamma ramp support seems broken
   GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
   GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory
4. Customized visualization with key press callbacks
   Press 'K' to change background color to black
   Press 'R' to load a customized render option, showing normals
   Press ',' to capture the depth buffer and show it
   Press '.' to capture the screen and show it
   GLFW Error: X11: RandR gamma ramp support seems broken
   GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
   GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory
5. Customized visualization playing a camera trajectory
   GLFW Error: X11: RandR gamma ramp support seems broken
   GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
   GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00000
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00001
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00002
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00003
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00004
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00005
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00006
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.
   Capture image 00007
   [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match.

[syncle]

After going trough lots of resources it would seem the segmentation fault is likely an issue introduced by the use of docker.

If the issue is due to docker, how about not using docker? You can make virtual machine instead. As @qianyizh have tested on Ubuntu systems, we just want to finalize this issue with GLFW3.3, instead of blocked by new issues regarding the docker.

[takanokage]

Actually that was a speculation on my part.
A later experiment showed a correlation between glfw 3.3 and the presence of the segmentation fault. The master branch works fine with docker.

[takanokage]

I have headless rendering working inside docker with the master branch of Open3D.
Next I'll be working on the documentation before submitting a pull request.

[takanokage]

Bind mounts (#254)
I propose we make the whole Open3D folder available both inside the docker container and on the host side. In this way we not only gain direct access to the images generated but we also enable developing Open3D on the host side and testing on both host side and docker side of things.

[takanokage]

Because we moved #254 under #17 I need to postpone the pull request until after I implement/test it.

[takanokage]

I got headless working together with bind mounting.
Bind mounting was a ton of work and should've been an independent pull request.

[takanokage]

Usage notes:

1. clone my branch
2. $ cd <Open3D path>/utilities/docker/ubuntu-xvfb/tools
3. $ ./build.sh to build the image
4. $ ./attach.sh to clone/build the master Open3D repo and start the Open3D container
5. $ ./headless_sample.sh to run the sample that renders some images and saves them to disk.
   The sample will render some images which can be accessed on the host at ~/Open3D_docker. These files don't go away if the container is stopped.

Notes:

• the sample will not return. Ctrl+c to exit. Need to update the sample to exit on it's own.
• the cloning is done for now to ~/Open3D_docker. We have options: find a way to let the user specify the destination or just reuse the current location of Open3D.
• TODO: uncomment entries in the dependencies section inside the Dockerfile.

[syncle]

Addressed in #273.