Open3d: can not capture screen using capture_screen_float_buffer

[yanghtr]

IMPORTANT: Please use the following template to report the bug.

---

Describe the bug
I try to use capture_screen_float_buffer to capture screen image with the following minimal code:

import numpy as np 
import matplotlib.pyplot as plt 
import open3d as o3d 

mesh = o3d.geometry.TriangleMesh.create_coordinate_frame() 
vis = o3d.visualization.Visualizer() 
vis.create_window(visible=False) 
vis.add_geometry(mesh) 
vis.poll_events() 
vis.update_renderer() 

color = vis.capture_screen_float_buffer(True) 
depth = vis.capture_depth_float_buffer(True) 
vis.destroy_window() 
color = np.asarray(color) 
depth = np.asarray(depth) 
plt.imshow(color) 
plt.show() 
plt.imshow(depth) 
plt.show()  

To Reproduce
Steps to reproduce the behavior:
run the code stated above

Expected behavior
The depth image can be captured correctly but the screen can not. The captured screen image is all black (0, 0, 0).

Screenshots
Outputs are as follows:
The depth image is correct:

BUT the color screen image is black:

Environment (please complete the following information):

• OS: Ubuntu 16.04
• Python version: 3.6
• Open3D version: 0.8.0.0
• Is this remote workstation?: no
• How did you install Open3D?: pip

Additional context
Related issue includes #1110 and #927. But the function is different and I don't need to save the image, I need to process the image directly. It seems that this is a similar bug unsolved.
I have also tried these similar methods, such as time.sleep() and capture_screen_image but they all do not work.

[sammo2828]

Not sure, but let's take a look at what is in your color array:

print(np.max(color), np.min(color))

[mels630]

@sammo2828 I've reproduced this issue. The output of

print(np.max(color), np.min(color))
print(np.max(depth), np.min(depth))

is

0.0 0.0
1.7997212 0.0

It seems clear that somehow the depth buffer fills properly but the screen buffer does not.

[yongxf]

Hi @yanghtr did you solve the problem?
I also have the same problem. It seems to me this is a bug.
When using vis.create_window(visible=False), the capture_depth_float_buffer returns the desired projection image but capture_screen_float_buffer returns the screen shot of the top window, instead of the created open3d window.
No problem if setting visible=True.

I installed open3d by pip recently in a ubuntu 16.04.
@syncle I checked #608 but it seems even the headless_rendering.py and voxel_carving.py has the same problem when using visible=False.

Thanks and looking forward to your reply!

[yanghtr]

@yongxf I did not solve this problem. I don't know how to solve it.

[heiwang1997]

One workaround I found is to add the following patch to Open3D source code and recompile the package. The idea is to create a new frame buffer object with a texture attachment to render to. It works for both visible=True and visible=False.

````diff
diff --git a/src/Open3D/Visualization/Visualizer/Visualizer.h b/src/Open3D/Visualization/Visualizer/Visualizer.h
index 7d23419..b8c9104 100644
--- a/src/Open3D/Visualization/Visualizer/Visualizer.h
+++ b/src/Open3D/Visualization/Visualizer/Visualizer.h
@@ -239,7 +239,7 @@ protected:
/// Function to do the main rendering
/// The function first sets view point, then draw geometry (pointclouds and
/// meshes individually).
- virtual void Render();
+ virtual void Render(bool texture_based = false);

 void CopyViewStatusToClipboard();

@@ -277,6 +277,11 @@ protected:
bool is_initialized_ = false;
GLuint vao_id_;

• // render targets (only for offscreen rendering when use texture = true)
• unsigned int render_fbo_;
• unsigned int render_rgb_tex_;
• unsigned int render_depth_stencil_rbo_;
  +
  // view control
  std::unique_ptr view_control_ptr_;

diff --git a/src/Open3D/Visualization/Visualizer/VisualizerRender.cpp b/src/Open3D/Visualization/Visualizer/VisualizerRender.cpp
index 085751d..d934f9d 100644
--- a/src/Open3D/Visualization/Visualizer/VisualizerRender.cpp
+++ b/src/Open3D/Visualization/Visualizer/VisualizerRender.cpp
@@ -44,6 +44,8 @@ bool Visualizer::InitOpenGL() {
return false;
}

• render_fbo_ = 0;
  +
  glGenVertexArrays(1, &vao_id_);
  glBindVertexArray(vao_id_);

@@ -64,11 +66,37 @@ bool Visualizer::InitOpenGL() {
return true;
}

-void Visualizer::Render() {
+void Visualizer::Render(bool texture_based) {
glfwMakeContextCurrent(window_);

 view_control_ptr_->SetViewMatrices();

• if (texture_based) {
• if (render_fbo_ != 0) {
• utility::LogError("Render Framebuffer is not released.");
• }
  +
• // Render to our texture buffer.
• int tex_w = view_control_ptr_->GetWindowWidth();
• int tex_h = view_control_ptr_->GetWindowHeight();
  +
• glGenFramebuffers(1, &render_fbo_);
• glBindFramebuffer(GL_FRAMEBUFFER, render_fbo_);
  +
• glGenTextures(1, &render_rgb_tex_);
• glBindTexture(GL_TEXTURE_2D, render_rgb_tex_);
• glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, tex_w, tex_h, 0, GL_RGB, GL_FLOAT, NULL);
• glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
• glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
• glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, render_rgb_tex_, 0);
  +
• glGenRenderbuffers(1, &render_depth_stencil_rbo_);
• glBindRenderbuffer(GL_RENDERBUFFER, render_depth_stencil_rbo_);
• glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8, tex_w, tex_h);
• glBindRenderbuffer(GL_RENDERBUFFER, 0);
• glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER, render_depth_stencil_rbo_);

• }
  +
  glEnable(GL_MULTISAMPLE);
  glDisable(GL_BLEND);
  auto &background_color = render_option_ptr_->background_color_;
  @@ -145,11 +173,12 @@ void Visualizer::CopyViewStatusFromClipboard() {

  std::shared_ptr Visualizer::CaptureScreenFloatBuffer(
  bool do_render /* = true*/) {
  +
  geometry::Image screen_image;
  screen_image.Prepare(view_control_ptr_->GetWindowWidth(),
  view_control_ptr_->GetWindowHeight(), 3, 4);
  if (do_render) {

• Render();

• Render(true);
  is_redraw_required_ = false;
  }
  glFinish();
  @@ -157,6 +186,14 @@ std::shared_ptr Visualizer::CaptureScreenFloatBuffer(
  view_control_ptr_->GetWindowHeight(), GL_RGB, GL_FLOAT,
  screen_image.data_.data());

• if (render_fbo_ != 0) {

• glBindFramebuffer(GL_FRAMEBUFFER, 0);
• glDeleteFramebuffers(1, &render_fbo_);
• glDeleteRenderbuffers(1, &render_depth_stencil_rbo_);
• glDeleteTextures(1, &render_rgb_tex_);
• render_fbo_ = 0;
• }
  +
  // glReadPixels get the screen in a vertically flipped manner
  // Thus we should flip it back.
  auto image_ptr = std::make_shared();
  ````

[yongxf]

@heiwang1997 Thanks alot, now the capture_screen_float_buffer(true) is working with vis.create_window(visible=False). @yanghtr I did exactly @heiwang1997 mentioned and recompile the source code, now the capture_screen_float_buffer(true) is working.

[yanghtr]

It works. Thank you very much!

[martinakos]

Have you made a PR with this change?

[SBCV]

The error is still not fixed in 0.10.0 - can you reopen this issue?

[heiwang1997]

@SBCV Thank you very much for the pull request.

[SBCV]

@heiwang1997 thank you for providing a solution ;)