Open3d: Multiple viewports per window

[sammo2828]

A common use case is comparing multiple point clouds side-by-side. In PCL, this could be done by creating multiple viewports in the same window. However, Open3D Visualizer only provides a single window with a single viewport and a single ViewControl.

Feature request to extend the API to be able to attach multiple viewports to a window.

With the current API, a workaround is to create multiple windows with multiple Visualizer instances:

vis = o3d.Visualizer()
vis.create_window(window_name='TopLeft', width=960, height=540, left=0, top=0)
vis.add_geometry(pcd)

vis2 = o3d.Visualizer()
vis2.create_window(window_name='TopRight', width=960, height=540, left=960, top=0)
vis2.add_geometry(pcd)

while True:
    vis.update_geometry()
    if not vis.poll_events():
        break
    vis.update_renderer()

    vis2.update_geometry()
    if not vis2.poll_events():
        break
    vis2.update_renderer()

vis.destroy_window()
vis2.destroy_window()

[qianyizh]

Yes, we are working on a better GUI that can have multiple viewports.
Meanwhile the workaround should work.

[hduonggithub]

Hi @qianyizh
Do we have multiple viewports yet? if yes, please advise. Thanks!

I did try the proposed code above, and got error with update_geometry(). How can I make it works?

[MaxChanger]

The new version of open3d (0.9.0 and 0.10.0) needs to add the parameters you want to update in update_geometry().
You can check here

vis.add_geometry(src_pcd)
while True:
    vis.update_geometry(src_pcd)
    if not vis.poll_events():
        break
    vis.update_renderer()

[hduonggithub]

Thanks @MaxChanger. It works now.

My 2nd question is it possible to have each viewport window with interaction features like draw_geometries_with_editing

[MaxChanger]

@hduonggithub It doesn't seem to be working properly. It can normally display like a Visualizer(), but only seems to show one geometry (Visualizer() can add multiple geometries in list), and I cannot edit it, a segfault occurs when I try to select a point, like this

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Do you have any test results?

[hduonggithub]

@MaxChanger I could display multiple windows and pick up a point but could not edit or even draw a polygon.
Below is what I tried. [Yes] is working, [N] is not working

[Yes] F : Enter freeview mode.
[Yes] X : Enter orthogonal view along X axis, press again to flip.
[Yes] Y : Enter orthogonal view along Y axis, press again to flip.
[Yes] Z : Enter orthogonal view along Z axis, press again to flip.
[Yes] K : Lock / unlock camera.
[Yes] Ctrl + D : Downsample point cloud with a voxel grid.
[Yes] Ctrl + R : Reset geometry to its initial state.
[Yes] Shift + +/- : Increase/decrease picked point size..
[Yes] Shift + mouse left button : Pick a point and add in queue.
[-] Shift + mouse right button : Remove last picked point from queue.

[Yes] -- When camera is locked --
[No] Mouse left button + drag : Create a selection rectangle.
[No] Ctrl + mouse buttons + drag : Hold Ctrl key to draw a selection polygon.
[No] Left mouse button to add point. Right mouse
[No] button to remove point. Release Ctrl key to
[No] close the polygon.
[No] C : Crop the geometry with selection region.

[MaxChanger]

@hduonggithub Hello，the version I used was 0.9.0, some test results and code are as follows, I guess yours is 0.10.0 ?

It's worth noting that I can't pick a point, but you can.

[Yes] F : Enter freeview mode.
[Yes] X : Enter orthogonal view along X axis, press again to flip.
[Yes] Y : Enter orthogonal view along Y axis, press again to flip.
[Yes] Z : Enter orthogonal view along Z axis, press again to flip.
[Yes] K : Lock / unlock camera.
[Yes] Ctrl + D : Downsample point cloud with a voxel grid.
[Yes] R : Reset geometry to its initial state. [Not Ctrl + R]
[Yes] +/- : Increase/decrease picked point size.. [Not Shift + +/-]

------failure ------
[No] Shift + mouse left button : Pick a point and add in queue.
[-] Shift + mouse right button : Remove last picked point from queue.

vis = o3d.visualization.VisualizerWithEditing()
vis.create_window(window_name='TopLeft', width=600, height=540, left=0, top=0)
vis.add_geometry(pcd)
# vis.add_geometry(point_mesh_sphere)

vis_result = o3d.visualization.VisualizerWithEditing()
vis_result.create_window(window_name='TopRight', width=600, height=540, left=800, top=0)
vis_result.add_geometry(pcd)

while True:
    vis.update_geometry(pcd)
    if not vis.poll_events():
        break
    vis.update_renderer()

    vis_result.update_geometry(pcd)
    if not vis_result.poll_events():
        break
    vis_result.update_renderer()

vis.destroy_window()
vis_result.destroy_window()

[hduonggithub]

@MaxChanger Yes, I use 0.10.0. I have the similar code, and got the same error (still can only pick one point and window closed immediately).

I did try to add 2 geometries as below to the same vis, but not able show with VisualizerWithEditing(), still can show with Visualizer() only.

points = [[ -551.84605701, -7629.73163958, 200.53585874],
[ -589.15420551, -5505.33706683, 159.12506382],
[ -590.45763989, -5427.83561194, 2598.92887357],
[ -552.75708436, -7574.54707034, 2661.2278636 ]]

lines = [[0, 1],
[1, 2],
[2, 3],
[3, 1]]

colors = [[1, 0, 0] for k in range(len(lines))]
line_set = o3d.geometry.LineSet(
points=o3d.utility.Vector3dVector(points),
lines =o3d.utility.Vector2iVector(lines),
)
points_pc = o3d.geometry.PointCloud()
points_pc.points = o3d.utility.Vector3dVector(points)
points_pc.paint_uniform_color(color[0])

Case 1:
o3d.visualization.draw_geometries_with_editing([line_set, points_pc])

Error 1:
[Open3D WARNING] [DrawGeometriesWithEditing] Failed adding geometry.
[Open3D WARNING] [DrawGeometriesWithEditing] Possibly due to bad geometry or wrong geometry type.

Case 2:
vis = o3d.visualization.Visualizer()
vis.create_window(window_name='TopLeft', width=960, height=540, left=0, top=0)
vis.add_geometry([line_set, points_pc])

Error 2:
Exception has occurred: TypeError
add_geometry(): incompatible function arguments. The following argument types are supported:
1. (self: open3d.open3d_pybind.visualization.Visualizer, geometry: open3d.open3d_pybind.geometry.Geometry, reset_bounding_box: bool = True) -> bool

Invoked with: Visualizer with name TopLeft, [geometry::LineSet with 4 lines., geometry::PointCloud with 4 points.]
File "D:\Open3D\examples\python\test_multiview.py", line 37, in
vis.add_geometry(pcd1)

Btw, I wonder why these features not available in Visualizer() only or o3d.visualization.draw_geometries(). They are very useful.
F : Enter freeview mode.
X : Enter orthogonal view along X axis, press again to flip.
Y : Enter orthogonal view along Y axis, press again to flip.
Z : Enter orthogonal view along Z axis, press again to flip.

[MaxChanger]

Indeed, after picking a point, it will be closed immediately, but the ID of the selected point will be shown in the terminal with the above errors. It seems that there is no difference between 0.9.0 and 0.10.0.

VisualizerWithEditing() can only add one geometry, I also found this problem, unfortunately, I did not find a solution.

I think one reason may be that if there are two geometries at the same time, it is very likely that you don’t know which geometry point you picked.

[MaxChanger]

@hduonggithub If you only want to display many point clouds and pick some points, can you concate the two point clouds together and spend some experience to save the number of each point cloud, and then distinguish it by the returned Point ID ?

But I think this is not the optimal solution, and it will not work for mesh.

In addition, displaying two windows at the same time seems to have some problems with the response to the keyboard keys. For example, I did not find a way to close one and keep the other open. When you press esc or q, both windows will be closed. #1917

[MaxChanger]

@hduonggithub The parameter of vis.add_geometry() cannot be a list, only Geometry, you can refer to the official documentation. But, you could use

vis.add_geometry(pcd)
vis.add_geometry(point_mesh_sphere)

to add multi-geometries to the Visualizer(), but will only show the first one in VisualizerWithEditing() I founded.

[hduonggithub]

@MaxChanger

@hduonggithub The parameter of vis.add_geometry() cannot be a list, only Geometry, you can refer to the official documentation. But, you could use

vis.add_geometry(pcd)
vis.add_geometry(point_mesh_sphere)

to add multi-geometries to the Visualizer(), but will only show the first one in VisualizerWithEditing() I founded.

Completely right.

I am looking for how to turn these features on Visualizer() only. Do you have any idea?
F : Enter freeview mode.
X : Enter orthogonal view along X axis, press again to flip.
Y : Enter orthogonal view along Y axis, press again to flip.
Z : Enter orthogonal view along Z axis, press again to flip.

[MaxChanger]

VisualizerWithEditing() inherits from Visualizer(), and if you want to implement these features, you may need to modify its c++ code.

But I don't know if there's a conflict, so why don't they just provide it.

[prewettg]

This is not possible with the current visualizer in 0.11. With the new rendering and gui module it is possible to have multiple scenes, as long as they are different scenes. Your stated use case would work in this case: create one scene with point cloud 1, and then create a second scene with point cloud 2, then put them next to each other. I have included an example below. If you need multiple viewports on the same scene, we are working on that, hopefully for 0.12.

#!/usr/bin/env python

import open3d as o3d
import open3d.visualization.gui as gui
import numpy as np

def make_box():
    box = o3d.geometry.TriangleMesh.create_box(1, 2, 4)
    box.compute_vertex_normals()
    return box

def main():
    gui.Application.instance.initialize()

    mat = o3d.visualization.rendering.Material()
    mat.base_color = (1.0, 0.0, 0.0, 1.0)
    mat.shader = "defaultLit"

    w = gui.Application.instance.create_window("Two scenes", 1025, 512)
    scene1 = gui.SceneWidget()
    scene1.scene = o3d.visualization.rendering.Open3DScene(w.renderer)
    scene1.scene.add_geometry("cube1", make_box(), mat)
    scene1.setup_camera(60, scene1.scene.bounding_box, (0, 0, 0))
    scene2 = gui.SceneWidget()
    scene2.scene = o3d.visualization.rendering.Open3DScene(w.renderer)
    mat.base_color = (0.0, 0.0, 1.0, 1.0)
    scene2.scene.add_geometry("cube1", make_box(), mat)
    scene2.setup_camera(60, scene2.scene.bounding_box, (0, 0, 0))

    w.add_child(scene1)
    w.add_child(scene2)

    def on_layout(theme):
        r = w.content_rect
        scene1.frame = gui.Rect(r.x, r.y, r.width / 2, r.height)
        scene2.frame = gui.Rect(r.x + r.width / 2 + 1, r.y, r.width / 2, r.height)

    w.set_on_layout(on_layout)

    gui.Application.instance.run()

if __name__ == "__main__":
    main()

[hduonggithub]

Thanks @prewettg

[MaxChanger]

The development is real and rapid. I started using version 0.8.0, and now I am stuck at 0.9.0. Some functions in the above example are already unfamiliar. Hahaha

By the way, I used PyVista to solve my needs @hduonggithub

[hduonggithub]

@MaxChanger Thanks. It is good to know!

[fabioruetz]

@prewettg Thanks a lot! Is there a way to synchronize the camera/viewpoints over the different viewports? I.e I would like to move one the viewpoint in one window and have the other follow the movement.

Cheers Fabio

[prewettg]

@fabioruetz Unfortunately there is no way to synchronize them at the moment. You'd need to monitor mouse movement so that you can update the appropriate camera, but that has not been exported to Python yet, although it's on my list of things to do. I just created a specific issue for this, #2610, if you want to follow.

[prewettg]

You can now add Python callbacks for mouse events on SceneWidget, so you should be able to synchronize multiple scenes now. Please see #2320 for an example of the callback usage. (For any C++ visitors from Google, you don't need any callbacks, just inherit from SceneWidget and override OnMouse()). This is available in the master branch now and will be in 0.12.