Open3d: Python multiprocessing and Visualizer

[samarth-robo]

I want to have an application where I modify geometry in one thread and update the Visualizer in another thread. Is that possible?
I guess I am looking for a version of the non-blocking visualization example where the ICP runs in one thread and updates the geometry, while the visualizer updates are run in a separate thread.

Here is what I have so far:

"""
Multi-threaded open3D viewer
"""
import open3d
import multiprocessing as mp

class Consumer(mp.Process):
  def __init__(self, vis_q):
    super(Consumer, self).__init__()
    self.vis_q = vis_q

  def run(self):
    while True:
      next_vis = self.vis_q.get()
      if next_vis is None:
        break

      next_vis.run()
      next_vis.destroy_window()

    return


class Viewer(object):
  def __init__(self):
    self.vis = {}

  def add_geometry(self, g, window_name):
    if window_name in self.vis:
      self.vis[window_name].add_geometry(g)
    else:
      self.vis[window_name] = open3d.Visualizer()
      self.vis[window_name].create_window(window_name, width=640, height=480)
      self.vis[window_name].add_geometry(g)

  def show(self, block=True):

    q = mp.JoinableQueue()
    num_workers = mp.cpu_count() * 2
    workers = [Consumer(q) for _ in xrange(num_workers)]
    for w in workers:
      w.start()

    for vis in self.vis.values():
      q.put(vis)

    for _ in xrange(num_workers):
      q.put(None)

    if block:
      q.join()

if __name__ == '__main__':
  pc1 = open3d.read_point_cloud('../data/camera/000.pcd')
  pc2 = open3d.read_point_cloud('../data/camera/001.pcd')

  v = Viewer()
  v.add_geometry(pc1, 'window1')
  v.add_geometry(pc2, 'window2')
  v.show()

It creates the two windows but they are blank and hang.

My idea was that once this works, I can externally modify the added geometry g and those changes should be automatically reflected in the visualization.

[qianyizh]

This is doable but is a bit complicated.

First, due to a limitation of glfw, the Visualizer class has to be handled in the main thread. So you should structure your program as main thread for rendering and worker threads for updating geometry.

Second, make sure you understand how the rendering loop works. See comments in https://github.com/IntelVCL/Open3D/pull/343 and https://github.com/IntelVCL/Open3D/issues/357. In short, the rendering loop acts as:

1. Call poll_events() to (a) render if necessary, and (b) check if there is mouse/keyboard event that needs to be responded
2. Bind geometry if necessary (call vis.update_geometry() to re-bind)
3. Bind OpenGL shader if necessary (call vis.update_renderer() to re-bind and render)
4. Go back to 1

This needs to be in the main thread.
If poll_events() is not called, the windows will stuck as happens in your case. If you needs multiple windows, they need to be in the same thread, see my example in https://github.com/IntelVCL/Open3D/issues/357

Finally, since you want to edit the geometry while rendering at the same time, there is indeed a race condition between editing and re-bind geometry. A lock needs to be introduced.

A sample pseudo code is as follows:

# in worker thread
while 1:
    obtain_lock(pc_i)
    do_something(pc_i)
    is_dirty_pc_i = True
    release_lock(pc_i)

# in main thread
vis1 = Visualizer()
vis1.create_window()
vis1.add_geometry(pc_1)
vis2 = Visualizer()
vis2.create_window()
vis2.add_geometry(pc_2)
while True:
    obtain_lock(pc_1)
    obtain_lock(pc_2)
    if is_dirty_pc_1:
        vis1.update_geometry()
    if is_dirty_pc_2:
        vis2.update_geometry()
    vis1.update_renderer()
    vis2.update_renderer()
    vis1.poll_events()
    vis2.poll_events()
    is_dirty_pc_1 = False
    is_dirty_pc_2 = False
    release_lock(pc_1)
    release_lock(pc_2)
vis1.destroy_window()
vis2.destroy_window()

There might be some issues though. And the main thread may get blocked a lot. I would still suggest to restructure the entire thing so that the worker thread can work in the main thread, and update the window once in a while. Something like:

vis1 = Visualizer()
vis1.create_window()
vis1.add_geometry(pc_1)
vis2 = Visualizer()
vis2.create_window()
vis2.add_geometry(pc_2)
while True:
    worker_run()
    vis1.update_geometry()
    vis2.update_geometry()
    vis1.update_renderer()
    vis2.update_renderer()
    vis1.poll_events()
    vis2.poll_events()
vis1.destroy_window()
vis2.destroy_window()

[samarth-robo]

@qianyizh, thanks for your help, and sorry for the late reply. This is what finally worked for me:

import open3d
import multiprocessing as mp
import time
import numpy as np
import transforms3d.euler as txe 
from Queue import Empty as queue_empty

class Viewer(object):
  def __init__(self):
    self.q = mp.Queue()

  def worker(self, q): 
    for _ in range(5):
      time.sleep(1)
      T = np.eye(4)
      T[:3, :3] = txe.euler2mat(np.deg2rad(20), 0, 0)
      q.put(T)
      print('put')
    q.put(None)  # poison pill

  def run(self):
    pc = open3d.read_point_cloud('narf/lidar_processed.pcd')
    vis = open3d.Visualizer()
    vis.create_window('cloud', width=640, height=480)
    vis.add_geometry(pc)

    p = mp.Process(target=self.worker, args=(self.q, ))
    p.start()

    keep_running = True
    while keep_running:
      try:
        T = self.q.get(block=False)
        if T is not None:
          print('got T')
          pc.transform(T)
        else:
          print('got poison. dying')
          keep_running = False
        vis.update_geometry()
      except queue_empty:
        pass
      vis.update_renderer()
      keep_running = keep_running and vis.poll_events()
    vis.destroy_window()
    p.join()

if __name__ == '__main__':
  v = Viewer()
  v.run()

I tried making pc a class member, and transforming it in self.worker, but that change was not reflected in the visualizer. I think when you make a mp.Process it might make a copy of pc and the worker might be modifying it's own copy.
So I ended up sending transforms from the worker using a Queue. This eliminates the need for a lock or a cloud_updated flag.

[samarth-robo]

Also, it is probably worth mentioning somewhere in the docs that poll_events() returns a flag that indicates whether the window should be closed. That can be very handy when someone wants to manage the rendering loop themselves.