Vtk-js: interaction with actors - mouse events - performance

[doczoidberg]

Is it possible to fire events e.g. when the user clicked on an actor or the mouse is over an actor?

This is possible in vtk but not in vtk-js?

[jourdain]

Look at https://github.com/Kitware/vtk-js/blob/master/Sources/Rendering/OpenGL/HardwareSelector/test/testHardwareSelector.js

[doczoidberg]

.

[doczoidberg]

x,y coordinates seem to be wrong for me, any ideas?

<div (mousedown)="mousedown($event)" (mousemove)="mousemove($event)" #vtkd id="vtkdiv" class="vtkdiv" style="width:400px;height:300px;">

    this.myvtk.sel = vtkOpenGLHardwareSelector.newInstance();
    this.myvtk.sel.setFieldAssociation(FieldAssociations.FIELD_ASSOCIATION_CELLS);
    this.myvtk.sel.attach(this.myvtk.openGlRenderWindow, this.myvtk.renderer);

  public mousedown(event) {
    console.log('mousedown ', event.screenX, ' ', event.screenY); //right
    this.myvtk.sel.setArea(event.screenX, event.screenY, event.screenX + 2, event.screenY + 2); // wrong??
    const res = this.myvtk.sel.select();
    console.log('sel.select(): ', res); // object output at wrong positions
  }

[jourdain]

screenX/Y won't work as they give coordinates based on the screen position not the (x,y) relative to the element.

You may have to use something like clientX and clientY. See doc to be sure.

Then you may have to reverse Y as VTK may have a 0 at the bottom, not at the top.

[doczoidberg]

ok, thanks. offsetX, offsetY btw.

[doczoidberg]

performance seems quite bad for setArea. Are there any opportunities to calculate it quicker so that I can implement smooth mouse over effects via mousemove event?

[martinken]

I don't have the code in front of me but you should be capturing the buffers on mouse down and then just reading the buffers during mouse movement. If you use the same code every move, then that means you are capturing the buffers every move which is expensive.

[doczoidberg]

not sure what you mean. my code on mousedown/mousemove is just:

    this.myvtk.sel.setArea(event.offsetX, 600 - event.offsetY, event.offsetX, 600 - event.offsetY);
    var res = this.myvtk.sel.select();

[martinken]

Adding the following to the api.md for that class

The hardware selector works by rendering the current scene using multiple passes in a special way where each pass stores information into the color buffer. Once this is done the color buffer for these passes can be quickly queried to retrieve information such as prop, prop id, composite id, process id, etc depending on what passes were captured. We do not always capture every pass, for example on a single process run there is no need to capture a process id pass.

Once the buffers are captured you can then call generateSelection to get the results for a specific x,y, area. Here are two examples of making a selection

const hws = vtkOpenGLHardwareSelector.newInstance();
hws.attach(myOpenGLRenWin, renderer);
hws.setArea(...);
const selection = hwl.select();

select() will capture the bufferes needed and then call generateSelection() on the area specified in setArea. If you want to perform multiple selections and you know your image is not changing, then you can capture the buffers once and then make multiple generateSelection calls for example

const hws = vtkOpenGLHardwareSelector.newInstance();
hws.attach(myOpenGLRenWin, renderer);
if (hws.captureBuffers()) {
  while (...) {
    // do a bunch of selections
    const selection = hwl.generateSelection(x,y,x2,y2);
    // do something with it
  }
  // all done, free up the captured buffers
  hws.releasePixBuffers();
}

[doczoidberg]

I have changed the code as above but I get only an empty array from generateSelection.

I have multiple polydata meshes and want to highlight the one the mouse is over.

[martinken]

I don;t have time to look into this, but you can certainly hop in the debugger and let me know what you find.

[doczoidberg]

in publicAPI.getPixelInformation i get null back because the first if returns null:

  if (inDisplayPosition[0] < model.area[0] || inDisplayPosition[0] > model.area[2] ||
        inDisplayPosition[1] < model.area[1] || inDisplayPosition[1] > model.area[3]) {
        return null;
      }

this is because model.area could be wrong?

Array[4]
0:410
1:218
2:410
3:218

[martinken]

yeh, try call setArea to the entire window size before your CaptureBuffers call

[doczoidberg]

edit: rendering is still slow. Seems that the renderwindow is updated after caputreBuffersand not after render()call. edit2: there is no need for calling the render function.

edit3: the window size/area is the main bottleneck. This should be much faster possible? At least when I click on a polydata actor the highlight should be immediately. This should be possible with OpenGL?

  public mouseup(event) {
    this.myvtk.sel.setArea(1, 1, 600, 600);
    this.myvtk.sel.captureBuffers();
  }

 public mousemove(event) {
    console.log('mousemove')

    var res = null;

        res = this.myvtk.sel.generateSelection(event.offsetX, 600 - event.offsetY, event.offsetX, 600 - event.offsetY);

      if (res)
        if (res[0]) {
          var actor = res[0].get().properties.prop.getActors();

          for (var i = 0; i < this.myvtk.actors.length; i++) {
            if (this.myvtk.actors[i] == actor && actor != this.lastactor) {
              this.lastactor = actor;
              this.myvtk.sel.releasePixBuffers();

              for (var j = 0; j < this.myvtk.actors.length; j++)
                 this.myvtk.actors[j].getProperty().setColor(0.5, 0.5, 0.8);
              this.myvtk.actors[i].getProperty().setColor(0, 1, 1);
              console.log('caputreBuffers')
              this.myvtk.renderWindow.render();
              this.myvtk.sel.setArea(1, 1, 600, 600);
              this.myvtk.sel.captureBuffers();
            }
          }
        }
  }

[jourdain]

I'm not sure the code snippet you've posted is still relevant but based on @martinken comment, you obviously don't want to do captureBuffers() and releasePixBuffers() on mousemove and even less inside a loop of that method.

[doczoidberg]

it is only executed when a the mouse is over a new actor. IMO the funciton is to slow. Also for mousedown it is not fast enough

[jourdain]

Ok, but why? You still don't need to recapture anything since the view didn't change?

[doczoidberg]

there is even a lag when rotating the scene (then I need a new buffer capture as far as I understand. I set the new buffer capture on mouseup event. Then I can't rotate for around 1sec what is bad usability.

[jourdain]

I agree that the capture could/should be faster to prevent the lag you are mentioning.

But I was focusing on the code snippet above that should not require any captureBuffers() and releasePixBuffers() when you are moving the mouse as overlay.

[doczoidberg]

I think I even than need it because I change color of an actor and therefore the scene

[jourdain]

You are not changing depth in pixels. Changing the color of an actor does not affect its position in the scene.

[jourdain]

I've used it in a custom application and it was fast when capturing the buffer at mouse up and using the same buffer on the mouse move.