Viewers: Segmentation UI for VTKjs.

[JamesAPetts]

Per IDC discussion yesterday:

We need the following 4 features for the segmentation panel whilst MPR 2D is open:

• Show/hide segment.
• Switch segmentation rendering options.
• Jump to segment.
• Switch segmentation.

Most of these operations require retreiving the API for the viewport, which can be grabbed from here:

https://github.com/OHIF/Viewers/blob/master/extensions/vtk/src/commandsModule.js#L103

Show/Hide segment

api.setSegmentAlpha can be used to set the segment alpha to 255 or 0. This is multiplied by the global alpha, so you can use this to show/hide segments. (In the future we may need a show/hide + individual segment opacity, but this will work for now).

Switch segmentation rendering options.

• Most of these are passed down as props here.
• Perhaps they should instead by API endpoints in the react-vtkjs-viewport, might have to have a look at how best to implement this.

Jump to segment:

We already have logic for finding the middle slice of the segment.

• Look through this slice and find the mean voxel position of that slice for the segment.
• Convert this voxel position to the world coordinates of the volume you can do this from the vtkImageData by getting its getIndexToWorld matrix.
• Add 0.5, 0.5, 0.5 to the index, before converting to world coordinates, so that we may jump to the middle of the voxel.
• On one API set the world position of the crosshairs, they are synced so this will update all 3 and jump the images to this location.

const api = apis[0];
api.svgWidgets.crosshairsWidget.moveCrosshairs =(worldPos, apis, 0)

Switch segmentation:

For this you'll need to create+cache another labelmap volume in the process that ussually happens in extensions/vtk/src/OHIFVTKViewport.js when props are initially passed in, and then update the props. The react-vtkjs-viewport should already support the switching of volumes once this labelmap is changed, but this hasn't had much testing/usage, so this may require some changes to the original library.

As this has not yet been discussed, we will leave out rendering of multiple segmentations in vtkjs for now, as this is a tricky and memory intensive issue that I don't think is important for the MVP. CC: @fedorov

Could @fedorov / @pieper verify this is as discussed/good to work on?

[pieper]

Looks right to me, thanks @JamesAPetts 👍

[igoroctaviano]

• We need to check what the labelmap options for the volume mapper have at the vtk.js level.
  
  
  
  • For this iteration, I'm setting the segment opacity (alpha) to its max and its min in order to accomplish show/hide.
  
  
• Currently, if the user changes the opacity of the fill to 0 in the segmentation configuration, the outline disappears. Probably a limitation in vtk.js.

@pieper / @fedorov Unfilled segment with outline is currently priority or this should be left until later?

[pieper]

@pieper / @fedorov Unfilled segment with outline is currently priority or this should be left until later?

I'd say not a high priority compared to ironing out UI quirks.

[fedorov]

we will leave out rendering of multiple segmentations in vtkjs for now

What will be the behavior when there are multiple segmentations? Did you mean segmentations or segments?

What will be the differences and how will they be communicated to the user to reduce confusion?

[JamesAPetts]

What will be the behavior when there are multiple segmentations? Did you mean segmentations or segments?

I mean segmentations. The active one will render, and you will be able to switch which segmentation is currently visible. This problem is degenerate with allowing overlapping segmentations in vtk. I've talked about this with @pieper before, but dcmjs currently assumes we have non-overlapping segments.

What will be the differences and how will they be communicated to the user to reduce confusion?

When the user clicks the cog icon to change the settings whilst MPR2D is active, there will be no options for "Render inactive segmentations", this will only show in 2D mode.

Appreciate any feedback on this aproach or whether any of this should priority instead of leaving it out 👍 .

[fedorov]

This problem is degenerate with allowing overlapping segmentations in vtk

That's what I suspected, and that's why I wanted to clarify segmentations vs segments. It is allowable to have overlapping segments within a single segmentation - how will that be supported?

I don't know if we have any datasets that would have overlapping segments within segmentation, so I can't say if this is in a critical path for MVP, but I wanted to clarify this point.

What will be the differences and how will they be communicated to the user to reduce confusion?

When the user clicks the cog icon to change the settings whilst MPR2D is active, there will be no options for "Render inactive segmentations", this will only show in 2D mode.

👍

[JamesAPetts]

It is allowable to have overlapping segments within a single segmentation - how will that be supported?

We can have multiple volumes in vtk, so in theory its just a matter of stacking volumes and sorting out the rendering order. The react-vtkjs-viewport api will need to be updated in order to support this, but the complications are actually mostly memory concerns.

Right now we have a viewport that is an isolated WebGL context, and even though we are displaying the same data, in the 3-UP view we are actually duplicating the textures as these can't be shared across contexts. We plan to eventually switch react-vtkjs-viewport to use one WebGL canvas with multiple render windows in it in a configurable layout, but this is quite a complex and large redesign and hasn't been on our radar yet. However, this would free up the memory by a factor of 3, and allow a few more labelmaps to be used, which allow for at least some degree of overlapping segments in vtkjs.

[pieper]

Let's also keep in mind the option of having one offscreen render context and then blitting the results into the onscreen viewports. Probably simpler logic.

[JamesAPetts]

That's also a nice approach, it remains that either case would require a ground up remaster of the viewport (and perhaps some of vtkjs) though.

[fedorov]

Until that functionality is supported, is it possible to detect situations when there are overlapping segments within a segmentation, and show them in the segmentations list in a way that makes it clear that they are not supported?