Mne-python: picking vertex in brain object to see time course

[agramfort]

I would be great to get a plot of a time course of a vertex on a 3D brain object.

a suggestion from @SophieHerbst is also to have a short cut to pick automatically the vertex with the highest amplitude in the current view.

@GuillaumeFavelier

[larsoner]

I would be great to get a plot of a time course of a vertex on a 3D brain object.

At least in principle there is vtkChart:

• https://vtk.org/doc/nightly/html/classvtkChart.html
• https://vtk.org/Wiki/VTK/Examples/Cxx/Plotting/LinePlot

a suggestion from @SophieHerbst is also to have a short cut to pick automatically the vertex with the highest amplitude in the current view.

I'm actually not sure the extent to which this will be possible. Picking with the mouse is usually done by rendering tricks or casting a ray (I think VTK does the latter). I'm not sure there is a way to ask for "all mesh vertices currently visible). And given a non-zero alpha, this isn't even necessarily well defined. So for a first pass I say we just make it so that clicking on a point in the mesh does mesh vertex picking and does a vtkChart plot of the time course. (If off the mesh, don't change the point currently used. This way clicking always will show the time course, and if you don't want it to change, click in the empty space around the mesh. Either that or use ctrl-click for picking or something.)

I leave it as an exercise to @GuillaumeFavelier to figure out where this plot should live :) Maybe a show_timecourse=True would make a new full-width (regardless of views) plot area at the bottom reserved for a time course chart like our volume viewer:

https://mne.tools/dev/auto_examples/inverse/plot_lcmv_beamformer_volume.html#plot-source-space-activity

[jasmainak]

why not extend the functionality of 4D viewer with cortical ribbon first?

[larsoner]

It's nicer to view surface source estimate data on the surface

[GuillaumeFavelier]

That's a nice idea and thank you so much for the pointers @larsoner, I have somewhere to start now.

[larsoner]

And for mouse click vertex picking in mayavi (so you can translate to PyVista) see

https://github.com/mne-tools/mne-python/blob/master/mne/gui/_fiducials_gui.py#L359

[GuillaumeFavelier]

Vertex picking is available by default in PyVista:

[SophieHerbst]

Would this allow to spit out MNI coordinates at the pick (or the label of a pre-defined atlas)?

[SophieHerbst]

Also, a very small add-on: labels for the 3 clim-scales would be helpful.

[GuillaumeFavelier]

I think any data associated to the point data/vertex id could be retrieved that way in theory.

[GuillaumeFavelier]

I discovered vertex_to_mni() to take care of the conversion.

[GuillaumeFavelier]

Also, a very small add-on: labels for the 3 clim-scales would be helpful.

fmax/fmid/fmin labels were present in previous version but were dropped in favour of a more compact UI.

[GuillaumeFavelier]

ToDo (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-577954381)

• [x] Display the MNI coordinates at the pick (suggested in https://github.com/mne-tools/mne-python/issues/7235#issuecomment-577898442)
• [x] Enable Picking & Plot of time courses with show_traces=True
• [x] Display the picked point in 3d with a dot glyph (sphere)
• [x] Add a time course to the chart by picking on the surface vertex
• [x] Remove an existing time course in the chart by clicking on the dot glyph
• [x] Display the chart with vtkChartXY in split view or embed the MPL figs in a Qt frame
• [x] Show the time course of the vertex with max(abs(t)) by default (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-579380243)
• [x] Use mouse click instead of p to trigger picking (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-579416421)
• [x] Remove the red bounding box appearing during picking
• [x] Update the 3d backend overview table
• [x] Use cell picking instead of point picking (expecting better precision)
• [x] Use a reasonable radius for the glyphs (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-580322104)
• [x] Scale when hemi=both or hemi=split
• [x] Use show_traces=separate to detach the matplotlib canvas (suggested in https://github.com/mne-tools/mne-python/issues/7235#issuecomment-579773408)
• [x] Add support for views (suggested in https://github.com/mne-tools/mne-python/pull/7247#issuecomment-583001281)
• [x] Add testing of picking features (suggested in https://github.com/mne-tools/mne-python/pull/7247#pullrequestreview-356689608)

[larsoner]

+10 to embedding a mpl figure rather than figuring out vtkChartXY, it gives us more flexibility going forward, e.g., including showing time-frequency images eventually, and will unify the user experience of 2D plots around MPL and its standard interactions (like zooming and view resetting). The "only" tricky parts of this approach will be:

• setting up our own Qt window instead of relying on PyVista and MPL to do it for us, and then also
• eventually making sure that when in a notebook we embed the canvases properly rather than creating a Qt window

[mmagnuski]

@larsoner Why would we have to set up our own Qt window? At least in interactive matplotlib mode - it might be easier and more flexible to open a separate matplotlib window per click/interaction.

[larsoner]

I don't think that would work as well. The design I'm advocating is basically a variant of what we already do in the volume plotter. You could apply your argument to that as well -- that we shouldn't render the time trace(s) in the same window as the brain volume rendering -- and I don't think it would be as nice as experience as having them contained in the same window.

[mmagnuski]

Yes, that's true - separate window can feel more clunky, but the benefit is that the user can place it however he wants. And although volume source plotter shows traces in the same window - many other mne interfaces open separate windows (time-frequency - when interacting with topomap [also tfr heatmap IIRC], evokeds - when selecting time window, ica source signal viewer when clicking on ic label, ica plot components opens properties windows). With separate windows you can also compare various traces without the need to implement specific option for that - you click on two vertices / channels and see corresponding signals in two separate windows.
In general I agree that it might be nicer to have everyting in one plot - my previous comment was mostly motivated by perceived difficulity of this goal. It seems easier to spawn a separate matplotlib window. But the one-window approach may be easier than I think - and the user experience may be worth it. :)

[GuillaumeFavelier]

my previous comment was mostly motivated by perceived difficulity of this goal.

Honestly I thought the same at first but it seems only a few lines of code were necessary to add this:

        vlayout = self.plotter.frame.layout()
        vlayout.addWidget(self.mpl_canvas)
        vlayout.setStretch(0, 2)
        vlayout.setStretch(1, 1)

[mmagnuski]

@GuillaumeFavelier - that's cool!
So maybe it wouldn't be difficult to add an option to detach the matplotlib figure? 😈

[GuillaumeFavelier]

It's possible too. How would you pass this option though? We already use show_traces=True in plot_source_estimates().

[larsoner]

show_traces='separate' or something. But let's figure this API out separately as there are additional questions we'll want to sort out (e.g., should each click open a new window, should we close the windows when clicking just like we remove traces from the all-one plot in the joint mode, etc.)

[mmagnuski]

@larsoner Sure!