Mne-python: What are some blockers for 3D javascript surfaces?

[choldgraf]

I know this isn't an MNE problem, but I've been keeping an eye on the javascript threejs bridges out there (in particular pythreejs and ipyvolume) and I think they're close to having what we'd need for surface plots.

Can we put together a list of features we'd need that are blocking on whether we'd incorporate something like this into MNE (or I suppose pysurfer if that'd make more sense)

Right now ipyvolume supports trisurf functionality...e.g. see this tweet (apologies for obnoxiously self-linking). It's got a bit of bugginess in the camera perspective but I bet that can get worked out relatively quickly. For any kind of plots where you just need to change the color of the triangles, that should already be good to go. For plots that have other stuff plotted on _top_ of brains, that might be trickier (e.g., contour plots).

So, what else would we need to see for this to be useful for MNE?

Todo:

• [ ] get transparent color
• [ ] improve computation speed by downsampling mesh passing upsampling matrix and downsampled data to JavaScript and upsampling there
• [ ] change the controller to show actual timing
• [ ] improve the viewing controls (in mayavi, we can use shift ctrl etc in cmbination with the mouse to better move the camera)
• [ ] pass/compute normals for the surface
• [ ] remove axes box

[choldgraf]

copy @agramfort @larsoner @kingjr in case you've got opinions :-)

[agramfort]

99% of usecase is stc.plot with time interactlion

I think next is showing labels (plain not even contour plots)

if we can do this with a pysurfer like API then it can be a dropin
replacement of mayavi at least in the notebook for now

[choldgraf]

where's a dataset I can test this with? e.g., how would I access the labels? I think this is possible.

[larsoner]

I suspect it's possible too, someone needs to find the time to implement. @agramfort and I did a little bit of work on this.

You can use sample data. Here is one I made a long time ago to try to use VisPy, it gets you data in the format you need (triangulated mesh + per-vertex values):

https://gist.github.com/larsoner/25fa656c6d6e0b02b56c40a571bfb77c

I wouldn't worry about labels for the first pass, since they are just the same thing limited to a smaller subset of vertices.

The annoying thing is getting Python<->browser two-way interaction without forcing people to use IPython. I think this is also possible, but again requires time to investigate.

[choldgraf]

@larsoner I have some demo code set up here:

http://beta.mybinder.org/v2/gh/choldgraf/mne3d/master

that's got the surface data + curvature...is there labels in the same folder in MNE?

re: two-way interaction w/o IPython, that will be tricky for sure. This is all using ipywidgets under the hood. Maarten and I were playing around with getting this to work using chrome-headless so you could run this as a proper script that would just save PNGs etc, and that seemed promising

[choldgraf]

(link to the repo for the binder link above: https://github.com/choldgraf/mne3d)

[kingjr]

The annoying thing is getting Python<->browser two-way interaction without forcing people to use IPython. I think this is also possible, but again requires time to investigate.

We could still move on to support jupyter support for now, until we find a way to run threejs from any python platform?

[choldgraf]

We could still move on to support jupyter support for now, until we find a way to run threejs from any python platform?

My 2 cents, there's enough value-added for something like ipyvolume that we could build some simple viz on top of it :-)

[kingjr]

Here's a gist to plot source estimates as a controllable movie. https://gist.github.com/kingjr/cdadefbb5f522ab7450ead746dc0a5b2

It is pretty straightforward. The movie control is still a bit slow, perhaps @larsoner has some ideas on how to downsample the mesh online?

The main issue that I have, is to set transparent colors to have both the curvature and the sources.

On the todo

• get transparent color
• improve computation speed by downsampling mesh
• change the controller to show actual timing
• ask the p3volume people to improve their viewing controls (in mayavi, we can use shift ctrl etc in cmbination with the mouse to better move the camera)

[larsoner]

Right now I assume you're piping the high-resolution data. This will be very slow. Instead, you'll need to pass the sparse upsampling matrix to JavaScript (there are libs that handle sparse matrices), and the low-resolution STC, and do the upsampling in JavaScript instead of Python. This upsampling is a standard FreeSurfer operation IIRC so it's possible that the ipyvolume people will be amenable to allowing someone to add this functionality.

[choldgraf]

FWIW we hacked on this a little bit at NeuroHackWeek this year and I think the plan is to move Jan's niwidgets package into nipy, and then to add surface widget that creates these plots with ipyvolume.

https://github.com/janfreyberg/niwidgets

It'll be a bit WIP, since a lot of the team working on this were first-time devs. Just giving you a heads up to keep an eye on it...

BTW - you can also get things like diffusion imaging plots...ipyvolume is nifty!

[larsoner]

(in other words, dont downsample the mesh, which will look terrible, stay in high-res space but be smart about passing data)

[larsoner]

You'll also need to ask the ipyvolume people to actually plot with vertex normals defined properly for rendering. (These can be calculated in standard ways from the triangulation, or passed in, whichever they're wiling to live with.) It will make the surfaces smooth instead of all blotchy like they are now.

[choldgraf]

also FWIW, the ipyvolume "people" are really just one guy...so I think he'd appreciate PRs etc if they come in ;-)

[larsoner]

I need to (re-)teach myself JavaScript to be able to do it, so it would take me a while :(

[larsoner]

@kingjr I'll move your todo list to the top and we can all edit/update

[kingjr]

BTW, the control speed of ipyvolume is decent now, (without downsmapling)

[agramfort]

do you think it's possible to have something like:

brain = stc.plot(backend='ipyvolume')

that works similarly to pysurfer when in the notebook?

[choldgraf]

Should be possible IMO

On Fri, Feb 2, 2018, 11:43 PM Alexandre Gramfort [e-mail]
wrote:

do you think it's possible to have something like:

brain = stc.plot(backend='ipyvolume')

that works similarly to pysurfer when in the notebook?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4517#issuecomment-362787691,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABwSHUJxfYRj2Ft6Uc3KyCoz6lCj4a8gks5tRA4OgaJpZM4PD8SV
.

[agramfort]

good it's the week end :)

[choldgraf]

Don't forget this Sunday is a world wide holiday

On Sat, Feb 3, 2018, 12:34 AM Alexandre Gramfort [e-mail]
wrote:

good it's the week end :)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4517#issuecomment-362790739,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABwSHc8vaxeY21aFKOXtDaSo7zo7SJ0uks5tRBodgaJpZM4PD8SV
.

[larsoner]

@GuillaumeFavelier is working on (several) solutions for this so I don't think we need to track it here