Mne-python: ENH: MNE-binder

[kingjr]

Todo (modified by @Eric89GXL):

• [x] finalize usable binder
• [x] update main page with link to plot_introduction.ipynb (#3535)
• [x] make plot_introduction.rst prettier (logo at top, more plots, etc.)
• [x] add hints about IPython (e.g., shift-enter)
• [x] fix sphinx-gallery, remove of e.g. .. _intro_tutorial: (https://github.com/sphinx-gallery/sphinx-gallery/issues/141)

[larsoner]

Is two steps okay?

$ python setup.py install
$ python -c "import mne; mne.datasets.sample.data_path()"

I assume the answer is "no", but knowing more about the application could help us come up with some solution.

[kingjr]

No, it has to be installed with git...

I'll let you know the application by Friday if I don't find, but I wanted to keep it as a surprise :)

[kingjr]

I guess I can just modify the setup.py

[larsoner]

No, it has to be installed with git...

I don't quite understand, but I can wait for the explanation :) If the goal is to get it in master we'll need to assess how wide-spread the application would be and why a two-step command won't suffice...

But yes if you need it to happen when doing setup.py install (or whatever) you can change setup.py to get it done. You could presumably add a command-line switch that did some post-install trickery --basically the command above.

[kingjr]

I'll try to see whether it can work, then I'm sure you guys will tell me how it should be actually done :)

[larsoner]

Please make sure you're working off of latest master though so that @dengemann's hosting costs go up instead of mine from all the dataset downloads :)

[kingjr]

Ok :]

[kingjr]

Well, I tried to call import mne; mne.datasets.sample.data_path() at the end of the install but it doesn't seem to work.

The aim is to have the examples directly testable in the browser using binder, and thus limiting the data transfer (and our costs!).

At the moment, we can use MNE (e.g. here), but the examples have to be redownloaded every time.

[jaeilepp]

I have updated the Makefile to download datasets in https://github.com/mne-tools/mne-python/pull/2849, but the doc build seems to run into problem after another, so we'll have to see when it gets merged.

[larsoner]

@kingjr that could _really_ help with new user transition / #1495.

Why does it need to be done during the install step, though? Is there no way in binder to give it additional install / caching steps?

[larsoner]

Specifically:

environment.yml for conda Python projects
Dockerfile for custom builds

If either one of these offers caching of arbitrary directories -- and it seems like they probably both do -- then we should be able to have one copy of the datasets somewhere, right?

[kingjr]

Yes, you're right: there seems to be: https://github.com/binder-project/example-dockerfile/blob/master/Dockerfile

Shall I open a PR to give it a try?

[larsoner]

Can you investigate the environment.yml solution first? That would better match what we do with Travis, AppVeyor, And CircleCI. If it can't be done with the YAML then we can live with the docker one.

[kingjr]

Sure, I'll have a look tonight. The following question will probably be: is there a way to translate .py into .ipynb on the fly to avoid duplicating the example codes.

[larsoner]

Good question. For a first pass, unless someone immediately knows the answer to that question, I'd try making one or two examples that we'd want to use to get people going -- probably the most basic, introductory ones -- into notebooks manually. See if we can get that working, then worry about automation or deduplication later. Even if we end up wiht _one_ example that people could play with online it would be really cool.

[kingjr]

Even if we end up wiht one example that people could play with online it would be really cool.

I have tested this already, and the examples that don't require mayavi worked without any issue.

[dengemann]

Thanks @kingjr for looking into this, this is a beautiful idea. Regarding mayavi I hope we can soon add a matplotlib fallback surface renderer, some work in progress by nilearn people.

[kingjr]

When using %matplotlib notebook at the beginning of the notebook we can get interactive 3D matplotlib rendering in the browser.

(When I see all this, I'm so happy to have switched to Python. )

[agramfort]

the new version of sphinx gallery offers the possibility to download examples as notebooks.

see : http://nilearn.github.io/auto_examples/plot_python_101.html#sphx-glr-auto-examples-plot-python-101-py

we just need to rebuild the docs with the new version of sphinx-gallery

[larsoner]

@kingjr where are we with binder? Do you have time to work on it? If not, I'd be up for taking a shot at it. If people could play with data easily in the browser it would be awesome.

[kingjr]

ok, let me try to regive it a go

[kingjr]

I managed to make the Dockerfile with all datasets (I haven't managed to set an environment.ylm that works, but this can be fixed later I think).

The main bottleneck I have now concerns the ipynb files. These notebooks are created by the sphinx build, and are not part of the github rep.

We can setup the docker to generate them at the installation step. However this installation occurs before binder populates the notebooks directory (in /home/main/notebooks) with the repo, which is the only visible folder from binder's Jupyter.

I'm also unclear on whether we should create a mne-tools/mne-binder repository (mne code need not be visible to the users) or whether we should just bind mne-tools/mne-python.

[agramfort]

you can get the generated notebooks from:

https://github.com/mne-tools/mne-tools.github.io

[kingjr]

This isn't the issue. We can add any files before the installation but they can only be in a parent folder from the binder jupyter directory, and therefore cannot be used as ipynb.

[agramfort]

no idea then. I am bit lost here...

[larsoner]

Can you use symlinks to get around the problem?

[kingjr]

Can you use symlinks to get around the problem?

Not if I create them dynamically at the installation step: the symlinks would have to outside of /home/main/notebooks and thus invisible.

If we create a dedicated rep for mne-binder, we can git add the notebooks, but that will be extra work in the future, because everytime we update the examples and tutorials, we'll have to also update the ipynb. But perhaps this can be done automatically, in a similar way that sphinx works?

[larsoner]

But perhaps this can be done automatically, in a similar way that sphinx works?

Yes, take a look here for the CircleCI-based doc autoupdate, it's only a couple lines. So to update your repo it would hopefully only be a couple more now that @jaeilepp has done the hard work to sort everything out.

How about this plan:

1. I'll start a mne-binder repo
2. You get admin access to it
3. git add and git commit directly to mne-binder/master as much as you want to get something working
4. Ping me once things start working and we'll iterate
5. Once we have something solid, we'll get the repo configured to be updated by mne-python/master builds on CircleCI

Sound good?

[larsoner]

Steps 1 and 2 are done, so if you like the plan above, feel free to get going :)

[larsoner]

I'm going to reopen this so we don't forget about it for 0.13

[kingjr]

@Eric89GXL I won't have anymore time to spend on the binder, in case you want to take over

[larsoner]

I'll try to get to it for 0.13. Do you have time to send me what you have so far and say how far you got / where you got stuck?

[kingjr]

@Eric89GXL Sure the todo list is here: https://github.com/mne-tools/mne-binder/issues/1

I had stopped because the binder was becoming unreliable (i.e. impossible to connect). Not sure why; I havn't looked at it for months, so there is a chance that it was a bug that is now fixed.

[larsoner]

@kingjr I made some minor cleanups / updates to mne-binder, can you see if this link works?

If so, we could start by putting a link on our homepage. Eventually we can modify sphinx-gallery as @agramfort suggests to make all example links, but when we do that we need to package all the data so it's a bit heavier. I say we wait until mayavi works in IPython via conda anyway.

We should update sphinx-gallery to get rid of the .. _intro_tutorial: in the .ipynb files, and make the intro a little bit cooler looking. Then I think this is good to go. I updated the description with these goals.

[kingjr]

Great thanks!

I say we wait until mayavi works in IPython via conda anyway.

This one indeed a big blocker. But do you think it'll be supported in notebooks?

We should update sphinx-gallery to get rid of the .. _intro_tutorial: in the .ipynb files, and make the intro a little bit cooler looking.

+1

[larsoner]

But do you think it'll be supported in notebooks?

I think @agramfort mentioned that they're working on it. But who knows what the timeline is like.

[agramfort]

support of mayavi in notebook worked on a toy for me

did not have time to play more for real