Mne-python: Improve documentation installation info

[choldgraf]

A few things I notice:

• Do we really need to recommend python 2.7 to new users? IMO it makes MNE seem behind-the-times. Mayavi does technically work on python 3 now, though it's a pain to install IME.
• The installation instructions are very text heavy. We should streamline.
• Do we really need to explicitly call out matlab in the instructions?
• I think we should have a separate page for installing the development version. If we have newbies coming to this page we don't want them seeing python setup.py develop IMO.
• We should figure out who this set of instructions doesn't work for so that we can improve them.

[choldgraf]

@kingjr @Eric89GXL

[larsoner]

I don't see a straightforward way to use Mayavi on Py3k until it works out of the box (conda-forge can also break things) so I'd rather wait until then to recommend 3+.

Otherwise I agree, do you have any time to try?

[choldgraf]

I can make a PR with these improvements though let's get a little list together here we agree on first. Anything else pop out at you? @kingjr any specific feedback from people on things that didn't work?

[kingjr]

Good. Perhaps clarifying the backends and environment variables in there.

[larsoner]

@choldgraf any time to tackle this in the next week or so? Would be good for release.

@kingjr on Gitter mentioned adding a section for Mayavi tricks, which would be good (@kingjr feel free to add links to suggestions if you have them, otherwise I can try to dig some up)

[dengemann]

As long as mayavi is not trivial to install on python 3 we must recommend
python 2.7. This is still the case.
On Wed, 16 Aug 2017 at 22:13, Eric Larson [e-mail] wrote:

@choldgraf https://github.com/choldgraf any time to tackle this in the
next week or so? Would be good for release.

@kingjr https://github.com/kingjr on Gitter mentioned adding a section
for Mayavi tricks, which would be good (@kingjr
https://github.com/kingjr feel free to add links to suggestions if you
have them, otherwise I can try to dig some up)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-322885305,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB0fipjaEJbpkXqTMgA7K6gE8dZRVe0hks5sY01PgaJpZM4Ohk5o
.

[dengemann]

Otherwise recent installation feedback during workshops was excellent.
On Wed, 16 Aug 2017 at 22:15, Denis-Alexander Engemann <
[email protected]> wrote:

As long as mayavi is not trivial to install on python 3 we must recommend
python 2.7. This is still the case.
On Wed, 16 Aug 2017 at 22:13, Eric Larson [e-mail]
wrote:

@choldgraf https://github.com/choldgraf any time to tackle this in the
next week or so? Would be good for release.

@kingjr https://github.com/kingjr on Gitter mentioned adding a section
for Mayavi tricks, which would be good (@kingjr
https://github.com/kingjr feel free to add links to suggestions if you
have them, otherwise I can try to dig some up)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-322885305,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB0fipjaEJbpkXqTMgA7K6gE8dZRVe0hks5sY01PgaJpZM4Ohk5o
.

[kingjr]

@dengemann woaw; that's actually surprising to me ;); I regularly still run into a bunch of incompatibility issues between mayavi, xgboost, tensorflow etc for very unclear reason.

My only tips for now is

• if conda install mayavi creates an incompatibility error, pip install mayavi doesn't
• if, on ubuntu, mayavi freezes, install pythonwx, and launch ipython --pylab=wx (way better than qt in my experience)

[kingjr]

also, overall, I'm would be against env variables as much as possible, and show in examples/tutorials etc to add variables via os.env, such that we work in pure python

[dengemann]

@kingjr Well I am talking about basic installations. A significant
proportion of users does not care about Mayavi. 2-3 years ago we had far
more reports about people not managing at all to run a basic example. This
has improved a lot I'd say thanks to anaconda and continued work on docs.
On Wed, 16 Aug 2017 at 22:21, Jean-Rémi KING [e-mail]
wrote:

also, overall, I'm would be against backend variables as much as possible,
and show in examples/tutorials etc to add variable via os.env, such that
we work in pure python

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-322887377,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB0fikSyccWcEylXIxpArsJn99eVYO19ks5sY09GgaJpZM4Ohk5o
.

[choldgraf]

I can spend some time on this. Can people add to the list in the top-level comment anything they'd like to see included?

[agramfort]

apparently mayavi is available for py3k on conda-forge at least on linux:

https://github.com/massich/mayavi_py35_MWE

cc @massich

[dengemann]

Yeah. But you cannot really expect from a newcomer to figure out all the
hacks it takes to actually make it work. Version numbers of QT python 3.x
(does not work on python 3.6), etc. I'd propose to switch to py3k
recommendation the moment we drop mayavi as backend or mayavi gets into the
conda py3k main channel. That would be a clear criterion.

On Thu, 17 Aug 2017 at 08:09, Alexandre Gramfort [e-mail]
wrote:

apparently mayavi is available for py3k on conda-forge at least on linux:

https://github.com/massich/mayavi_py35_MWE

cc @massich

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-322977874,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB0finAwRdNGqfYZOM7AApW0vzqRKI9Sks5sY9kHgaJpZM4Ohk5o
.

[agramfort]

We could provide an environment.yml file for conda and recommend people to use it

We could have one Travis build to use it

[choldgraf]

what is the main thing blocking us from using something like pythreejs or ipyvolume as a backend? The main drawback I see is that these are pure web packages, so it'll be clunkier to get them working in an old-fashioned (see what I did there @agramfort ;-) ) python workflow. For those who don't mind working within a browser context, it should be a great option for plotting! Maybe at least worth exploring the idea

[agramfort]

I agree it can be a good option for the future but Iet's the workflow
of the "older" people working :)

until we all switch to jupyterlab...

[larsoner]

Removing the 0.15 milestone. @choldgraf if you think you can make the most critical changes in the next few days / week please do.

I think this will be facilitated greatly by #4523 once we get it done

[choldgraf]

it looks like #4523 is closed but not merged...can you ping me whenever this gets fixed up so I can make changes?

I can put some time into this next week if things are in place for me to do it...let me know. Worst comes to worst we can always backport instructions

[larsoner]

I don't think that the environment (as a suggested one for users) will be
ready by release. Do you have suggestions in the meantime, or prefer to
wait for it to be fully usable?

[choldgraf]

Lemme take a stab at it this week, when is .15 being released?

[kingjr]

IMO, we need to fix/complement the install documentation before the release.

1. Mayavi:

On fresh mac os (El Capitain) install + anaconda with python 2.7, mayavi/pysurfer fail.
We seem to need to do conda remove navigator-updater; pip install vtk mayavi PySurfer.

On Ubuntu (16.04), (and, IIRC after matplotlib upgrade), I have repeatedly needed to do:
conda remove mayavi; install mayavi --upgrade

(and no, the maptloblib backend is not a good solution, as it can be mega slow)

2. environment variables, freesurfer and matlab dependencies

Ideally, we should have a script that automatically detects or setup the environment variables and paths related to Freesurfer, MNE C and Matlab. (Most users don't know what .bash_profile is, don't code in shell, and don't know the difference between sh and csh.)

We could have a pure python function that helps users set-up their environment:

def setup_paths(FREESURFER_DIR=None, MATLAB_ROOT=None, MNE_ROOT=None):
    if FREESURFER_DIR is None:
         if not 'FREESURFER_DIR' in os.environ:
             os.environ['FREESURFER_DIR'] = subprocess('which', 'recon-all')
...

In my experience, these two points are total blockers for newbies. Most I meet come from Fieldtrip where they just need to add the paths through the Matlab menu, and everything else works.

[larsoner]

we need to fix/complement the install documentation before the release.

I agree it could be better. Someone needs to find the time to really beef it up. I won't have it because I'm working on too many other bugs and spread too thin currently. Personally I'm hoping #4570 and eventually #4523 will take care of it, because then the installation instructions can be:

conda env create -f environment.yml

... and this env will be tested on Linux, OSX, and Windows CIs. But maybe this won't happen soon enough :(

On fresh mac os (El Capitain) install + anaconda with python 2.7, mayavi/pysurfer fail.

Our CIs run OSX w/PySurfer. Can you clarify the problem?

On Ubuntu (16.04), (and, IIRC after matplotlib upgrade), I have repeatedly needed to do:

Anaconda unfortunately does not resolve dependencies properly all the time. If you want to upgrade a package, you're best off doing something like:

conda install numpy scipy matplotlib mayavi

To force it to look at all of those dependencies. If you just do:

conda install matplotlib

... it will often (always?) just install the latest matplotlib and pyqt5 instead of pyqt4, which will break some of Mayavi's dependencies. I wonder if there is an Anaconda bug about this (if there isn't, we should add one).

Ideally, we should have a script that automatically detects or setup the environment variables and paths related to Freesurfer, MNE C and Matlab.

The calls MNE-Python makes to MNE-C have been eliminated, and the ones to FreeSurfer are pretty limited. So I assume you're not talking about those? If you source these scripts in the Python environment and then exit, then the env vars don't persist AFAIK. So I assume your goal is to get the env vars set up in Python so you can do subprocess.Popen calls to e.g. set up your FreeSurfer reconstructions? If so it sounds more like the goal is to make a freesurfer-Python sort of package, so it seems a bit out of scope.

[choldgraf]

What if we just agreed to backport these documentation changes when we
merge them? I agree this is super important... Just haven't had time
because I'm on vacation in England right now.

On Thu, Sep 28, 2017, 2:12 AM Eric Larson [e-mail] wrote:

we need to fix/complement the install documentation before the release.

I agree it could be better. Someone needs to find the time to really beef
it up. I won't have it because I'm working on too many other bugs and
spread too thin currently. Personally I'm hoping #4570
https://github.com/mne-tools/mne-python/pull/4570 and eventually #4523
https://github.com/mne-tools/mne-python/pull/4523 will take care of it,
because then the installation instructions can be:

conda env create -f environment.yml

... and this env will be tested on Linux, OSX, and Windows CIs. But maybe
this won't happen soon enough :(

On fresh mac os (El Capitain) install + anaconda with python 2.7,
mayavi/pysurfer fail.

Our CIs run OSX w/PySurfer. Can you clarify the problem?

On Ubuntu (16.04), (and, IIRC after matplotlib upgrade), I have repeatedly
needed to do:

Anaconda unfortunately does not resolve dependencies properly all the
time. If you want to upgrade a package, you're best off doing something
like:

conda install numpy scipy matplotlib mayavi

To force it to look at all of those dependencies. If you just do:

conda install matplotlib

... it will often (always?) just install the latest matplotlib and pyqt5
instead of pyqt4, which will break some of Mayavi's dependencies. I
wonder if there is an Anaconda bug about this (if there isn't, we should
add one).

Ideally, we should have a script that automatically detects or setup the
environment variables and paths related to Freesurfer, MNE C and Matlab.

The calls MNE-Python makes to MNE-C have been eliminated, and the ones to
FreeSurfer are pretty limited. So I assume you're not talking about those?
If you source these scripts in the Python environment and then exit, then
the env vars don't persist AFAIK. So I assume your goal is to get the env
vars set up in Python so you can do subprocess.Popen calls to e.g. set up
your FreeSurfer reconstructions? If so it sounds more like the goal is to
make a freesurfer-Python sort of package, so it seems a bit out of scope.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-332698775,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABwSHVMCctWLY5vCOcDI6T5bdj89E0Grks5smvKLgaJpZM4Ohk5o
.

[choldgraf]

Also, can we come up with some actionable ways in which the documentation for installation should be improved? It seems like there is disagreement about how best to do this / what information needs to be in there. I am happy to give a shot at a PR, but it is hard to know what we should say as the "best" approach. (e.g. I still haven't really gotten mayavi working reliably on my system so am hesitant to be the one to recommend something that works all the time)

[kingjr]

Our CIs run OSX w/PySurfer. Can you clarify the problem?

installing pysurfer / mayavi through pip or mayavi crashes.

To force it to look at all of those dependencies. If you just do: conda install matplotlib

In my experience, this does not solve the dependencies issue. I had to manually remove navigator-updater else it wouldn't work. This happened on multiple platforms, including mac OS X and ubuntu with fresh anaconda installs.

The calls MNE-Python makes to MNE-C have been eliminated, and the ones to FreeSurfer are pretty limited. So I assume you're not talking about those?

Don't you need those for making the bems (make_watershed_bem) and the surface?

If you source these scripts in the Python environment and then exit, then the env vars don't persist AFAIK.

That's ok IMO, as long as the script is standalone and explicit (i.e. does not depend on preexisting shell variables).

So I assume your goal is to get the env vars set up in Python so you can do subprocess.Popen calls to e.g. set up your FreeSurfer reconstructions? If so it sounds more like the goal is to make a freesurfer-Python sort of package, so it seems a bit out of scope.

For the studies I have done, I systematically had to run the following anatomical pipeline in python (but scalp_surface) after Freesurfer's recon-all:

make_watershed_bem
mne.commands.mne_make_scalp_surfaces
setup_source_space
make_bem_model
make_bem_solution
coregistration

This required having freesurfer and MNE C setup properly, and thus repeatedly blocked students. But I guess there is a better and simpler way to do this?

[larsoner]

None of those commands require MNE-C anymore. You are right that make_watershed_bem and mne_make_scalp_surfaces require FreeSurfer, but the others do not. So I think we're just talking about making these last two easier to work with in some way.

These two functions are actually just thin wrappers around FreeSurfer calls. (I think they're mostly provided for historical purposes?) If you are already doing recon-all, if you can add a mri_watershed and a mkheadsurf call to that same script, then you won't need FreeSurfer at all once you're working in Python. That seems like a better option to me, do you agree? That would make it a documentation problem, rather than trying to hack together Python calling of FreeSurfer functions (which, from PySurfer and MNE-C call-from-Python experiences, I know is non-trivial and can create library conflicts.)

[kingjr]

Ok; I agree with just adding recon-all mri_watershed and mkheadsurfto the documentation. The paths / env variables etc wouldn't be an issue anymore. Thanks

[choldgraf]

another plug here to make documentation change suggestions as actionable and straightforward as possible, unless y'all wanna open the PR ;-)

[kingjr]

@choldgraf you're right. IMO, two PRs would be useful:

1. Clarifiy dependencies

• add a quick tutorial in Documentation > Source-level analysis > prepare anatomical data and/or in the cookbook, that includes a shell script including recon-all, mri_watershed and mkheadsurf for a given subject.
• perhaps explicit in the install doc that MNE C should not be necessary for any processing
• add in the installation doc that freesurfer is necessary for preprocessing the anatomical MRI, and that MNE doesn't do any segmentation/BEM etc

2. troubleshooting doc for mayavi

For Mayavi / PySurfer related issues, we should probably start a troubleshooting section that starts to list the issues with mayavi and their fixes. We could backport this documentation along the way, such that it's not a blocker for the release.

[larsoner]

I can do (1) with @kingjr as a tester once the dust settles on the release. I won't have time before.

For (2) we could tell people to use our current environment.yml (#4570) with ETS_TOOLKIT=wx env var. It should work. But we should really test it for some months before advertising it.

[larsoner]

Okay so over in #4626 I've confirmed that Linux/OSX/Windows can pass tests with the environment.yml. Locally I need to do ETS_TOOLKIT=wx before running any interactive Mayavi stuff to prevent conflicts with matplotlib. However, there is a way to automatically have the environment (when doing source activate mne) set env vars. I'm using it locally and it works. After 0.15 lands, we could modify the install page to recommend this procedure, and we could all try it.

Or we can wait for the 3.6 + PyQt5 environment to stabilize (@wmvanvliet has had good luck so far trying it today), which would be simpler and nicer. But I don't know how long it will take. In the meantime if others want to try it (or the 2.7 env), please do.

[kingjr]

Yes, wx has been a successful backend on linux for me sofar, just requires
installing pythonwx.

I'm not sure what's the best decision with regard to waiting or not for
pyqt5

On 29 September 2017 at 11:13, Eric Larson [e-mail] wrote:

Okay so over in #4626 https://github.com/mne-tools/mne-python/pull/4626
I've confirmed that Linux/OSX/Windows can pass tests with the
environment.yml. Locally I need to do ETS_TOOLKIT=wx before running any
interactive Mayavi stuff to prevent conflicts with matplotlib. However,
there is a way to automatically have the environment (when doing source
activate mne) set env vars. I'm using it locally and it works. After 0.15
lands, we could modify the install page to recommend this procedure, and we
could all try it.

Or we can wait for the 3.6 + PyQt5 environment
https://raw.githubusercontent.com/agramfort/mne-python/6dfdfe805380c02813b6dad95d9b7e054215c465/environment.yml
to stabilize (@wmvanvliet https://github.com/wmvanvliet has had good
luck so far trying it today), which would be simpler and nicer. But I don't
know how long it will take. In the meantime if others want to try it (or
the 2.7 env), please do.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-333154331,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEp7DG_DwmXwgDEc9cHV5gVwcQOxY9RWks5snQk1gaJpZM4Ohk5o
.

[larsoner]

I'm not sure what's the best decision with regard to waiting or not for pyqt5

The advantage is that you can use matplotlib 2+ which has some nice enhancements. Also numpy/scipy/scikit-learn etc. are all more up to date in the 3.6 environment because they aren't held back by old mayavi/vtk etc.

Here is some pseudo-documentation to get people off the ground if they want to try either environment. They can be tested with python -i tutorials/plot_mne_dspm_source_localization.py (or any other script that uses both matplotlib and mayavi.

Python 3 (experimental)

Use this file: https://raw.githubusercontent.com/agramfort/mne-python/6dfdfe805380c02813b6dad95d9b7e054215c465/environment.yml

conda env create -f environment3.yml -n mne3
source activate mne3

Python 2 (CIs use it)

Use this file: https://raw.githubusercontent.com/mne-tools/mne-python/57fe40c98a97db7d1be9143b859ff86d89e1a7d8/environment.yml

conda env create -f environment.yml -n mne2
pushd ~/anaconda2/envs/mne2/
mkdir ./etc/conda
mkdir ./etc/conda/activate.d
mkdir ./etc/conda/deactivate.d
printf '#!/bin/sh\n\nexport ETS_TOOLKIT=wx\n' > ./etc/conda/activate.d/env_vars.sh
printf '#!/bin/sh\n\nexport unset ETS_TOOLKIT\n' > ./etc/conda/deactivate.d/env_vars.sh
sed -i "s/ENABLE_USER_SITE = .*/ENABLE_USER_SITE = False/g" ./lib/python2.7/site.py
popd
source activate mne2

[choldgraf]

Just a thought: can we make mayavi a soft dependency? As in, it doesn't get installed by default and throws an error if you try to do 3D viz without having it installed? I'm worried that, especially for beginners, it becomes a big blocker to any MNE functionality, even though there's tons of stuff you can do with MNE that has nothing to do with 3D viz...

[larsoner]

Just a thought: can we make mayavi a soft dependency? As in, it doesn't get installed by default and throws an error if you try to do 3D viz without having it installed?

You mean in the environment.yml that we're trying to build? The only reason we really need it is to get Mayavi working. From what I've seen Mayavi causes 99% of the problems with Anaconda for us. If you don't want Mayavi, everything pretty much works with conda install numpy scipy matplotlib ... so I'm not sure we even need the environment.yml at that point (even the default Anaconda distribution is probably good enough!).

[choldgraf]

gotcha - maybe this is just a documentation thing. Maybe just worth highlighting that mayavi is not required and may cause headaches if it is used. I don't like that we still have to recommend people use python 2 over python 3 at the very beginning of install instructions :-/

[kingjr]

IMO it's better to say we require python 2 with a working mayavi, than
python 3 and a non working mayavi.

On 3 October 2017 at 12:32, Chris Holdgraf [e-mail] wrote:

gotcha - maybe this is just a documentation thing. Maybe just worth
highlighting that mayavi is not required and may cause headaches if it is
used. I don't like that we still have to recommend people use python 2 over
python 3 at the very beginning of install instructions :-/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-333902624,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEp7DFQRr_MNLmGU-6qJ6sL16qTfOu0yks5somGGgaJpZM4Ohk5o
.

[larsoner]

Okay over on #4641 I think we've finally arrived at an environment that works. 3.6, latest numpy/scipy/matplotlib/mayavi, and CIs pass on Linux, OSX, and Windows. I propose that we add this to the doc, tagged as exerimental for 0.15. Then lots of us (and CIs) test it for a while and if it's all good, use it as the default instructions for 0.16. Thoughts?

[choldgraf]

+1

[wmvanvliet]

This has been working better for me than my previous mess with python2, so yes, +1!

[larsoner]

Updated doc

https://6073-1301584-gh.circle-artifacts.com/0/tmp/circle-artifacts.2fIiuDA/html/install_mne_python.html

[choldgraf]

def an improvement!

--

On Wed, Oct 4, 2017 at 8:05 AM Eric Larson [e-mail] wrote:

Updated doc

https://6073-1301584-gh.circle-artifacts.com/0/tmp/circle-artifacts.2fIiuDA/html/install_mne_python.html

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/issues/4419#issuecomment-334186343,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABwSHW3icJ_dlkIDbU44PbVUJDag0UF3ks5so57DgaJpZM4Ohk5o
.