Mne-python: Tab completion crashes IPython

[gbrookshire]

Describe the bug

When using mne in IPython, pressing tab for tab-completion often (but not always) causes IPython to crash. This doesn't happen when using objects that are not from mne-python.

Steps to reproduce

In IPython, execute the following code:

import mne
import os
from mne.datasets import multimodal

fname_raw = os.path.join(multimodal.data_path(), 'multimodal_raw.fif')
raw = mne.io.read_raw_fif(fname_raw)
cond = raw.acqparser.get_condition(raw, 'Auditory right')
epochs = mne.Epochs(raw, **cond)

Then type epochs.in and press tab.

Expected results

The list of tab-completion possibilities appears, or it autocompletes to epochs.info.

Actual results

IPython hangs for several seconds, then crashes. (Sometimes it works correctly.)

In [4]: epochs.in                                                                                                      
Traceback (most recent call last):
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/cache.py", line 109, in wrapper
    return dct[key]
KeyError: ((), frozenset())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/cache.py", line 109, in wrapper
    return dct[key]
KeyError: ((), frozenset())

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/__main__.py", line 14, in <module>
    start_ipython()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/__init__.py", line 126, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/terminal/ipapp.py", line 356, in start
    self.shell.mainloop()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 558, in mainloop
    self.interact()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 541, in interact
    code = self.prompt_for_code()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 469, in prompt_for_code
    **self._extra_prompt_options())
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/shortcuts/prompt.py", line 997, in prompt
    return self.app.run()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 810, in run
    return loop.run_until_complete(self.run_async(pre_run=pre_run))
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 777, in run_async
    return await _run_async2()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 765, in _run_async2
    await self.cancel_and_wait_for_background_tasks()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 862, in cancel_and_wait_for_background_tasks
    await task
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1854, in new_coroutine
    await coroutine(*a, **kw)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 1684, in async_completer
    document, complete_event
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 270, in get_completions_async
    document, complete_event
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 196, in get_completions_async
    for item in self.get_completions(document, complete_event):
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/terminal/ptutils.py", line 90, in get_completions
    yield from self._get_completions(body, offset, cursor_position, self.ipy_completer)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/terminal/ptutils.py", line 100, in _get_completions
    for c in completions:
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/core/completer.py", line 438, in _deduplicate_completions
    completions = list(completions)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/core/completer.py", line 1815, in completions
    for c in self._completions(text, offset, _timeout=self.jedi_compute_type_timeout/1000):
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/core/completer.py", line 1872, in _completions
    signature = _make_signature(jm)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/IPython/core/completer.py", line 989, in _make_signature
    return '(%s)'% ', '.join([f for f in (_formatparamchildren(p) for p in completion.params) if f])
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/cache.py", line 111, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/api/classes.py", line 428, in params
    for signature in self._get_signatures():
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/api/classes.py", line 505, in _get_signatures
    return [sig for name in names for sig in name.infer().get_signatures()]
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/api/classes.py", line 505, in <listcomp>
    return [sig for name in names for sig in name.infer().get_signatures()]
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/cache.py", line 111, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/compiled/mixed.py", line 124, in infer
    tree_values = tree_value.py__getattribute__(self.string_name)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/base_value.py", line 88, in py__getattribute__
    names = self.goto(name_or_str, name_context, analysis_errors)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/base_value.py", line 77, in goto
    names = finder.filter_name(filters, name_or_str)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/finder.py", line 35, in filter_name
    for filter in filters:
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/base_value.py", line 63, in _get_value_filters
    for f in self.get_filters(origin_scope=origin_scope):
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/value/instance.py", line 176, in get_filters
    class_value = self.get_annotated_class_object()
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/value/instance.py", line 350, in get_annotated_class_object
    return self._get_annotated_class_object() or self.class_value
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/cache.py", line 43, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/value/instance.py", line 339, in _get_annotated_class_object
    all_annotations = py__annotations__(signature.value.tree_node)
  File "/home/local/ADF/brookshg/anaconda3/envs/mne/lib/python3.7/site-packages/jedi/inference/gradual/annotation.py", line 189, in py__annotations__
    for function_param in funcdef.get_params():
AttributeError: 'Class' object has no attribute 'get_params'

If you suspect this is an IPython 7.12.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Then it exits IPython and returns you to the login prompt.

Additional information

Platform:      Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
Python:        3.7.6 (default, Jan  8 2020, 19:59:22)  [GCC 7.3.0]
Executable:    /home/local/ADF/brookshg/anaconda3/envs/mne/bin/python
CPU:           x86_64: 4 cores
Memory:        39.1 GB

mne:           0.19.2
numpy:         1.18.1 {blas=mkl_rt, lapack=mkl_rt}
scipy:         1.4.1
matplotlib:    3.1.3 {backend=Qt5Agg}

sklearn:       0.22.1
numba:         0.48.0
nibabel:       3.0.1
cupy:          Not found
pandas:        1.0.0
dipy:          1.1.1
mayavi:        4.7.1 {qt_api=pyqt5, PyQt5=5.9.2}
pyvista:       0.23.1
vtk:           8.1.2

[gbrookshire]

In case this is useful information, the problem does not appear on a computing cluster with these stats:

Platform:      Linux-3.10.0-862.11.6.el7.x86_64-x86_64-with-centos-7.5.1804-Core
Python:        3.7.2 (default, Aug 30 2019, 15:08:46)  [GCC 8.2.0]
Executable:    /rds/bear-apps/2019a/EL7-haswell/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7
CPU:           x86_64: 18 cores
Memory:        110.0 GB

mne:           0.19.1
numpy:         1.16.2 {blas=openblas, lapack=openblas}
scipy:         1.2.1
matplotlib:    3.0.3 {backend=TkAgg}

sklearn:       0.21.3
numba:         0.43.1
nibabel:       2.4.0
cupy:          Not found
pandas:        0.24.2
dipy:          1.0.0
mayavi:        4.7.1 {qt_api=pyqt5, PyQt5=5.12.1}
pyvista:       0.22.4
vtk:           8.1.2

[drammock]

This is probably outside our ability to fix. Tab completion is handled by an external library (in your case "jedi") that is loaded by the ipython console, IDE or notebook. These may have jedi-specific settings (like memory limits) or may have an alternative completion library you can switch to. Searching for "ipython jedi settings" or "ipython jedi alternative" is probably a good place to start. Hopefully others will chime in with additional suggestions.

[larsoner]

Indeed I cannot replicate on my system where I just did pip install --upgrade jupyter[notebook]:

In [3]: epochs.info 
                  info            
                  interpolate_bads   
                  instance  

So I think this is a bug with jedi and you probably / hopefully just need to update.

[gbrookshire]

Thanks -- good to know! It looks like setting IPython to use a different autocompleter fixes the problem. In case anyone else runs into this problem, here's the solution:
%config IPCompleter.use_jedi = False

[cbrnr]

I frequently (but not always) encounter this issue myself, therefore I'm reopening this issue. So far, this happens only when trying to auto-complete MNE objects. Google only finds this issue, so I'm not sure that this is a bug with Jedi. Disabling Jedi is also not a fix, but only a workaround.

Here's how to reproduce:

• Start IPython
• import mne
• Load some file, e.g. raw = mne.io.read_raw_fif(mne.datasets.sample.data_path() + "/MEG/sample/sample_audvis_raw.fif")
• Type raw.set and hit Tab to auto-complete: nothing happens
• Hit Enter -> voilà -> IPython crashes

I think it might have something to do with Raw objects - maybe Jedi can't find the definition somehow?

[drammock]

I cannot reproduce:

[larsoner]

Start IPython
import mne
Load some file, e.g. raw = mne.io.read_raw_fif(mne.datasets.sample.data_path() +
"/MEG/sample/sample_audvis_raw.fif")
Type raw.set and hit Tab to auto-complete: nothing happens

At this step I get:

In [1]: import mne                                                                                                                                                                  
In [2]: import faulthandler                                                                                                                                                         
In [3]: faulthandler.enable()                                                                                                                                                       
In [4]: raw = mne.io.read_raw_fif(mne.datasets.sample.data_path() + "/MEG/sample/sample_audvis_raw.fif")                                                                            
In [5]: raw.set_annotations 
                set_annotations() set_meas_date    
                set_channel_types set_montage      
                set_eeg_reference                        
                function(annotations, emit_warning=True) 

Same if I don't import faulthandler; faulthandler.enable().

Can you try it with enabling faulthandler? It might allow you to make a minimal example to report to the upstream / jedi library.

[cbrnr]

You guys are on Linux, right? Can someone on a Mac try to reproduce this?

Using the faulthandler doesn't change anything, the output upon crashing is the same. I've set %config Application.verbose_crash=True for a more detailed crash report though, which I could use when reporting upstream (upstream meaning IPython I think, not Jedi).

[cbrnr]

BTW, this happens with both Homebrew and Anaconda Python.

[drammock]

You guys are on Linux, right?

The OP was also using Linux, and experienced a similar issue. Did you try their solution @cbrnr ?

[cbrnr]

OK. Yes, turning off Jedi works, but I guess I'd want to use Jedi so this is just a workaround.

[agramfort]

a PR in jedi you shall send :)

[cbrnr]

This was a Jedi bug which has already been fixed in master (it's not in a release yet).