Jedi: Slow pandas completion

[dpavlic]

I'm getting significant lag in emacs (company-mode with anaconda backend), while completions seem to break entirely with vim (YouCompleteMe) where it won't complete anything anymore for a long, long time when using pandas 0.15.1. Here is some example code:

import pandas as pd
a = pd.DataFrame({'v1': ['a', 'b', 'c']})
a.v1.cat.

It is at this point where lag in emacs becomes unbearable, while vim will happily continue to work, but all completions from YouCompleteMe will not work anymore (including completions for a different library).

Now, it seems the issue is that the getting the completion candidates is a costly affair in jedi itself:

import jedi
%timeit jedi.Script("import pandas as pd; a = pd.DataFrame({'v1': ['a','b','c']}); a.v1.cat.").completions()

Which gives me:

1 loops, best of 3: 2.66 s per loop

A similar timeit operation on os.path. completion gives me 9 ms, so it seems to me like that might be the source of the problem.

[davidhalter]

Thanks for the report!

[FrankFeng]

Similar problems happened to me with Vim. At first I used jedi-vim, and it took one or two minutes to do completion for pandas every time (not only the first time, and I don't know why), on my machine with 4 cpu cores and 16G ram. Then I switched to YouCompleteMe (still with jedi), and it became much faster.

However, with YouCompleteMe, some completion can't be done for pandas. For example, this can be done:

import pandas
pandas.read_csv()

This can't be done -- it can't find any candidates for completion:

import pandas
pandas.read_html()

Maybe pandas is too complicated, and it depends on numpy. I really appreciate you if you could spend some time to improve that.

Versions:
YouCompleteMe was installed with Vundle yesterday (Jedi was installed via YouCompleteMe as submodule). Pandas is 0.13.1. Vim is VIM 7.4 (Included patches: 1-295). Python-mode is NOT installed. OS is Ubuntu 12.04.

[tomsheep]

Any update? I came across the same issue with Pandas 0.16.0 + MacOSX 10.10

[davidhalter]

@FrankFeng YouCompleteMe uses Jedi with an async client/server model. This obviously makes it "fast", but some things will not complete.

@tomsheep Not really. I don't really spend a lot of time on Jedi at the moment, sorry.

[davidhalter]

Still that bad with the current dev branch?

[FrankFeng]

Now I feel Jedi is obviously faster for pandas, compared to when I made the comment above on 11 Feb 2015. Thanks very much for your work!

[davidhalter]

I finally had time to understand what the problem was here.

I brought this example down from a few seconds to 0.02s. The slowest thing (DataFrame completions, all types and docstrings with signatures) went from more than 10s to 0.25s. I feel like this is good enough.

The changes are kind of a "PyCharm mode". I just disabled a lot of dynamic features for pandas that are problematic, like decorators and parameter resolving (**kwargs to what they effectively are), as well as ignoring __getattr__ and if branching. These features are nice, but don't really provide any value for pandas (and other libraries like numpy/tensorflow), so I just disabled them for some big libraries. I called it "PyCharm mode", because those are all things PyCharm always ignores, which is a bad thing in my opinion, but makes perfect sense for those libraries.

Feel free to test already on master, before release.

Further improvements might happen for #1116 and to be really fast we might eventually need to do #1059.