Jedi: Jedi hangs for a long time with DataFrame containing timestamps

[pwaller]

Hi! Neat tool, thanks for publishing it and maintaining it.

I'm using jedi 0.10.2. Python 3.6, and Jupyter Notebook 5 and ipython 6.

Quick summary

• Completions take upwards of 30 seconds with a DataFrame with 1Mrow and 2 columns, if one of the columns is a timestamp.
• The runtime scales with the number of rows in the DataFrame.
• Waiting 30 seconds or more for completions is too long - during which time no other computation can take place.
• If you interrupt the kernel during jupyter doing the completion, the kernel dies - leading to dataloss and inconvenience (maybe this is a jupyter notebook or ipython bug?).
• It was tricky to figure out this was a jedi problem. It felt for a long time like a jupyter or ipython problem.
• Sending KeyboardInterrupt during a manual Interpreter() invocation reveals that jedi is running df.values, which takes many seconds to run for the aforementioned DataFrame (maybe this is a pandas bug?).

---

Since I've upgraded to the latest ipython and jupyter, I've experienced regular poor performance and hanging of the kernel at odd times during everyday use. I wasn't experiencing this before. I currently suspect jedi may be the culprit. What I observe is that I'll be typing something in, and maybe if I press tab, the kernel becomes unresponsive. I see 100% CPU usage. The hang lasts at least 30 seconds, perhaps longer. If I try to interrupt the kernel in order to get a backtrace, the ipython kernel dies.

I was able to reproduce the problem with the following - tab completing on a dataframe with 1Mrow, and 2 columns: 1 timestamp, and one plain integer:

%%time
import pandas as pd
import numpy as np

from jedi import Interpreter

rows = 1000000

df = pd.DataFrame({
    "foobar": np.random.randint(int(1e18), size=rows).astype("<M8[ns]"),
    "baz": [1] * rows,
})
s = "df.hello"
print(Interpreter(s, [{'df': df}] , line=1, column=len(s)).completions())

This chunk of code runs for more than 15 seconds. The runtime increases if you increase the number of rows.

If I interrupt it, I get a stack trace:

---

Click here to expand stack trace

~/.local/lib/python3.6/site-packages/jedi/api/__init__.py in completions(self)
    177             self._pos, self.call_signatures
    178         )
--> 179         completions = completion.completions()
    180         debug.speed('completions end')
    181         return completions

~/.local/lib/python3.6/site-packages/jedi/api/completion.py in completions(self)
     94 
     95     def completions(self):
---> 96         completion_names = self._get_context_completions()
     97 
     98         completions = filter_names(self._evaluator, completion_names,

~/.local/lib/python3.6/site-packages/jedi/api/completion.py in _get_context_completions(self)
    167             elif symbol_names[-1] in ('trailer', 'dotted_name') and nodes[-1] == '.':
    168                 dot = self._module_node.get_leaf_for_position(self._position)
--> 169                 completion_names += self._trailer_completions(dot.get_previous_leaf())
    170             else:
    171                 completion_names += self._global_completions()

~/.local/lib/python3.6/site-packages/jedi/api/completion.py in _trailer_completions(self, previous_leaf)
    208             for filter in context.get_filters(
    209                     search_global=False, origin_scope=user_context.tree_node):
--> 210                 completion_names += filter.values()
    211         return completion_names
    212 

~/.local/lib/python3.6/site-packages/jedi/evaluate/compiled/__init__.py in values(self)
    343         names = []
    344         for name in dir(obj):
--> 345             names += self.get(name)
    346 
    347         is_instance = self._is_instance or fake.is_class_instance(obj)

~/.local/lib/python3.6/site-packages/jedi/cache.py in wrapper(self, *args, **kwargs)
    119             return dct[key]
    120         except KeyError:
--> 121             result = method(self, *args, **kwargs)
    122             dct[key] = result
    123             return result

~/.local/lib/python3.6/site-packages/jedi/evaluate/compiled/__init__.py in get(self, name)
    326         obj = self._compiled_object.obj
    327         try:
--> 328             getattr(obj, name)
    329             if self._is_instance and name not in dir(obj):
    330                 return []

~/.local/lib/python3.6/site-packages/pandas/core/frame.py in transpose(self, *args, **kwargs)
   1757         """Transpose index and columns"""
   1758         nv.validate_transpose(args, dict())
-> 1759         return super(DataFrame, self).transpose(1, 0, **kwargs)
   1760 
   1761     T = property(transpose)

~/.local/lib/python3.6/site-packages/pandas/core/generic.py in transpose(self, *args, **kwargs)
    511         new_axes = self._construct_axes_dict_from(self, [self._get_axis(x)
    512                                                          for x in axes_names])
--> 513         new_values = self.values.transpose(axes_numbers)
    514         if kwargs.pop('copy', None) or (len(args) and args[-1]):
    515             new_values = new_values.copy()

~/.local/lib/python3.6/site-packages/pandas/core/generic.py in values(self)
   3270         will result in a flot64 dtype.
   3271         """
-> 3272         return self.as_matrix()
   3273 
   3274     @property

~/.local/lib/python3.6/site-packages/pandas/core/generic.py in as_matrix(self, columns)
   3251         self._consolidate_inplace()
   3252         if self._AXIS_REVERSED:
-> 3253             return self._data.as_matrix(columns).T
   3254         return self._data.as_matrix(columns)
   3255 

~/.local/lib/python3.6/site-packages/pandas/core/internals.py in as_matrix(self, items)
   3448             return mgr.blocks[0].get_values()
   3449         else:
-> 3450             return mgr._interleave()
   3451 
   3452     def _interleave(self):

~/.local/lib/python3.6/site-packages/pandas/core/internals.py in _interleave(self)
   3475         for blk in self.blocks:
   3476             rl = blk.mgr_locs
-> 3477             result[rl.indexer] = blk.get_values(dtype)
   3478             itemmask[rl.indexer] = 1
   3479 

~/.local/lib/python3.6/site-packages/pandas/core/internals.py in get_values(self, dtype)
   1695         if is_object_dtype(dtype):
   1696             return lib.map_infer(self.values.ravel(),
-> 1697                                  self._box_func).reshape(self.values.shape)
   1698         return self.values
   1699 

---

It appears from the above that jedi is coming to call df.values. Running this on my DataFrame takes 5-10 seconds, which is quite a slow thing to do during tab completion.

I've searched jupyter, ipython, ipykernel and jedi's issue trackers and didn't find anyone else having the problem, so I'm wondering if it is something peculiar to my DataFrame. I also wonder if the issue is quite hard to attribute to jedi, so maybe people are experiencing the issue and failing to know where to report it. I had difficulty making a reproducing case I could share, since it requires both a timestamp column and a non-timestamp column (likely the mixed types are what make df.values slow, I guess).

[davidhalter]

@pwaller Can you check the dev branch? It might be the issue https://github.com/davidhalter/jedi/pull/922.

[davidhalter]

See also #919.

[pwaller]

A large repr seems unlikely to be the explanation. The repr is 2kiB and takes 4ms to generate. Note also, that if I interrupt at a random moment, the stack trace I get (see PR body) does not contain __repr__ - it contains .values.

Have tested the dev branch, and get the same result.

[pwaller]

Is there a quick workaround to prevent this from hanging? It's harming my workflow currently, since I keep accidentally tab completing on something which cases the hang. So for now I'm disabling it with c.IPCompleter.use_jedi = False.

See also this issue on IPython about slow tab completion.

[mangecoeur]

Seeing similar issues, and likewise don't think it's caused by the string repr. Only solution I found for now is to disable jedi completion. I suspect (although I haven't dug into this), that jedi is recursively inspecting every object within the dataframe - possibly it looks through getitem/getattr recursively, and since a dataframe gives access to each column series, and each series gives access to each item in this way, it might result in it searching every item for completions (possibly even twice, because the Series object also has the .values attribute which points to the underlying arrays, which might get searched separately)

[mangecoeur]

p.s. A seemingly related issue when using xarray to work with lazy-loaded datasets - requesting tab-completion triggers the load of the full dataset (not sure why but again, guess it's something to do with recursively querying the object). This basically kills my system as it tries to load multi-GB files into memory.

[davidhalter]

I can reproduce it. I'll try to find a solution.

[joonro]

I was having this issue as well and it seems updating jedi to 0.11.0 from 0.10.2 solved this for me. It would be great if others can confirm this. Thank you!

[davidhalter]

Yes, this one should be fixed. I'm pretty sure. Please tell me if there's still an issue. Thanks for noticing.