Vscode-jupyter: entering code and run in the interactive window is extremely slow

[StevenLi-DS]

Version: 2020.1.58038 (13 January 2020)

I have a dataframe with shape of (810103, 26). If I type code and run a cell from the editor, everything is fine. But it will be extremely slow if I run code from the interactive window directly such as df.head(), df.columns. I can even make a screenshot without shortcut😂

[rchiodo]

That's really weird. We have reports of the opposite. Running from the editor being really slow, but directly from the interactive window being fast.

The problem is we can't repro in house. Do you have some code and data you can share?

[StevenLi-DS]

I found the issue when cleaning this CDC dataset. Just read the csv file and then df.head()

I also tried with the toy dataset tips in seaborn, which didn't cause any problem. I'm using 2019.11.50794 now, which has no issues.

[rchiodo]

Hmm. I'm reproing. Only goes slow from the interactive window directly. And it's the kernel that's taking a super long time (not our code). But if I run it from the editor, df.head() is instant.

[rchiodo]

And we submit the same code in both cases. I'm guessing this is a kernel issue.

In the editor case, we run two executions actually:

*** REQUEST EXECUTE : {"code":"__file__ = 'd:\\\\Training\\\\SnakePython\\\\foo.py'","stop_on_error":false,"allow_stdin":true,"store_history":false} 
*** REQUEST EXECUTE : {"code":"df.head()\n","stop_on_error":false,"allow_stdin":true,"store_history":true}

whereas in the interactive window we only run the second. And running both of those in the interactive window has no effect on the execution, it still takes a long time.

[rchiodo]

And this is not a regression. It repros with 2019.11.50794

[rchiodo]

I take that back. Only sometimes with 2019.11.50794.

[StevenLi-DS]

Hi, @rchiodo. just wondering... will this be fixed any time soon?

[rchiodo]

Not likely. We have no idea what causes it. It's jupyter that's taking all the time (not our code), so we'd have to debug the jupyter kernel.

[StevenLi-DS]

@rchiodo I don't know what the advantage of this jupyter kernel thing but it adds some env I don't want:

I feel like I prefer the old extension approach better.

[rchiodo]

Those two new 'kernels' represent the environment you're running your python code in. They are an exact match for an environment generated by conda.

The advantage is that they will load all of the same packages that the associated conda environment loads.

There is no disadvantage other than they are listed there. They have nothing to do with the slowdown you're seeing.

The slowdown is in the IPython code itself (which is what I meant by we had to debug the jupyter kernel). The slowdown is not in the code we ship with the ms-python extension.

[StevenLi-DS]

@rchiodo any process on this buddy? This happens so frequently and even with small datasets from time to time.

[DonJayamanne]

Do you have the variables list open when this issue occurs?

[StevenLi-DS]

not at all I never used that thing

[rchiodo]

Sorry but we are not actively investigating this. I believe this is an issue in the IPython code itself and not our own. We don't do anything special to send code to the kernel.

[StevenLi-DS]

Could you guys please take a look. This is so weird and really interrupt my workflow.

[rchiodo]

I have noticed if I interrupt the kernel when this is taking a long time it comes back immediately. At least for df.head()

[rchiodo]

Some of the advice here seems to alleviate the problem:
https://github.com/jupyter/notebook/issues/3224

I typed this in to the interactive window:

%config Completer.use_jedi = False

and now it responds immediately. At least with the csv you mentioned above.

[rchiodo]

That config line causes Jupyter to stop giving us autocomplete information. So you won't get column names with autocomplete anymore, but that's probably worth it for the speedup

I take that back, it still seems to work. Not sure what the turning off of Jedi does then.

[StevenLi-DS]

I don't even use Jedi but only Microsoft language server

[rchiodo]

This setting is not for us. It's for jupyter. They use jedi as well for autocomplete.

[StevenLi-DS]

If I encounter this issue and try to Interrupt the kernel, the kernel will be forced to restart.
This is really annoying since sometimes it will take a long time to clean some data and you have to re-run everything.

[rchiodo]

So you're saying the config setting doesn't work?

%config Completer.use_jedi = False

[rchiodo]

And interrupting the kernel doesn't always reset the kernel. Depends upon the thing being interrupted.

[StevenLi-DS]

So you're saying the config setting doesn't work?

I'm a little bit confused about it. Should I set use_jedi to False? Because based on your comments, it seems like you later found an issue with it. Is there any side effects of it?

And interrupting the kernel doesn't always reset the kernel. Depends upon the thing being interrupted.

This is why I don't expect to restart the kernel and now it always does.

[rchiodo]

Did you mean this comment?

That config line causes Jupyter to stop giving us autocomplete information. So you won't get column names with autocomplete anymore, but that's probably worth it for the speedup

I take that back, it still seems to work. Not sure what the turning off of Jedi does then.

Sorry that was confusing.

I was worried turning of jedi support in jupyter would prevent jupyter from returning autocompletions, but it does not.

So I don't see any side effects to this setting:

%config Completer.use_jedi = False

[StevenLi-DS]

But this will turn off all the autocompletion while I enter code in the interactive window right? and do I have to do this whenever I launch the vscode or the kernel?

[rchiodo]

No. It doesn't. That's why I said this:

I was worried turning of jedi support in jupyter would prevent jupyter from returning autocompletions, but it does not.

After changing that setting I was still able to get column names from panda dataframes. This data is not provided by the python extension and explicitly comes from jupyter. So Jupyter still seems to provide autocomplete data even after its jedi usage is turned off.

[rchiodo]

You should be able to put this in your startup commands:

[rchiodo]

Finally had a chance to profile the IPython kernel. The autocomplete from typing is hanging the kernel and preventing the execution of a cell. We should be able to fix this by interrupting the kernel if it's in the middle of autocompletion.

[radusuciu]

Some of the advice here seems to alleviate the problem:
jupyter/notebook#3224

I typed this in to the interactive window:

%config Completer.use_jedi = False

and now it responds immediately. At least with the csv you mentioned above.

This fixed the issue for me as well.