Notebook: Very large memory usage of the notebook server python process

[jorisvandenbossche]

Excuse me in advance for the possibly vague issue, but I am experiencing the following issue:

• I am using the notebook 4.0.4 on Windows 7
• If I look in the 'Windows task manager' -> 'Processes' tab, I see a python.exe process that takes up to several hundreds of MBs (eg now ca 750 Mb)
• This python process does not seem to be related to one of my notebook individual running kernels (if I shut down all running kernels, it is still there).
• However, it is gone when I shut down the notebook server and restart it.

If this number is true, this seems a problem to me (as the server only shouldn't take that much of memory). However, (I am no expert on this matter), Windows maybe only thinks that memory is being used but it is not actually used by the server? (are there ways I can figure this out?)

I have the vague observation that this python process of the server grows while some of the python processes of one of the kernels grows (due to the computation I am doing), but does not decrease when that kernel is restarted or shutdown.

[Carreau]

Hum I am not sure how to profile a python process on windows.
For comparison I'm around 35 Mb stable on OS X.

Might be possible to use something like https://pypi.python.org/pypi/memory_profiler , but I have never used it.

[jhamrick]

I experienced something similar a while back, where starting a python 3 kernel would use up ~2GB of memory for no particular reason. Can you verify that it is something having to do with the notebook server as opposed to a kernel -- i.e., does the process appear if you have just the notebook server running and no kernels? Or does it only appear once you start running kernels?

[CicoZhang]

I have the same problem, basically.

[brazilbean]

I am seeing the same problem. My notebook server has been running for several days and now uses 5GB (5,056,764K) of memory.

Note: by including the "Command Line" column in the Task Manager Processes tab, you can see what script each "python.exe" process actually is. This way you can tell which python processes are kernels vs the notebook server.

The guilty process is C:\Anaconda3\python.exe "C:\Anaconda3\Scripts\jupyter-notebook-script.py"

I agree with @jorisvandenbossche, the server should not be using this much memory. This is not good. :(

I don't like needing to restart my server every few days. To do this I have to find a time when all the notebooks can be closed; but at any given time I'm always working on overlapping tasks.

If anyone needs more information I would be happy to try to help. Thanks!

[brazilbean]

Could this be related to https://github.com/ipython/ipython/issues/8267?

[brazilbean]

Also possibly related: http://stackoverflow.com/questions/35494282/freeing-memory-tied-to-errors-in-ipython-notebook

[jhemmelg]

I am seeing the same thing on OSX. My workflow involves pointing the notebook at a directory and running all cells below, over and over. There are usually 5-10 matplotlib inline plots generated each run. I also set the savefig.dpi to 300, which pumps up the size of the figures. Activity Monitor currently shows my notebook server process with 3.58 GB memory used and 2.69 GB compressed which means most of the memory is not being accessed again after it is used initially. Could it be something with the handling of inline plots? I don't see the same memory usage when just doing some calculations without the plots. (Anaconda 2.5.0, jupyter 4.0.6, python 3.4, OSX 10.11.4)

[minrk]

Can you upgrade jsonschema to 2.5.1:

 pip install --upgrade jsonschema

? Anaconda still includes jsonschema 2.4, which has a major memory leak bug.

[jhemmelg]

I upgraded to jsonschema 2.5.1 as you suggested. That seems to fix the problem in my initial testing. I will keep monitoring memory usage and let you know if there is still a problem. Thanks!

[brazilbean]

Thank you @minrk for the suggestion. I also updated jsonschema. I'll let you know what I see.

[jhemmelg]

Update: a couple weeks later and my notebook server processes are staying under a hundred MB with jsonschema 2.5.1.

[jorisvandenbossche]

I am also not noticing this issue any more after updating jsonschema.
So closing this!