Papermill: Kernel died while waiting for execute reply.

[mkoliba]

Hello I was trying to run heavy notebooks with lot of plots on Ubuntu Server in python virtual environment with papermill and I run into following issue:

python papermill_exec.py >> log.txt
Executing:  34%|                                                                                         | 24/71 [01:03<01:04,  1.38s/cell]Kernel died while waiting for execute reply.
Executing:  35%|                                                                                     | 25/71 [01:08<02:06,  2.75s/cell]
Traceback (most recent call last):
  File "/home/username/thesis/venv/lib/python3.7/site-packages/nbclient/client.py", line 414, in _poll_for_reply
    msg = self.kc.shell_channel.get_msg(timeout=timeout)
  File "/home/username/venv/lib/python3.7/site-packages/jupyter_client/blocking/channels.py", line 57, in get_msg
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "papermill_exec.py", line 31, in <module>
    execute_paper_mill(dist_name, optimisation_name, distance, file_name)
  File "papermill_exec.py", line 12, in execute_paper_mill
    dist_file_name=dist_file_name)
  File "/home/username/venv/lib/python3.7/site-packages/papermill/execute.py", line 104, in execute_notebook
    **engine_kwargs
  File "/home/username/venv/lib/python3.7/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/home/username/venv/lib/python3.7/site-packages/papermill/engines.py", line 343, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/home/username/venv/lib/python3.7/site-packages/papermill/engines.py", line 402, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "/home/username/venv/lib/python3.7/site-packages/papermill/clientwrap.py", line 38, in execute
    self.papermill_execute_cells()
  File "/home/username/venv/lib/python3.7/site-packages/papermill/clientwrap.py", line 65, in papermill_execute_cells
    self.execute_cell(cell, index)
  File "/home/username/venv/lib/python3.7/site-packages/nbclient/client.py", line 563, in execute_cell
    exec_reply = self._poll_for_reply(parent_msg_id, cell, timeout)
  File "/home/username/venv/lib/python3.7/site-packages/nbclient/client.py", line 419, in _poll_for_reply
    self._check_alive()
  File "/home/username/venv/lib/python3.7/site-packages/nbclient/client.py", line 446, in _check_alive
    raise DeadKernelError("Kernel died")
nbclient.exceptions.DeadKernelError: Kernel died

Upgrading python to version 3.7 and pip to 20.0.2 did not helped.

[MSeal]

Check if you're sending a lot (100's per second) of display or print messages. If you send them at a very rapid rate the ZMQ channel used to communicate between the kernel and the notebook executor can overflow resulting in an empty queue error.

Alternatively the code you're running inside the notebook may have been killed by OOM process or by a segfault. Check that you're not running out of resources and if you're using any low level systems that they're not dumping segfault logs.

[Arullaldivakar]

Im also facing the same issue , @mkoliba were you able find the root cause of the issue

[MSeal]

@Arullaldivakar I have found also many cases it's an OOM issue if you haven't checked that, make sure you're not running out of memory.

[mckeown12]

I just ran into this same issue inside docker. In my case it was that my pyarrow version was incompatible with snowflake connector version. Exporting my_notebook.ipynb to a my_script.py and running with python made the error message less opaque:

UserWarning: You have an incompatible version of 'pyarrow' installed, please install a version that adheres to: 'pyarrow<0.18.0,>=0.17.0; extra == "pandas"'
  warn_incompatible_dep('pyarrow', _installed_pyarrow.version, _expected_version)
Segmentation fault