Ipython: Hang after running pdb in a cell, Kernel/Interrupt doesn't help

[nealmcb]

If you debug a cell via pdb, then don't quit it before rerunning the cell, you can't get back to your IPython environment. Reproduce it via Test case for pdb hang in Jupyter notebook. Here's a screenshot:

For the record and search engines, to hang your notebook:

• Run the following cell
• Do not "quit" via the pdb prompt
• Then run it again.

def test():
    import pdb; pdb.set_trace()  # XXX BREAKPOINT
    return 0

test()

Note that Kernel/Interupt doesn't help. You can save the notebook, but to run cells again, you need to restart the kernel, at which point "All variables will be lost"

This happens to me not infrequently if I am searching thru the rest of the notebook for hints on how to debug the cell, then forget to quit before changing the cell and running it again. Yes, I can be more careful, but others have the same experience. Thanks to @wmayner for a convenient test case described at #3400.

I'm running Jupyter notebook 4.3.1 on Python 3.4.3 (default, Nov 17 2016, 01:08:31) [GCC 4.8.4]
Kernel Information: Python 3.4.3 (default, Nov 17 2016, 01:08:31)

Input from @takluyver at #10499

pdb runs in the same process, and we do know when it's expecting input. I think it needs someone to do the work so that when the HTML input box is removed in the frontend, a message is sent back to the kernel which will raise EOFError.
That may need a new message type, or we could do it as a metadata field on the existing stdin reply message.

[danolson1]

I am running into the same problem.

[iNLyze]

Absolutely, same here. This is a royal pain in the rather sensitive parts of my body. If a convenient way to stop pdb from without the running cell existed this could be alleviated. Alternatively, at least a way of saving your work before shutting down the notebook would be important (e.g. trained neural network and the notebook crashes on save after 2000 epochs).

[vsbuffalo]

I also have this issue, with Jupyter version 4.30, Python 3.6.2. Restarting the kernel appears to be the only way to handle this issue.

[ihincks]

Just lost an 8 hour calculation due to this...time to start again.

[hsorsky]

Just lost a 64 hour calculation on which I'm meant to be presenting results today -.-

No one found a solution?

[WilliamCarlos]

Just got bitten by this as well. How do people typically debug in jupyter notebook? Using pdb is like juggling a grenade :(

[olabreche]

Please please please fix this! I am constantly getting this issue.

[cristianmtr]

Same here. Have to redo a very long calculation

[joel-odlund]

same here.

[zsal]

someone please fix and I will pay you $5

[sbliven]

@zsal If you're serious, you could add this is issue on https://www.bountysource.com

[jerrykaplan]

Same problem here. Need a way to interrupt a runaway process or one that is "waiting" on input and get control back ... even if I can no longer get to the input prompt.

I'm a newby - How on earth do advanced programmers debug in notebooks? Surely they don't use PDB with this "time bomb" in it? Please, share your secrets! :)

[handrew]

+1

[carrutstick]

+1

[ShapingView]

+1

[daveliu-riviera]

+1

[louismartin]

Up!
This problem has been a pain for me for years now every time I use pdb and forget to quit before I re-run the cell.

[louismartin]

I created a bounty on BountySource. Maybe this will finally be fixed if we can gather enough money.
https://www.bountysource.com/issues/44958889-hang-after-running-pdb-in-a-cell-kernel-interrupt-doesn-t-help

[murtazacornerstone]

+1

[michaelkyu]

+1

This has driven me crazy for years.

[nekketsuuu]

How about using IPython.core.debugger.Pdb instead?

from IPython.core.debugger import Pdb; Pdb().set_trace()

(Or %debug magic can be used for debugging.)

[itamarst]

Running Pdb().set_trace() twice has the same issue.

[itamarst]

I am investigating this. The basic way interrupts work is that they (on Unix) send a signal to the process, which then raises KeyboardInterrupt.

There are appear to be multiple issues with this happening:

1. SIGINT won't interrupt zeromq. This is fixed hopefully by https://github.com/zeromq/pyzmq/pull/1368
2. Even once zeromq is interruptible, I believe pdb has a whole bunch of KeyboardInterrupt-catching code that prevents it from percolating up. This may perhaps be fixable in IPython.core.debugger.Pdb.

[minrk]

1. is not accurate, zeromq does get interrupted by SIGINT and raises KeyboardInterrupt, at least unless the SIGINT handler has itself been overridden.

[itamarst]

@minrk is correct and I was wrong, sorry.

[itamarst]

To get this fix in, the following needs to happen:

• [x] Merge https://github.com/ipython/ipython/pull/12168
• [x] New IPython release
• [x] Merge https://github.com/ipython/ipykernel/pull/490
• [x] Merge https://github.com/ipython/ipykernel/pull/498
• [ ] New IPykernel release

[louismartin]

Thanks so much for looking into this!!

[itamarst]

As soon as there is a new IPykernel release, this can be closed as fixed.

[itamarst]

Done! This can be closed.

[vityas]

import IPython
import ipykernel
print(IPython.__version__)
print(ipykernel.__version__)
IPython.core.debugger.set_trace()

7.15.0
5.3.0

``` 250 sys.stdout.flush()
251
--> 252 def __call__(self, result=None):
253 """Printing with history cache management.
254

--KeyboardInterrupt--
--KeyboardInterrupt--
```

KeyboardInterrupt seems to do nothing in my case. Is that the expected behavior?

[itamarst]

I would expect that to interrupt. I will see if I can reproduce. This is Windows, yes?

[vityas]

macOS 10.15.5

Python 3.6.2 (default, May  4 2018, 19:40:30)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin

Oh, so this is not expected. Let me know if there is any info I can add to clarify.

pip list
```Package Version

---

appnope 0.1.0
attrs 19.3.0
backcall 0.2.0
bleach 3.1.5
decorator 4.4.2
defusedxml 0.6.0
entrypoints 0.3
ipykernel 5.3.0
ipython 7.15.0
ipython-genutils 0.2.0
ipywidgets 7.5.1
jedi 0.17.1
Jinja2 2.11.2
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.3
jupyter-console 6.1.0
jupyter-core 4.6.3
MarkupSafe 1.1.1
mistune 0.8.4
nbconvert 5.6.1
nbformat 5.0.7
notebook 6.0.3
packaging 20.4
pandocfilters 1.4.2
parso 0.7.0
pexpect 4.8.0
pickleshare 0.7.5
pip 20.0.2
prometheus-client 0.8.0
prompt-toolkit 3.0.5
ptyprocess 0.6.0
Pygments 2.6.1
pyparsing 2.4.7
pyrsistent 0.16.0
python-dateutil 2.8.1
pyzmq 19.0.1
qtconsole 4.7.5
QtPy 1.9.0
Send2Trash 1.5.0
setuptools 46.0.0
six 1.15.0
terminado 0.8.3
testpath 0.4.4
tornado 6.0.4
traitlets 4.3.3
wcwidth 0.2.5
webencodings 0.5.1
wheel 0.34.2
widgetsnbextension 3.5.1```

[mhellb]

For me problem also persisted after updating. Also on the remote linux compute nodes I use for work, see:

import IPython
import ipykernel
import os
import platform

print(IPython.__version__)
print(ipykernel.__version__)
print(os.name,platform.system(),platform.release())
IPython.core.debugger.set_trace()

7.15.0
5.3.0
posix Linux 4.19.94-300.el7.x86_64

Currently working in a pretty bulky environment so cannot rule out some dependencies might be causing it? I can try to see if the error also reproduces in a minimal environment if that would be useful?

[Carreau]

This needs master of IPykernel that has not been released yet.