Vscode-python: No breakpoint for Uncaught Exceptions while debugging tests

[ChadBailey]

UPDATE: Seems this is an enhancement that may be "coming soon". If you've found this bug report in hopes of a fix, and are willing to accept the risks look at this reply for instructions on manually enabling the functionality

Searched for duplicates, none found. See the following post/thread for back story: https://github.com/microsoft/vscode-python/issues/850#issuecomment-696992737

Environment data

Version: 1.49.0
Commit: e790b931385d72cf5669fcefc51cdf65990efa5d
Date: 2020-09-10T17:39:53.251Z (1 wk ago)
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 18.7.0

• Extension version (available under the Extensions sidebar): v2020.8.109390
• OS and version: macOS Mojave 10.14.6 (bug happens on every OS I've tried - Linux Mint & Windows 10)
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.1 (64 bit) - appears unrelated
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: pytest-5.4.3
• Relevant/affected Python-related VS Code extensions and their versions: N/A
• Value of the python.languageServer setting: Microsoft Python Language Server version 0.5.51.0

[Info  - 9:39:50 AM] Analysis cache path: /Users/####/Library/Caches/Microsoft/Python Language Server
[Info  - 9:39:50 AM] Microsoft Python Language Server version 0.5.51.0
[Info  - 9:39:50 AM] Workspace root: /Users/####/Documents/Code/####
[Info  - 9:39:51 AM] GetCurrentSearchPaths /usr/local/opt/[email protected]/bin/python3 
[Info  - 9:39:51 AM] Interpreter search paths:
[Info  - 9:39:51 AM]     /usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8
[Info  - 9:39:51 AM]     /usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload
[Info  - 9:39:51 AM]     /Users/####/Library/Python/3.8/lib/python/site-packages
[Info  - 9:39:51 AM]     /usr/local/lib/python3.8/site-packages
[Info  - 9:39:51 AM] User search paths:
[Info  - 9:39:55 AM] Initializing for /usr/local/opt/[email protected]/bin/python3
[Info  - 9:39:55 AM] Analysis caching mode: None.

Expected behaviour

When debugging a unit test with the breakpoint of Uncaught Exceptions enabled, the debugger should break on the exception rather than finishing the test as a failure. This is the behavior when invoking the debugger via the typical way through Run -> Start Debugging.

Actual behaviour

When debugging a unit test with the breakpoint of Uncaught Exceptions enabled, the debugger does not break on the exception and finishes as a failed test.

Steps to reproduce:

1. Create new folder with the following files:

main.py

def main():
    raise Exception("Demo")


if __name__ == "__main__":
    main()

test_main.py

from main import main


def test_main():
    main()

1. Begin debugging (F5) main.py with standard debug options
2. Observe that debugger breaks on exception, allowing you to use the interactive debugging console

-> main
 > <function main at 0x108dbd940>

1. Install pytest & discover tests (may need to add "python.testing.pytestEnabled": true to settings.json)
2. Click Debug Test over test_main in test_main.py
3. Observe that the debugger does not break and instead the test results come back as a failure

============================= test session starts ==============================
platform darwin -- Python 3.8.1, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /Users/####/Documents/Code/test_demo
collected 1 item

test_main.py F                                                           [100%]

=================================== FAILURES ===================================
__________________________________ test_main ___________________________________

    def test_main():
>       main()

test_main.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def main():
>       raise Exception("Demo")
E       Exception: Demo

main.py:2: Exception
- generated xml file: /var/folders/lz/hcbk8xvj2gdf9htx6r3w6cmrgy1jf9/T/tmp-79341Yr1at8rMslzO.xml -
=========================== short test summary info ============================
FAILED test_main.py::test_main - Exception: Demo
============================== 1 failed in 0.15s ===============================

Logs

Intentionally left out, due to difficulty parsing out private data. See logs above

[ChadBailey]

Update:

In response to @qubitron 's suggestion, I have attempted to disable justMyCode without success.

launch.json

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python: Current File",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "justMyCode": false
    }
  ]
}

Could not find an additional place to configure this setting (i.e. within settings python.test.*)

[ChadBailey]

Update

Oddly enough seems the latest update may have resolved this in spite of the conspicuous omission in the changelog and the fact that it has been a long-standing bug for me spanning multiple systems. I haven't tried on my mac yet (the machine I submitted this report on), but it's now breaking as expected in Windows 10.

Version: 1.49.1 (user setup)
Commit: 58bb7b2331731bf72587010e943852e13e6fd3cf
Date: 2020-09-16T23:27:51.792Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Windows_NT x64 10.0.18363

Python extension version: v2020.9.111407

[karthiknadig]

@ChadBailey We did not change any thing in the extension specific to exception handling. I am not sure what changed to get this working.

The issue here is that the debugger debugpy supports this, we call this feature userUnhandled. When running with pytest all exceptions are technically handled by pytest. From python's point of view there won;t be any uncaught exceptions. But in the debugger there is a mode which treats any exception crossing user code to library code boundary as user-uncaught. Previously we tried to expose this feature in the extension had to immediately revert it because people did not like the default and could not agree on if they want default on or off. So for now we removed it from the extension. Debugger fully supports this as it is needed by Visual Studio 2019 nd can be set via the Exception settings.

[ChadBailey]

It is entirely possible the issue was localized as I did originally have the extension Python Test Explorer installed and removed it as a troubleshooting step. Perhaps the update just managed to cause something to refresh? I'm unsure, I will report back once I'm able to test on my mac.

Meanwhile, if it's of any help I have also been getting some strange console behavior since the update which I've never seen before: Error: TypeError: message must be set

[karthiknadig]

Can you file a separate bug for this: Error: TypeError: message must be set?

[ChadBailey]

Can you file a separate bug for this: Error: TypeError: message must be set?

Done, #14067. I'm not certain the error isn't due to something on my end, so I'm not sure I would have filed if not for you suggesting. Hoping it's helpful.

[ChadBailey]

Update

I've completely wiped all settings and extensions... This is definitely not fixed on my mac yet (which was upgraded to Catalina 10.15.6 since I made this report yesterday)

It would be helpful if someone else could weigh in if they are getting this behavior or not.

[int19h]

The unhandled exception behavior is currently by design in VSCode due to the way "unhandled" is defined, as described above. We're planning to add some UX to expose user-unhandled in exception filters, but that's tracked as a feature.

[ChadBailey]

@int19h I'm having trouble following your comments. It doesn't matter to me what you call it, I really need a way for the debugger to break on exceptions while running tests. This is almost mandatory for debugging longer integration/E2E tests. Sure in a small unit test it's just a minor inconvenience. When you have an E2E test suite that takes 15 minutes to run though, a silently failing exception when the user explicitly clicked "debug" is not a great user experience.

[int19h]

Absolutely! That's why debugpy implemented this feature already. It just takes time to figure out how to best surface it in VSCode UI. I was just saying that this is not a regression or an unexpected malfunction, just the unfortunate consequence of not having the feature there yet.

(@luabud for UX considerations)

[ChadBailey]

Ah, my apologies I understand now. Thanks for clearing that up. I didn't realize the feature didn't originally exist. In my opinion - it's worth as much as you paid for it - I think that following the setting for "Uncaught Exceptions" would be an intuitive approach. The only caveat is, in the event where the end user is issuing a command and intentionally checking for a specific exception that of course should not break (see below).

Example

main.py

def main():
    raise Exception("Demo")


if __name__ == "__main__":
    main()

test_main.py

import pytest
from main import main

# _SHOULD_ break on exception with `Uncaught Exceptions` selected
def test_main():
    main()  # Note: Main raises Exception("Demo")

# Should _NOT_ break on exception with `Uncaught Exceptions` selected
def test_main():
    with pytest.raises(Exception):
        main()  # Note: Main raises Exception("Demo")

I hope this is helpful.

[int19h]

We can't change the meaning of "uncaught" to mean "user-uncaught", because it's a standard exception filter, and its current behavior is required for VS to behave as it should. So this has to be another, distinct filter (which can still be enabled by default). VS, which already supports this, hides that fact by picking one or the other category depending on Just My Code setting. But in VSCode, the filters that are advertised by the debug adapter are exposed directly by debugging UI - the extension doesn't control that part. So, the challenge is to add that third filter in a way that makes the distinction between the two sufficiently clear.

You can edit your local copy of debugpy (bundled with the extension, wherever it's installed) to try it out already. Just uncomment this line:

https://github.com/microsoft/debugpy/blob/cf65911a7acbc7ae8d4edd47d00e84cf27fadada/src/debugpy/adapter/clients.py#L165

This will make the filter show up. However, note that its usability is hampered by https://github.com/microsoft/debugpy/issues/399 at the moment.

[ChadBailey]

So, the challenge is to add that third filter in a way that makes the distinction between the two sufficiently clear.

I see, thank you so much for taking the time to explain. If it helps at all, I often mis-call this Unhandled Exceptions. Perhaps that would be a succinct way of putting it that users would understand. Perhaps equally as likely is that this only makes sense in the reality which resides between my ears.

You can edit your local copy of debugpy (bundled with the extension, wherever it's installed) to try it out already.

This is super helpful! Thank you so much for showing me this, I may actually do that... this can become very disruptive when unexpected things break and I find myself chasing down all the possible places an exception could be raised.

[ChadBailey]

You can edit your local copy of debugpy (bundled with the extension, wherever it's installed) to try it out already. Just uncomment this line:

https://github.com/microsoft/debugpy/blob/cf65911a7acbc7ae8d4edd47d00e84cf27fadada/src/debugpy/adapter/clients.py#L165

Woohoo!! I got it working!

Here's some steps for posterity:

1. Open VSCode and visit console Ctrl+`

2. Click the OUTPUT tab, select Python from the dropdown

3. Hit Ctrl+F and search for pythonFiles (Note: You may need to issue a command such as discover tests first)

4. Copy discovered system path to pythonFiles (i.e. ~/.vscode/extensions/ms-python.python-2020.9.111407/pythonFiles/) then open this folder using VSCode

5. Search all files Ctrl+Shift+F for the term userUnhandled

6. Select the search result found in clients.py and modify line to be the following

               {"filter": "userUnhandled", "label": "User Uncaught Exceptions", "default": True},
   

   NOTE: I think I originally had a syntax error causing this, but it did not work until I toggled default to True

7. Save file and restart VSCode