Dart-code: Run All Test fails with SIGINT

[ComplexCarbos]

I'm seeing a strange issue where I can no longer execute all my tests using Dart: Run All Tests.

All tests run successfully when invoking via Visual Studio Code terminal: flutter test.

When executing Dart: Run All Tests from the command palette some tests execute, but then the process fails:

✓ Game News Item Get Contents ✓ Game News Item Get Author Shell subprocess terminated by ^C (SIGINT, -2) after tests finished. Test: /home/rich/Documents/visual_studio_code_projects/GamerNews/test/screens/news_list_test.dart Shell: /home/rich/Development/flutter/bin/cache/artifacts/engine/linux-x64/flutter_tester dart:async/stream_controller.dart 595:43 _StreamController.addError dart:async/stream_controller.dart 862:13 _StreamSinkWrapper.addError package:stream_channel/src/guarantee_channel.dart 140:14 _GuaranteeSink._addError package:stream_channel/src/guarantee_channel.dart 131:5 _GuaranteeSink.addError package:flutter_tools/src/test/flutter_platform.dart 576:29 _FlutterPlatform._startTest.<fn> ===== asynchronous gap =========================== dart:async/zone.dart 1053:19 _CustomZone.registerUnaryCallback dart:async/runtime/libasync_patch.dart 77:23 _asyncThenWrapperHelper package:flutter_tools/src/test/flutter_platform.dart _FlutterPlatform._startTest.<fn> package:flutter_tools/src/test/flutter_platform.dart 782:26 _FlutterPlatform._startTest ===== asynchronous gap =========================== dart:async/zone.dart 1053:19 _CustomZone.registerUnaryCallback dart:async/runtime/libasync_patch.dart 77:23 _asyncThenWrapperHelper package:flutter_tools/src/test/flutter_platform.dart _FlutterPlatform._startTest package:flutter_tools/src/test/flutter_platform.dart 462:36 _FlutterPlatform.loadChannel package:flutter_tools/src/test/flutter_platform.dart 417:46 _FlutterPlatform.load ===== asynchronous gap =========================== dart:async/zone.dart 1053:19 _CustomZone.registerUnaryCallback dart:async/runtime/libasync_patch.dart 77:23 _asyncThenWrapperHelper package:test_core/src/runner/loader.dart Loader.loadFile.<fn> package:test_core/src/runner/load_suite.dart 98:31 new LoadSuite.<fn>.<fn> ===== asynchronous gap =========================== dart:async/zone.dart 1045:19 _CustomZone.registerCallback dart:async/zone.dart 962:22 _CustomZone.bindCallbackGuarded dart:async/timer.dart 52:45 new Timer dart:async/timer.dart 87:9 Timer.run dart:async/future.dart 174:11 new Future package:test_api/src/backend/invoker.dart 399:21 Invoker._onRun.<fn>.<fn>.<fn> ✓ NewsList Screen with no news. Exited (sigint)

Sometimes, a notification pops up that says: "No Debug Adapter Found"

Flutter doctor output:

`rich@rich-GS73VR-7RF:~/Documents/visual_studio_code_projects/GamerNews$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, v1.2.1, on Linux, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[!] Android Studio (version 3.3)
✗ Flutter plugin not installed; this adds Flutter specific functionality.
✗ Dart plugin not installed; this adds Dart specific functionality.
[✓] VS Code (version 1.33.1)
[!] Connected device
! No devices available

! Doctor found issues in 2 categories.`

Capture Log output is attached.

Dart-Code-Log-2019-04-03 08-15-03.txt

The logs suggest the test process was interrupted because it couldn't find one of the test files. But I can see that file in the editor and it does exist on the file system:

rich@rich-GS73VR-7RF:~$ ls -l /home/rich/Documents/visual_studio_code_projects/GamerNews/test/screens/game_list_test.dart -rw-r--r-- 1 rich rich 2728 Apr 30 21:05 /home/rich/Documents/visual_studio_code_projects/GamerNews/test/screens/game_list_test.dart rich@rich-GS73VR-7RF:~$

[DanTup]

Hmmm, the sigint is being sent because the debug adapter says it was told to terminate:

[08:15:13 GMT-0400 (EDT)] [FlutterTest] [Info] <== [{"event":"test.startedProcess","params":{"observatoryUri":"http://127.0.0.1:32947/"}}]
[08:15:13 GMT-0400 (EDT)] [Observatory] [Info] Connecting to VM Service at http://127.0.0.1:32947/ws
[08:15:13 GMT-0400 (EDT)] [Observatory] [Info] Disconnect requested!

The last line here is in the disconnectRequest handler which is normally called by VS Code when it wants to stop a session (for ex. if you click the Stop button on the debug toolbar):

https://github.com/Dart-Code/Dart-Code/blob/c695c3bed261b5d0662ff47675bf2083b1c25247/src/debug/dart_debug_impl.ts#L427-L431

I think the "failed to load" message is just a result of us starting to terminate processes as a result of being told to quit.

Does this happen only for tests? Does it happen if you run a single test? Does it happen if you create the flutter create empty project and then try to run it from there?

[ComplexCarbos]

I only see this behavior when attempting to run all my tests by executing:

Dart: Run All Tests.

I can run individual tests (clicking run or debug in the source file).

I can execute all tests in a single file by right clicking and selecting "Start Debugging".

I just created an empty project through the IDE. I made no changes to the default project. I was able to execute Dart: Run All Tests.

✓ Counter increments smoke test
Exited

[DanTup]

Thanks, I'll see what I can find. Based on your comment above, I believe this has worked for you in the past sometime? Do you know when it might've become broken?

[ComplexCarbos]

Yep, you are correct. It was working fine until last night. All I did last night was add a test file with several new tests. Deleting the new test file doesn't do anything - still see the same error. Since I can run all tests with no errors from the command line, I thought I might have stumbled on some obscure issue with the VS Code - flutter integration.

The only change that sticks out in my mind is running an Ubuntu software update in the morning.

Thank you for looking!

[DanTup]

Weird, I can't repro here, and the logs make no sense. I don't see why termination is being requested.

Could you try making a copy of your project and trimming your tests down, to see if you can make a small reproducible project that you could share? You can email it to [email protected] if you don't want to attach publicly.

Also - do you have easy access to a Windows or macOS machine to see whether it repros there? (I'm currently testing on macOS, though we do have automated tests on Travis that include Linux).

If that gets us nowhere, I can make a custom build of the extension with more verbose logging to help track it down.

[DanTup]

Hmmm, looking at your log again, I see something a bit weird going on that maybe we've never accounted for. When we run multiple files like this, we actually just run a single command:

[08:15:10 GMT-0400 (EDT)] [FlutterTest] [Info] Spawning /home/rich/Development/flutter/bin/flutter with args ["test","--machine","--start-paused","test"]

We expect a single VM debugger connection, however the logs show that we were given multiple - one for each file. I wonder if what's happening here is that one file is finishing, which disconnects our VM, which causes us to signal the end of the run. I'll try making some longer-running test files so they don't all finish quickly and see if I can find anything.

[DanTup]

Shell subprocess terminated by ^C (SIGINT, -2) after tests finished.

Aha, looks like I've reproduced it. We might have to change how this works since my assumptions about how flutter test would run multiple files don't seem to be accurate.

[DanTup]

Right, I think I've tracked this down. In 7a2a372334508f91f4e829aa679ff1e5d547ed7d we enabled connecting to the VM service even when running "without debugging", in order to allow VM-service features to work (toggle debug banners, performance overlays, etc.).

This resulted in us connecting the VM when running tests too - and based on the commit message for 5d10d7ffa9c7606fc98d379232f07cf85f81c689 I previously knew this wouldn't work for Flutter.

So, I should be able to fix by not trying to connect to the VM service for Flutter Test runs in noDebug mode. First I'd like to try and get some failing tests around this though. I don't know how quick I'll ship a fix, but once I have one I could at least make you a build that includes it.

[ComplexCarbos]

Awesome. No hurry on this - I have a work around with flutter test. Anything more you need from me?

Really appreciate you looking into the issue.

[DanTup]

I don't think so - I've got tests that fail now (runs all tests if given a folder), and a fix running that makes them go green locally, so hopefully all sorted :-)

A version with the fix is likely mid-end of May. If it gets annoying enough that you want the fix sooner, just shout - it's not a big deal to make a build.

Thanks for reporting it and capturing the logs to help me track it down!

[jpv123]

Sorry to reopen the issue, it's happening to me in VSCode with a launch.json file configured like this:

{
        "name": "All Tests",
        "program": "test/",
        "request": "launch",
        "type": "dart",
}

Error launches

Failed to load "/Users/........../test/Widgets/ProjectCell_test.dart":
Shell subprocess terminated by ^C (SIGINT, -2) before connecting to test harness.

Running flutter test works correctly

[jpv123]

Forgot to add some important info sorry:

In test/Widgets folder I got only two testing files. One works correctly, the other one fails with the launch.json file. Both works with flutter test

I got the last version of VSCode 1.41.1
And the last version of flutter 1.12.13 + hotfix.5

[DanTup]

@jpv123 thanks, I can repro this intermittently. I think this is what https://github.com/Dart-Code/Dart-Code/issues/2082 is for, so let's use that issue for this.

[DanTup]

As a workaround, if you add "noDebug": true to the launch.json, it should fix it (albeit at the expense of debugging, and a warning from VS Code that it's not a valid option).