Vscode-java-test: Output Window: Failed to parse the message: %ERROR 2,

[spottedmahn]

Problem

\Unhandled Exceptions in @BeforeAll doesn't tell me what's going on. It just exits and puts a less than useful error message in the output window.

Failed to parse the message: %ERROR 2,miked.javatestsissue.LibraryTest.

---

Stepping through the code I see the "real" error:

Exception has occurred: org.junit.platform.commons.JUnitException
"org.junit.platform.commons.JUnitException: @BeforeAll method 'void miked.javatestsissue.LibraryTest.beforeAll() throws java.lang.Exception' must be static unless the test class is annotated with @TestInstance(Lifecycle.PER_CLASS)."

---

Version Info

Name: Java Test Runner
Id: vscjava.vscode-java-test
Description: Run and debug JUnit or TestNG test cases
Version: 0.22.1
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=vscjava.vscode-java-test

---

Verbose log

I enabled verbose logging and I see this:

%TESTC 1 v2
%TSTTREE2,miked.javatestsissue.LibraryTest,true,1,false,1,LibraryTest,,[engine:junit-jupiter]/[class:miked.javatestsissue.LibraryTest]
%TSTTREE3,testSomeLibraryMethod(miked.javatestsissue.LibraryTest),false,1,false,2,testSomeLibraryMethod(),,[engine:junit-jupiter]/[class:miked.javatestsissue.LibraryTest]/[method:testSomeLibraryMethod()]
Failed to parse the message: %ERROR 2,miked.javatestsissue.LibraryTest.%ERROR 2,miked.javatestsissue.LibraryTest
%TRACES
org.junit.platform.commons.JUnitException: @BeforeAll method 'void miked.javatestsissue.LibraryTest.beforeAll() throws java.lang.Exception' must be static unless the test class is annotated with @TestInstance(Lifecycle.PER_CLASS).
at org.junit.jupiter.engine.descriptor.LifecycleMethodUtils.assertStatic(LifecycleMethodUtils.java:57)
at org.junit.jupiter.engine.descriptor.LifecycleMethodUtils.lambda$findMethodsAndAssertStatic$0(LifecycleMethodUtils.java:81)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1085)
at org.junit.jupiter.engine.descriptor.LifecycleMethodUtils.findMethodsAndAssertStatic(LifecycleMethodUtils.java:81)
at org.junit.jupiter.engine.descriptor.LifecycleMethodUtils.findBeforeAllMethods(LifecycleMethodUtils.java:40)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.prepare(ClassBasedTestDescriptor.java:161)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.prepare(ClassBasedTestDescriptor.java:77)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$prepare$1(NodeTestTask.java:107)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.prepare(NodeTestTask.java:107)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:75)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:137)
at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:89)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:542)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:770)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:464)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
%TRACEE

---

Sample Project
https://github.com/spottedmahn/experiments-2020/tree/master/java-tests-issue

[jdneo]

Thank you for the detailed information. Root cause is that the regex in https://github.com/microsoft/vscode-java-test/blob/master/src/runners/junitRunner/JUnitRunnerResultAnalyzer.ts#L100: /\d+,(@AssumptionFailure: |@Ignore: )?(.*?)\((.*?)\)/ cannot match when the error occurs:

%ERROR 2,miked.javatestsissue.LibraryTest

[spottedmahn]

I'm guessing every @Test gets a unique ID but things like @BeforeTests don't?

And getTestId is assuming everything passed into it has a test id?

[spottedmahn]

It seems deeper than the regex... there won't be a result in the testResultManger as @BeforeAll is not a test. At least I don't see it in there through my debugging.

Even if the regex is fixed this code will fail (I believe):
const failedResult: ITestResult | undefined = testResultManager.getResultById(testId); source link

I was thinking I could patch the regex but it seems deeper than that. Let me know you're thoughts! Thanks!

[jdneo]

@spottedmahn Yes as you said. Currently only the methods will be saved into the testResultManger.

Provided a fix in #932.

[spottedmahn]

Provided a fix in #932

Awesome! 👏👏 Thanks 🤝

[jdneo]

@spottedmahn 😄

I'll spend a little more effort to check the output format of the test runner when the message is starting with %ERROR before the PR is merged. -- Make sure mo regression introduced here.

[spottedmahn]

Make sure mo regression introduced here.

I was surprised that I couldn't find any unit tests in this project. Maybe I missed them?