Vscode-java-test: Use socket to deliver the test results

[deduper]

Windows 7 Pro 64 Bit
Java Open JDK 11.0.2+9 64 Bit
Java Test Runner 0.18.0
Java Extension Pack 0.7.1
Language Support For Java 0.46.0

Visual Studio Code
Version: 1.35.0 (system setup)
Commit: 553cfb2c2205db5f15f3ee8395bbd5cf066d357d
Date: 2019-06-04T01:18:19.664Z
Electron: 3.1.8
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT ia32 6.1.7601

---

Out of twenty-five or so test classes, some test methods in about five or six of them are always skipped. Some test methods in the same test class file are not skipped.

In DependencySummarizerTest.java, there are four test methods. Clicking the class level Run Tests link results in three skipped tests, only one check mark in the status bar and one successful test in the test result panel. Yet, two of the three skipped test methods have the check mark in their Code Lens heading. Even though they appear in neither the status bar, nor in the result panel.

The skipping happens when I either launch the tests by clicking on the Code Lens Run Test link or when I launch from the Test Explorer.

Building/testing using Gradle works as expected.

I cannot think of anything that is different between the skipped and not-skipped tests. There is some multithreading in some the classes exercised by some of the skipped test.

Some tests in which the issue occurs:

• AbiTest.java
• JarSummarizingAbiInspectorTest.java
• DependencySummarizerTest.java

What I've tried

1. Developer: Reload Window
2. Java: Force Java Compilation (Full)
3. Java: Update Project Configuration
4. Java: Clean the Java language server workspace
5. Delete the bin folder
6. Close/Reopen VS Code.

[jdneo]

Hi @deduper,

I tried to import the project into the VS Code, but there are some compile errors in the project

Would you mind to take a look?

[deduper]

Hi,

Thanks for looking into this. I will see why that's happening and get back to you.

Thanks again.

[deduper]

Hi again,

I forgot that that version of the bitcoinj SNAPSHOT is not their published one. I build that version locally and use it for testing. I've edited build.gradle to replace the unpublished version (0.16-SNAPSHOT) with the published one (0.15-SNAPSHOT). You should be OK now. Please let me know otherwise?

You gotta make sure you're in the refactor.summary.accuracy branch though. Not master.

By the way, I noticed I get the same skipped test problem with the test runner when working in Linux and JDK 12:

---

Version: 1.35.1
Commit: c7d83e57cd18f18026a8162d042843bda1bcf21f
Date: 2019-06-12T14:27:31.086Z
Electron: 3.1.8
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Linux x64 4.15.0-51-generic snap

The version of the test runner and the Java Language Pack are the same as those in my first comment.

Thanks again for looking into this.

[deduper]

I forgot to mention. It's important to that code's functionality to have a Gradle distribution installed plus a GRADLE_USER_HOME environment variable set on your machine.

[jdneo]

@deduper Here is some analysis for this issue

Root Cause

The Test Runner will output test results into system.out and the client side will receive the output from system.out and parse the result.

In the sample project, some methods like AbiIo::summarize() will close the steam so that the client side could not receive output anymore.

Workaround

Honestly, I could not find a good way to workaround this issue. If we do not use try-with-resource in the method (not close the system.out), it seems the tests will work fine.

Follow-up

Instead of communicating using system.out, we should consider using a socket to deliver the test results.

[deduper]

That is exceptionally well done analysis, @jdneo! Thank you very much. It never would have occurred to me that System.out would be at the root of the problem.

It was determined during earlier development of AbiIo::summarize() that closing that PrintStream solved another issue that was encountered at that time. I will look into whether the feature can be implemented without the need to use the try-with-resource.

I have another question. But about the Test Explorer. If I click the Play button at the very top-right of the Test Explorer (located next to the debug button) when the root project node is selected (abi.cli in my case), shouldn't the runner run every test in the project?

Similarly for package nodes in the explorer? If I select a package and right-click then select Run Test, shouldn't all of the tests in that package be executed? At the moment, doing either of the two results in the same skipped tests issue. I'm assuming the root cause is the same. The System.out thing? Right?

[jdneo]

@deduper Yes, it should be caused by the system.out thing too.

[jdneo]

Hi @deduper,

I've update the implementation to use socket instead of output to deliver the test result. Feel free to have a try with it if you want. (It's not released yet, so you might need to build it by yourself, more details: https://github.com/microsoft/vscode-java-test/blob/master/CONTRIBUTING.md#setup)

[deduper]

Hi jdneo. Sorry for the delayed reply.

I've updated my Test Runner and it works marvelously! Thanks for that!