Vscode-java-test: Single File Test Support

[praenubilus]

The plugin currently requires a maven/gradle project to run the unit tests which does not make sense.

1. The debugger support single non project file run/debug.
2. Individual file test works smoothly for python, so I hope the java test runner can also run like that, even all output is just in console.

Is there a timeline for when single/individual file unit test running will be supported?

[jdneo]

So the workflow might be:

1. The user creates a single java file
2. The user adds dependencies (e.g. JUnit) into the single-file project
3. After the project is set up correctly, the user can write some test cases and run the tests

@praenubilus Is this the workflow you want to have to support single file test?

[praenubilus]

So the workflow might be:

1. The user creates a single java file
2. The user adds dependencies (e.g. JUnit) into the single-file project
3. After the project is set up correctly, the user can write some test cases and run the tests

@praenubilus Is this the workflow you want to have to support single file test?

That looks good. But is there a way to put the dependency into the settings.json? In python plugin, you can configure which test framework you want to use for the workspace, and their path(pytest, unittest etc). I guess the best way here is add a test framework configuration in settings, let user to choose (JUnit, TestNG or other test framework), configure the dependencies path. The setting works either for users or workspace.

[jdneo]

@praenubilus The way to manage the dependency in single-file project is worth discussing. Actually, this is a common problem that needs to be solved. I created an issue here: https://github.com/Microsoft/vscode-java-dependency/issues/76. Hopefully, this can be achieved after the dependency extension supports this.

[jdneo]

Now the Java Language Server supports to manage the dependencies into the lib folder, this does not need the user to have building tools like Maven or Gradle in their project. See: https://github.com/redhat-developer/vscode-java/issues/501#issuecomment-461234115

I will start to investigate how to support this kind of project in the Test Runner and unblock those kinds of users.

[praenubilus]

I hope you can let the plugin configure a universal default lib/jar path, that means if the user does not explicitly configure a custom lib, jars, classes under that path will always being included by default. Then we can put some default framework jar files under that lib.
Another idea is, let the user to specify a testing library path globally/workspace-wise, it can be junit/test-ng and extra test libraries if necesary(e.g. assertj)

[jdneo]

Thank you @praenubilus. About the default lib path, this could be a feature request for https://github.com/eclipse/eclipse.jdt.ls, which is also known as the Java Language Server.

And for the testing lib path, I'll consider it. Thank you.

[jdneo]

Now the test runner can run the test even if the project does not have Maven or Gradle. See:

The limitation is that there is no Code Lens and cannot explore in the Test Explorer. This issue should be tracked in #470

[praenubilus]

@jdneo thanks for the update. But I'm just a little bit confusing, how does the vscode now where is my junit.jar file?

[jdneo]

Hey @praenubilus,

For the projects that do not have Maven or Gradle, the Java Language Server will search the dependency in the lib folder.

[rkulow]

Now the test runner can run the test even if the project does not have Maven or Gradle. See:

The limitation is that there is no Code Lens and cannot explore in the Test Explorer. This issue should be tracked in #470

I'm trying to recreate your test environment @jdneo to run tests from a single file (without the project environment) - unfortunately this causes an error:

@@<TestRunner-{"name":"error","attributes":{"message":"No test found to run for suite MyTest#test. Details: MyTest.","trace":"java.lang.ClassNotFoundException: MyTest \tat java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583) \tat java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) \tat java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) \tat java.base/java.lang.Class.forName0(Native Method) \tat java.base/java.lang.Class.forName(Class.java:398) \tat com.microsoft.java.test.runner.junit4.JUnit4RunnerUtil.getRequestForOneMethod(JUnit4RunnerUtil.java:55) \tat com.microsoft.java.test.runner.junit4.JUnit4RunnerUtil.createTestReferences(JUnit4RunnerUtil.java:44) \tat com.microsoft.java.test.runner.junit4.CustomizedJUnit4CoreRunner.run(CustomizedJUnit4CoreRunner.java:25) \tat com.microsoft.java.test.runner.junit4.JUnit4Launcher.execute(JUnit4Launcher.java:30) \tat com.microsoft.java.test.runner.Launcher.main(Launcher.java:53) "}}-TestRunner>

The opened folder contains the MyTest.java file with exact the same content as yours and I also added the junit-4.12.jar file to lib folder. I also verified using test runners version 0.16.0.
Is there maybe something missing in my small test?

[jdneo]

@KulowRainer Not sure if you have the same problem like this one: https://github.com/redhat-developer/vscode-java/issues/882

Is the problem still occur if you reload the VS Code?

[rkulow]

Good mention @jdneo, but the problem ist still occurring after reloading VS Code.
The java code checker returns no errors.

I also cloud reproduce the same error on a fresh ubuntu 18.04 installation with vs code, meanwhile the screenshot above was taken in Windows 10.

[jdneo]

@KulowRainer Could you please try Java: Clean the Java Language Server workspace > Restart and Delete

[rkulow]

@jdneo Now I can confirm Java: Clean the Java Language Server workspace > Restart and Delete solved this problem.

[praenubilus]

@jdneo Any plan for the testing lib path support (other than the lib path from language server) in the near future?

[jdneo]

@praenubilus No such a plan so far at the Test Runner side. I think it should be better to let the Java Language Server support customized classpath for the invisible project.

[praenubilus]

Is there a Document for how Test Discover(JUnit5/4) works for invisible project? In my observation, it seems only working for Test classes at top level (workspace folder), but if I put test Class down to some levels, the test runner cannot discover the unit test classes. Can test Classes in nested folders being explored? Can we specify which folder will be used for test discover? Just like the python plugin, you can specify which folder(s) will be used for test discover.

Also, for invisible project lib (I understand it is from Language Server), it looks like symblink will not work, is it correct?

[jdneo]

@praenubilus

but if I put test Class down to some levels, the test runner cannot discover the unit test classes.

Would you mind to share a sample project with us? I would like to have a try to see what's happening.

for invisible project lib, it looks like symlink will not work

From my observation, no it's not working for a symlink. But I suggest raising the question at the Language Server repo to get the correct answer

Thanks

[praenubilus]

@jdneo
This is weird. I have two machines: One Ubuntu workstation w/ java11 everything seems working. Another Mac OS laptop, I cannot remember whether the invisible project used to work or not.

But now any invisible projects, the test runner cannot discover the tests. Even after reinstallation of vscode and all plugins. There is no errors raised for unresolved package, so I assume the jars under lib has been included. But no luck with Test Runner. I have tried another gradle project, it works. Not sure what is wrong, because I do not have any more detailed trace information(does the test runner having such a log? The only prompt I have is:"No test items found."). Here is a sample project I used(works on Ubuntu, but not working on Mac)
javaexp.zip

[jdneo]

Hi @praenubilus,

I took a look at your sample project and find the code is written as:

@Test
void myTest() {
    assertEquals(1, 1);
}

The method myTest() is not decorated with the public modifier, so just add the public to it should solve your problem I guess.

[praenubilus]

The method myTest() is not decorated with the public modifier, so just add the public to it should solve your problem I guess.

@jdneo
Thanks for the quick reply. I tried to update the modifier to public, but it is still not working(I have cleaned language server workspace and restarted). And I tested the exact same code on _Ubuntu_ machine, it works.

[jdneo]

Hmm... so wired. I tried on my Mac and it works fine 😞

Could you please try to uninstall the following plugins:

• redhat.java
• vscjava.vscode-java-debug
• vscjava.vscode-java-test

Then remove the extension folders under $home/.vscode/extensions/<id-version>.

Then reinstall them again to see if this could help.

[praenubilus]

@jdneo
Followed your instructions. I did following steps

• delete plugins(ms java extension pack), all plugins deleted
• delete subfolders under .vscode/exteionsions
• install them back
  Boom, test discovered. Code-Lens appeared.

HOWEVER, delete the public modifier, everything go away. Change back to public, everything failed. Even in other invisible projects. Then I

• delete the plugins
• install them back(before discover happens, the modifier is public)
• everything works again across all invisible projects
  That is really crazy, I hope I can record the screen, but the video size is too large to upload to github

I can repro this behavior every time through change public modifier

[jdneo]

Is the java.autobuild.enabled set to true?

[praenubilus]

By default, it is checked. After I turned it off, the current working space works every time I delete public, add back public.
But for another irrelevant workspace(project), that one has previously being "polluted" w/ the operations I mentioned previously, the test discover and codelens never come back, unless I delete the plugins and install them back.
And this only happens on my Mac.

[jdneo]

Have no ideas so far.

If anyone meets the same problem as @praenubilus described, please feel free to let us know. We need to repro it for further investigation.

[praenubilus]

@jdneo
I can repro the problem sometime even on Linux now. For invisible project, is there any hidden folder, test runner(especially for test discover) will have cache on local disc? For example, somewhere under ~/.vscode folder?

[jdneo]

@praenubilus there is no cache for the test discovery. Can you just rename your workspace folder to have a try? rename the workspace folder will force the language server to create new workspace storage.

[praenubilus]

@jdneo rename workspace is not working(as I tried). Also to clarify for the linux version, a restart of vscode brings back the codelens and discovered all tests. So the problem only haunting me on Mac. Here I attached a sample project w/ verbose Java Language Server Trace Log(before and after rename workspace), hope that will be helpful to locate the problem.
javatest.zip

Some other error message from language support for java

[Error - 12:32:27 AM] Request codeLens/resolve failed.
  Message: Internal error.
  Code: -32603 
java.util.concurrent.CompletionException: java.nio.file.FileSystemAlreadyExistsException
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
    at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:645)
    at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.nio.file.FileSystemAlreadyExistsException
    at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.newFileSystem(ZipFileSystemProvider.java:99)
    at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:344)
    at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:293)
    at org.eclipse.jdt.internal.core.builder.ClasspathJrtWithReleaseOption.initialize(ClasspathJrtWithReleaseOption.java:132)
    at org.eclipse.jdt.internal.core.builder.ClasspathJrtWithReleaseOption.<init>(ClasspathJrtWithReleaseOption.java:73)
    at org.eclipse.jdt.internal.core.builder.ClasspathLocation.forJrtSystem(ClasspathLocation.java:147)
    at org.eclipse.jdt.internal.core.search.matching.JavaSearchNameEnvironment.mapToClassPathLocation(JavaSearchNameEnvironment.java:164)
    at org.eclipse.jdt.internal.core.search.matching.JavaSearchNameEnvironment.computeClasspathLocations(JavaSearchNameEnvironment.java:133)
    at org.eclipse.jdt.internal.core.search.matching.JavaSearchNameEnvironment.<init>(JavaSearchNameEnvironment.java:77)
    at org.eclipse.jdt.internal.core.search.matching.IndexBasedJavaSearchEnvironment.create(IndexBasedJavaSearchEnvironment.java:334)
    at org.eclipse.jdt.internal.core.search.matching.MatchLocator.initialize(MatchLocator.java:1222)
    at org.eclipse.jdt.internal.core.search.matching.SuperTypeNamesCollector.collect(SuperTypeNamesCollector.java:167)
    at org.eclipse.jdt.internal.core.search.matching.MethodLocator.initializePolymorphicSearch(MethodLocator.java:128)
    at org.eclipse.jdt.internal.core.search.matching.MatchLocator.locateMatches(MatchLocator.java:1433)
    at org.eclipse.jdt.internal.core.search.JavaSearchParticipant.locateMatches(JavaSearchParticipant.java:112)
    at org.eclipse.jdt.internal.core.search.BasicSearchEngine.findMatches(BasicSearchEngine.java:250)
    at org.eclipse.jdt.internal.core.search.BasicSearchEngine.search(BasicSearchEngine.java:595)
    at org.eclipse.jdt.core.search.SearchEngine.search(SearchEngine.java:670)
    at org.eclipse.jdt.ls.core.internal.handlers.CodeLensHandler.findReferences(CodeLensHandler.java:145)
    at org.eclipse.jdt.ls.core.internal.handlers.CodeLensHandler.resolve(CodeLensHandler.java:98)
    at org.eclipse.jdt.ls.core.internal.handlers.JDTLanguageServer.lambda$15(JDTLanguageServer.java:606)
    at org.eclipse.jdt.ls.core.internal.handlers.JDTLanguageServer.lambda$39(JDTLanguageServer.java:875)
    at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)
    ... 6 more