Junit5: Cannot select @TestTemplate or @TestFactory invocations by Unique ID

[noopur2507]

Overview

With reference to bug request 520923 in Eclipse, I am trying to rerun a single invocation of a parameterized test (and similarly, a repeated and a dynamic test) using its unique ID like this:

LauncherDiscoveryRequest request = LauncherDiscoveryRequestBuilder.request().selectors(DiscoverySelectors.selectUniqueId(uniqueId)).build();

A new Launcher instance is created to discover the above request.

This method works for a normal test with ID:

[engine:junit-jupiter]/[class:test.FirstTest]/[method:myFirstTest(org.junit.jupiter.api.TestInfo)]

But it gives this error for a dynamic test:

org.junit.jupiter.engine.discovery.JavaElementsResolver resolveUniqueId
WARNING: Unique ID '[engine:junit-jupiter]/[class:test.dynamic.DynamicTestsDemo]/[test-factory:dynamicTestsFromIntStream()]/[dynamic-test:#2]' could not be resolved

or a parameterized test:

org.junit.jupiter.engine.discovery.JavaElementsResolver resolveUniqueId
WARNING: Unique ID '[engine:junit-jupiter]/[class:jb.Junit5SimpleParamTest]/[test-template:dummy(java.lang.String)]/[test-template-invocation:#1]' could not be resolved

In JUnit 4, it was done using org.junit.runner.Request.filterWith(Filter filter).

Question

What is the recommended way to do this in JUnit 5?

Related Issues

• #1026
• #1031

Analysis

Given a _Unique ID_ such as [engine:junit-jupiter]/[class:test.dynamic.DynamicTestsDemo]/[test-factory:dynamicTestsFromIntStream()]/[dynamic-test:#2],
JavaElementsResolver.resolveUniqueId(TestDescriptor, List<Segment>) returns true if all segments of the supplied Unique ID could be resolved.

So, in such a case, the test class and test factory method do in fact get resolved, but the dynamic test itself does not get resolved.

That means that the test class and test factory method will be executed in the test plan even though there is a warning (which actually only applies to the last segment of the Unique ID).

Proposals

1. Selection of any invocation of a test template or test factory results in selection of the test template or test factory as a _container_ and consequently the _selection_ of all invocations.
   
   
   
   • This appears to already be the case: see _Analysis_ for details.
   
   
2. Selection of a specific invocation of a test template or test factory results in selection of only that specific invocation of the test template or test factory.
   
   
   
   • This could potentially be achieved via a special type of _filter_ that is applied to the stream of dynamic tests or test template invocations.
   
   

Deliverables

• [x] Ensure that an invocation of a @TestTemplate method can be selected by _Unique ID_.
  
  
  
  • This applies to @RepeatedTest and @ParameterizedTest as well as any _test template_ method.
  
  
• [x] Ensure that an invocation of a dynamic test supplied by a @TestFactory method can be selected by _Unique ID_.

[sbrannen]

There is no way to do that.

It's a bug.

I actually assumed that this bug existed but forgot to raise an issue, so.... thank you for raising the issue!!!

[sbrannen]

In general, there is no way to select a single invocation of a @TestTemplate or @TestFactory method by _unique ID_.

[sbrannen]

I have reworded the title of this issue accordingly.

[sbrannen]

Added _Deliverables_ section

[sbrannen]

@junit-team/junit-lambda

I added the _team discussion_ label to discuss "what" to select in such cases.

See my _Proposal_ in _Deliverables_.

[sbrannen]

Moved _Proposal_ to stand-alone section and added a deliverable for dynamic tests.

[sbrannen]

Update:

Now that I've read the original bug description in the Eclipse issue tracker, I realize that the user wants to be able to select a single dynamic test or invocation of a test template method.

@junit-team/junit-lambda, so we'll have to discuss the topic in detail to come up with a decision on what to do pre/post 5.0 GA.

[sbrannen]

Added 2nd, alternate proposal

[sbrannen]

Update

• added _Analysis_ section
• added new task in _Deliverables_ to improve the warning log message

[sbrannen]

Added _Implementation Notes_

[sbrannen]

Deliverable 1: _in progress_

[sbrannen]

FYI: work on "Proposal 1" has been moved to #1031.

[sbrannen]

Furthermore, this issue is now slated for the 5.1 backlog.

[akozlova]

In IDEA, we have a feature to rerun failed tests or run selected test (also for parameterized test). Unfortunately it looks like, that it's impossible to support them for junit 5 in case of dynamic/template tests: all tests from the container are executed what is actually not, what user expects as the number of dynamic tests could be huge.

[sormuras]

Back to 5.1 Backlog after talking to Anna at #devoxx

[k1w1m8]

I'm currently using the old school JUnit 3 API with TestSuite and TestCase to write a wrapper around non Java tests and have them runnable inside JUnit/Maven/Jenkins/IDEA. Since the non java tests don't correspond to an actual Java .class method, you cannot click and run them inside IDEA. :-(

I looked into using JUnit 5 for this purpose. The Dynamic Tests api lets me create dynamic tests quite similarly to the old JUnit 3 TestSuite/TestCase structure, but I still cannot run tests which don't correspond to a method.

I'm hoping this ticket when implemented will allow me to finally re-run a single Dynamic Test.

[jlink]

A simple implementation that would allow IDEs like IntelliJ to use it (see https://youtrack.jetbrains.com/issue/IDEA-169198?replyTo=27-2628598) is to use the parent descriptor in cases where the child descriptor cannot be invoked.

As it is now, Jetbrains is not willing to do it "the right way" because Jupiter will throw up an exception :-/

[marcphilipp]

Jupiter does not throw an exception when resolving a UniqueIdSelector for a dynamic test or a test template invocation. It currently resolves it into the test factory or test template method, respectively.

@jlink Am I missing something?

[jlink]

Don't think so. I guess it was a misunderstanding between me and Anna that led me to believe otherwise. So the only thing missing is "the real implementation".

[marcphilipp]

I've started working on it tonight. 🙂

@akozlova For it to work in IDEA, I'm afraid you'll have to implement support for launching tests using UniqueIdSelectors.

[akozlova]

Already done in master. We will start EAP soon. Thanks!

[marcphilipp]

@noopur2507 This should be fixed in the next available milestone. Would be great if you could give it a try.

[marcphilipp]

Works in Eclipse! 👍

[marcphilipp]

@akozlova I've checked the latest EAP and it's not working there, yet. Is your fix already included?

[akozlova]

We don't include features in EAPs of released versions. I was pointing to upcoming EAP of 2018.1. Should be out next week. Thanks!

[sbrannen]

Works in Eclipse! 👍

That's awesome to see that working in Eclipse! 🎉

Thanks for making this happen, @marcphilipp!

And... looking forward to seeing it in action in IDEA as well!

Yes... thrice "!".

[sbrannen]

Note: this issue has been resolved but remains open until the feature has been tested in IntelliJ IDEA EAP 2018.1.

[marcphilipp]

With the just release EAP 2018.1, it now also works in IDEA:

Thanks for the great work, @akozlova! 🎉

[sbrannen]

Thanks for the great work, @akozlova! 🎉

I second that!

Kudos to both @noopur2507 _and_ @akozlova, the Ladies of the IDEs! 😄

[k1w1m8]

Awesome! Can't wait to try this! Thanks all!