Junit5: Will JUnit 5 support anything like @Theories from JUnit 4?

[childnode]

hi,

yes, I asked my aunt google for quickly but not digged too deep in any discussion: so please regret this question ;)

since Theories had some nice ideas and Popper has absorbed it but then "accidently" died:
❓ anything new, cool aside Parameterized tests in JUnit5?

Anything that will adress Theories?

Anything that will help to write a test class with different dataproviders for different methods? i.e. > run test method1 with {A,B} run test method2 with {C,D}

Anything that will help to test combinations of different dataProvider data sets?!
i.e. > for two dataprovider {A,B} and {C,D} run each test method with {AC,AD,BC,BD}

/me is just bored from rewriting workarounds and pasting snippets for test class templates

~Marcel

documentation

[marcphilipp]

Parameterized tests are on our agenda for 5.0.0. We'll consider providing an option of how to combine multiple data providers. However, it's a bit early to say what they will look like exactly. On the one hand we would really want to enable using Streams to use large numbers of input parameters. On the other hand, this might hinder combining them to use all possible combinations.

[nipafx]

I like JUnit's core principle to "prefer extension points over features". Could this be an case for that?

If there were a good way to write extensions that create dynamic tests (e.g. with #371), then extensions could try different approaches to how parameterized tests could be created with maximum ease (likely for different scenarios).

This does not exclude JUnit5 from shipping with such a feature but it would be great if it could itself only use the extension point (much like @Disabled) and have to "privileged access" to internals.

[smoyer64]

This is yet another use case where the AfterDiscoveryCallback proposed in #354 could be used to "expand" the number of tests run (similar to JUnitParams and UpREST).

[childnode]

hi there,

thanks for the input. @marcphilipp I'm somewhat confused about the development cycle ... you have released a "milestone" but do not have a plan which "milestones" are coming; but: a forecasted release date of late 2016; no proposals for old features or how to migrate to Junit5 yet?
While I'm not saying it's wrong to +not+ make a "upfront, shitty big-pictured-waterfalled-detail-plan" :)

@smoyer64 I'll have a look at this, thx

~Marcel

[marcphilipp]

I like JUnit's core principle to "prefer extension points over features". Could this be an case for that?

Absolutely. We will make as much of the parameterization extensible as possible.

This is yet another use case where the AfterDiscoveryCallback proposed in #354 could be used to "expand" the number of tests run (similar to JUnitParams and UpREST).

Maybe. However, it's not always wise to compute the complete set of input parameters up front. Thus, to create TestDescriptors on the fly (e.g. using dynamic tests under the hood) might be a better solution.

I'm somewhat confused about the development cycle ... you have released a "milestone" but do not have a plan which "milestones" are coming; but: a forecasted release date of late 2016; no proposals for old features or how to migrate to Junit5 yet?

Understood. The current roadmap is deliberately not very detailed. We've just reached our goal to get M1 out in order for IDEs and build tools to start working towards native JUnit 5 integration. Basic features are in, with migration infos in the user guide. It's true that advanced features like Theories and Parameterized are missing, yet. As I've said this will be addressed in one of the next milestones. We'll let you know when we have a more detailed plan. :-)

[smoyer64]

@marcphilipp

I'm not proposing that extensions based on the AfterDiscoveryCallback replace @TestFactory - There are definitely cases where you want to generate the tests at run-time. I don't want to appear to be pushing #354 over a better solution - it's simply one way to accomplish the task.

More importantly, I think that there are some criteria for tests like these where dynamic tests currently fall short. I know the team is going to be working on addressing these issues and basically any mechanism that allows JUnitParams to be created as an extension will also work for me.

In addition to the Theories and Parameterized (assuming you're not talking about the built-in JUnit4 version) you've mentioned above, I think @Scenarios (or @Flow, @UseCase, etc) is the other important missing feature. The ability to order tests is critical - something we want to use all the time for Selenium2 UI acceptance tests.

I know a bunch of these issues are labelled "team discussion", so I'm going to stop talking (typing) now. Let me know if I can help in any way.

[gabrielefalace]

In Theories, being the _universal quantification_ implicit, was (at least in some cases) nice to write very sharp tests _declaratively_, avoiding many "for" or "if" statements. I think that is the most important aspect. Will this be in any way similar with dynamic tests?

[marcphilipp]

You can certainly use dynamic tests to emulate what you could do with Theories.

However, there's no lifecycle support, e.g. no _before/after each_. We plan to provide an extension point that will make it possible to formulate a theory as a method that will support the full lifecycle including all extensions for #14. We might even provide migration support for Theories.

[sbrannen]

Although there is currently no lifecycle support for dynamic tests, we plan to address that in #378.

[ddimtirov]

Consider also that it would be useful if a parameter provider has a way to access the test results - this can be used for implemention property-based testing, and in particular expanding and shrinking[*] strategies (rather than generating inputs at random).

[*] Can't find a good link, but this SO post goes into a bit more detail on expanding / shrinking.

[marcphilipp]

I think, once M4 is out, it would be good to try to implement a POC for s.th. like theories. I'm sure there are still a few pieces missing here and there. Having those written down would make them actionable.

[marcphilipp]

@ddimtirov Would you have time to try and do that based on @TestTemplate?

[pholser]

I know @jlink is working on a JUnit 5 engine for both example-based tests and theory-style tests.

As the author of junit-quickcheck, I'll be interested to see whether this will incorporate shrinking and other QuickCheck-y stuff.

[bechte]

I think the idea of creating a separate TestEngine for Theories is good, because Theories often do not follow our more general test discovery and execution path. Moreover, they need very special actions and initialization and maybe also a guaranteed order of test execution. All this would be better supported by an additional TestEngine, that can respect the special needs for those kind of tests. Still, using the extension model from Jupiter, you are also able to create extensions that would support Theories within the Jupiter Engine. To me its a matter of complexity and how we would like to tackle it. I think keeping certain things clearly separated will help us maintain and support in the long term.

[nipafx]

@bechte Would theories be an example for what I claim in #371 (comment), namely that an extension point that provides a whole hierarchy of dynamic tests would be useful? As I see it, theories could use such an extension point to create a test plan from a class any way they like while embedding the resulting tests into Jupiter's extension mechanism.

[marcphilipp]

Closing with same rationale as described in https://github.com/junit-team/junit5/pull/1422#issuecomment-389644868.