Junit5: Introduce @NullSource, @EmptySource, & @NullAndEmptySource for parameterized tests

[ttddyy]

Overview

Similar to #1635, I stumbled upon a situation where I need to validate null in parameterized test.

@ParameterizedTest
@ValueSource(strings = { "wrong-input", "", " " })
void wrongInput(String input) {
  ...
}

It is certainly convenient to have a way of testing null use case without creating a custom provider or another test method just for null.

Original Proposal

How about adding a flag attribute to @ValueSource annotation such as testNull, checkNull, etc.

@ParameterizedTest
@ValueSource(strings = { "wrong-input", "", " " }, testNull=true)
void wrongInput(String input) {
  ...
}

Related Issues

• Supersedes #1635

Deliverables

• [x] Introduce @NullSource that provides null for any reference type and throws an exception for any primitive target type.
• [x] Introduce @EmptySource that provides an _empty_ object for supported types and throws an exception for unsupported types.
  
  
  
  • Initial supported types: String, List, Set, Map, and arrays of any type.
  
  
• [x] Introduce @NullAndEmptySource _composed annotation_ that combines @NullSource and @EmptySource.
• [x] Document in User Guide.
• [x] Document in Release Notes.

[sbrannen]

We typically would not want to introduce an annotation attribute that is only applicable part of the time. For example, null does not make sense for a primitive target type.

On the other hand, you can already pass null arguments via @MethodSource or @CsvSource as follows.

@ParameterizedTest
@CsvSource({ "wrong-input,", ",", "' '," })
void test(String s) {
    System.out.println(s == null ? null : "'" + s + "'");
}

That's perhaps a bit of a _hack_, but... it works. 😇

[sbrannen]

Oh yeah, the output for the above is:

'wrong-input'
null
' '

[marcphilipp]

You could define a @NullSource and use it in addition to @ValueSource.

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@ArgumentsSource(NullArgumentsProvider.class)
public @interface NullSource {}

class NullArgumentsProvider implements ArgumentsProvider {
    @Override
    public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
        return Stream.of(Arguments.of(new Object[context.getRequiredTestMethod().getParameterCount()]));
    }
}

[sbrannen]

@junit-team/junit-lambda, Tentatively slated for 5.4 M1 for the purpose of _team discussion_ in conjunction with #1635.

[sbrannen]

Good point, @marcphilipp.

In light of that proposal and the discussions in #1635, if we decide to include any such built-in sources, I think it might be best to introduce the following.

• @NullSource: provides null for any reference type and throws an exception for any primitive target type.
• @EmptySource: provides an _empty_ object for supported types and throws an exception for any primitive target type.
  
  
  
  • Initial supported types: String, List, Set, Map, and arrays of any type.
  
  

For the _blank_ String source proposed in #1635, I think we should just let people continue to use @ValueSource( strings = { ... } ) for that purpose.

Thoughts?

[sbrannen]

That would lead to _solutions_ similar to the following for the proposals in #1635.

class Tests {

    @ParameterizedTest
    @NullSource
    @EmptySource
    void nullAndEmptyStrings(String str) {
        assertTrue(str == null || str.isEmpty());
    }

    @ParameterizedTest
    @NullSource
    @EmptySource
    void nullAndEmptyLists(List<?> list) {
        assertTrue(list == null || list.isEmpty());
    }
}

[ttddyy]

Sounds great to have them in junit-jupiter-params

[sbrannen]

And for _blank_ strings, one can simply use @ValueSource in contrast to the proposal in #1635.

The following would supply null, _empty_, and _blank_ strings as arguments.

class Tests {

    @ParameterizedTest
    @NullSource
    @EmptySource
    @ValueSource(strings = {" ", "   ", "\t", "\n"})
    void nullEmptyAndBlankStrings(String str) {
        assertTrue(str == null || str.trim().isEmpty());
    }
}

If somebody wants to reuse all three of those sources, it's straightforward to introduce a custom _composed annotation_ that is meta-annotated with those three source annotations.

Of course, JUnit Jupiter could also provide a pre-built @NullAndEmptySource _composed annotation_ out of the box if we want to.

[sbrannen]

You could define a @NullSource and use it in addition to @ValueSource.

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@ArgumentsSource(NullArgumentsProvider.class)
public @interface NullSource {}

class NullArgumentsProvider implements ArgumentsProvider {
  @Override
  public Stream<? extends Arguments> provideArguments(ExtensionContext context) {
      return Stream.of(Arguments.of(new Object[context.getRequiredTestMethod().getParameterCount()]));
  }
}

I just noticed that your proposal would generate nulls for all arguments, which would be risky since that would prevent arguments being supplied by other ParameterResolvers.

[sbrannen]

In light of my previous comment, I'm thinking that @NullSource and @EmptySource should only _provide_ a single argument (namely, the first argument).

[jmaniquet]

Hi

If i understand right, those new annotations can be combined with another source annotation, to supplement it with the null or empty value?

That would solve the problem i tried to solve in #1635, so i like it.

As for a @NullAndEmptySource, i think it is one of the most common use case (if not the most common one). So it would be valuable to have it baseline in the framework.

[sbrannen]

If i understand right, those new annotations can be combined with another source annotation, to supplement it with the null or empty value?

Yes, that is correct.

All sources are combined into a single stream:

https://github.com/junit-team/junit5/blob/45a8c8f5a695e2f8805b343680f7f208a8f9d4b4/junit-jupiter-params/src/main/java/org/junit/jupiter/params/ParameterizedTestExtension.java#L68-L93

That would solve the problem i tried to solve in #1635, so i like it.

Great. In light of that, I'll close #1635.

As for a @NullAndEmptySource, i think it is one of the most common use case (if not the most common one). So it would be valuable to have it baseline in the framework.

OK. We will take that into consideration.

Thanks for the feedback!

[sbrannen]

Update: I have renamed this issue, reworked the description, and added _Deliverables_.

[sbrannen]

In response to this blog which I discovered via this tweet, there is no need to implement a custom NullableConverter for use with @CsvSource.

Given the following DateRange:

import java.time.LocalDate;

public class DateRange {

    private final LocalDate startDate;
    private final LocalDate endDate;

    public DateRange(LocalDate startDate, LocalDate endDate) {
        if (startDate == null && endDate == null) {
            throw new IllegalArgumentException();
        }
        if (startDate != null && endDate != null && startDate.isAfter(endDate)) {
            throw new IllegalArgumentException();
        }
        this.startDate = startDate;
        this.endDate = endDate;
    }

    @Override
    public String toString() {
        return "DateRange [startDate=" + startDate + ", endDate=" + endDate + "]";
    }

}

... the solution to the problem mentioned in that blog is to use _empty_ values as follows:

@ParameterizedTest
@CsvSource({ "2017-06-01, 2018-10-15", ", 2018-10-15", "2017-06-01," })
void shouldCreateValidDateRange(LocalDate startDate, LocalDate endDate) {
    System.out.println(new DateRange(startDate, endDate));
}

@ParameterizedTest
@CsvSource({ "2018-10-15, 2017-06-01", "," })
void shouldNotCreateInvalidDateRange(LocalDate startDate, LocalDate endDate) {
    assertThrows(IllegalArgumentException.class, () -> new DateRange(startDate, endDate));
}

[marcphilipp]

Team Decision:

Let's add all three annotations.

[sbrannen]

_in progress_

[sbrannen]

FYI: work on this issue can be viewed here:

https://github.com/junit-team/junit5/compare/issues/1637-null-and-empty-sources

[sbrannen]

The work for this issue has been merged into master in c6f75a7537d57f3333d7c06a77cbdbf88547e90d.