Assertj-core: check() method from AbstractSoftAssertions fails on first assertion

[tifhorn]

Hello,

I have an issue related to assertj-core 3.15.0.

When I'm using check() method from AbstractSoftAssertions, I get only first fail from assertion passed to this method.

For example, if I have following code:

String text = "assertJ-Core";
softly.check(() -> assertThat(text).startsWith("core").isEmpty());

only first assertion error will be caught (in this case "startsWith" assertion).

I don't know if it's intended behaviour, but in my opinion it clashes with soft assertions purpose and chainability of assertions provided by AssertJ.

[dstekanov]

Hello @tifhorn, this is expected behavior.

The purpose of check() is to allow you to use the standard or your custom assertions.

Each assertion should be described separately. You can go to SoftAssertions and read its documentation for more examples. Anyway, in your case you can use something like this:

SoftAssertions.assertSoftly(softly -> {
    softly.check(() -> Assertions.assertThat(text).startsWith("core"));
    softly.check(() -> Assertions.assertThat(text).isEmpty());
});

assertSoftly is used to avoid having to call assertAll manually

[tifhorn]

Hello @dstekanov, thanks for reply.

Too bad it works like that, because this way I can't use full potential of AssertJ with its fluent assertion chaining..

Guess I need to find another way - I thought that it would work with chaining like regular soft assertions do. The reason I need this is for taking screenshots of application at EVERY soft assertion failure - it's still impossible (or I just don't know "elegant" way to do it), due to package-privacy of SoftProxies and no option to override ErrorCollector.

[joel-costigliola]

@tifhorn what is you use case exactly? Being able to provide an ErrorCollector that you can inspect later on?

Once thing that might help: assertAll() throws a specific AssertionError either MultipleFailuresError or SoftAssertionError (depending if you have JUnit5 and opentest4j in your classpath), these classes have the list or errors (as a List for SoftAssertionError and List<Throwable> getFailures() for MultipleFailuresError).

[tifhorn]

Hello @joel-costigliola

My use case is simple - I need to find a way to take a screenshot of application under test just exactly when soft assertion fails (to be more specific - when errorCollector is adding an error to its list).

I cannot do it when SoftAssertionErrors are being thrown at the end of the test, because application state is different at the end, than at the exact moment of assertion failing.

I'm trying to do it since version 3.10 :) The only option I found is to modify SoftProxies/ErrorCollector and I thought that maybe "check()" method would be helpful in this case, but misunderstood its purpose.

[joel-costigliola]

@tifhorn if we introduce an ErrorCollector callback like: AfterErrorAddedCallback and provide users a way to register their own implementation then I guess that should solve your problem, right?

[tifhorn]

@joel-costigliola, sorry for delayed reply :)

Yes, I think that this solution will provide exactly what I need in this case.

[joel-costigliola]

I'll let you know when I have a draft to get some feedback

[tifhorn]

Hello.

I see that now in version 3.16.1 there are SoftAssertionsProviders, which extends AssertionErrorCollector, but the question is why? I cannot see where AssertionErrorCollector's methods are invoked, it always falls to standard ErrorCollector's method being invoked through SoftProxies's ByteBuddy-added code.

Am I missing something?

[joel-costigliola]

sorry this one fell under my radar, I have labelled this issue for the next release to be sure to look at it.

I'm not sure I understand your question, AssertionErrorCollector could have been included in SoftAssertionsProvider but we made the choice to put all the error collectors contract in a different interface. the method are used SoftAssertionsProvider or classes implementing it.

I probably haven't answered your question, I can try again if you clarify it for me a bit more.

[tifhorn]

Sorry, I didn't explain it clearly ;)

My question is, simplified - is there any place where AssertionErrorCollector's method "collectAssertionError(AssertionError error)" is invoked? I tried to override it, when extending SoftAssertionsProvider, but with no effect - it just doesn't get invoked in case of soft assertion error or I am missing some configuration.

[joel-costigliola]

it's called in check

  default void check(ThrowingRunnable assertion) {
    try {
      assertion.run();
    } catch (AssertionError error) {
      collectAssertionError(error);
    } catch (RuntimeException runtimeException) {
      throw runtimeException;
    } catch (Exception exception) {
      throw new RuntimeException(exception);
    }
  }

and assertAlso

  default void assertAlso(AssertionErrorCollector collector) {
    collector.assertionErrorsCollected().forEach(this::collectAssertionError);
  }

It is overridden in AbstractSoftAssertions too.

I'm guessing you are trying to solve your use case, is that right?

If you override it from AbstractSoftAssertions and call super.collectAssertionError and then capture whatever you want to capture, it might work.

[tifhorn]

Nope, it still doesn't get invoked.

public class SimpleSoftAssertions extends AbstractSoftAssertions implements SimpleSoftProvider { @Override public void collectAssertionError(AssertionError assertionError) { super.collectAssertionError(assertionError); System.out.println("Collected"); } }

SimpleSoftProvider is just an interface extending SoftAssertionsProvider:

public interface SimpleSoftProvider extends SoftAssertionsProvider { default SimpleObjectAssert assertThat(SimpleObject actual) { return proxy(SimpleObjectAssert.class, SimpleObject.class, actual); } }

The overriden method from SimpleSoftAssertions is never invoked. In pom I have only assertj-core 3.16.1 and junit-jupiter 5.7.0-M1.

[joel-costigliola]

Ah alright, can you provide a reproducible test case? that would make it easier to look into it.

ps: use triple backtick ``` to show code block ;-)

[tifhorn]

assertj-core-3.16-test.zip
In the archive there are all classes needed to check it (assertion in the test is just dummy code, so it will always fail).

[joel-costigliola]

Alright, the code is a bit of mess but I can clean it up and provide an easy to register a callback either explicitly:

  @Test
  void test(SimpleSoftAssertions softly) {
    softly.setAfterAssertionErrorCollected(error -> System.out.println("-------------- " + error));
    softly.assertThat(new SimpleObject()).isEqualTo(new SimpleObject());
  }

or by overriding method void onAssertionErrorCollected(AssertionError error) in your custom SoftAssertions class:

public class SimpleSoftAssertions extends AbstractSoftAssertions implements SimpleSoftProvider {

  @Override
  public void onAssertionErrorCollected(AssertionError error) {
    System.out.println(">>>>>>>>>>>>>>>> " + error);
  }
}

in that case the following test will print ">>>>>>>>>>>>>>>> " + error

  @Test
  void test(SimpleSoftAssertions softly) {
    softly.assertThat(new SimpleObject()).isEqualTo(new SimpleObject());
  }

This is still WIP but I think providing both approach should be enough.

Comments welcome!

[tifhorn]

Hello @joel-costigliola

I'm really happy how it turned out, beacuse this will allow to do exactly what I want and many more, I suppose :)

Big thanks!

[kriegfrj]

I've just stumbled upon the fix for this while working on #1970. I'm glad it's here because I think it will help me with my implementation. But I agree with this comment above by @joel-costigliola:

Alright, the code is a bit of mess

One question that I have: you left a comment in AssertionErrorCollector:

   * <b>Warning:</b> this is not the method used internally by AssertJ to collect all of them, overriding it to react to each
   * collected assertion error will not work.

Is there a reason we have to leave it this way? Should we change the proxy code so that all calls are channeled through collectError()? I think it would tidy up that piece of code significantly and makes for cleaner separation between the role of the proxy generation and the error collection.