Behat: [Feature] Test all scenario outcomes even if one of them failed

[brunowowk]

When a step fails in a Scenario, the followings steps are skipped. This makes sense for context (Given) and event (When) steps because if any of those steps fail the Scenario would not reach the desired state and any assertions over its outcomes would be wrong.

However, in the case of failure on an outcome (Then) step, the following outcome steps could still be tested, because the Scenario is in the desired state, but perhaps only some of it's desired outcomes were not achieved.

Illustrative Scenario

Scenario: Creating a user on the system
    Given I'm logged in as admin
    When I run the command to create a user
    Then a user should be created
      And a folder should be created for that user
      And the log should be updated with an entry about the user creation
      And the console output should show the created user details
    When I log out
      And I enter the new user's credentials
    Then I should be logged in as the new user
      And I should have access to the new user's folder

In a case where the user is created but his folder is not:

Current behavior

Step | Result
------------ | -------------
Given I'm logged in as admin | :white_check_mark: passed
When I run the command to create a user | :white_check_mark: passed
Then a user should be created | :white_check_mark: passed
And a folder should be created for that user | :x: failed
And the log should be updated with an entry about the user creation | :heavy_minus_sign: skipped
And the console output should show the created user details | :heavy_minus_sign: skipped
When I log out | :heavy_minus_sign: skipped
And I enter the new user's credentials | :heavy_minus_sign: skipped
Then I should be logged in as the new user | :heavy_minus_sign: skipped
And I should have access to the new user's folder | :heavy_minus_sign: skipped

Proposed behavior

Step | Result
------------ | -------------
Given I'm logged in as admin | :white_check_mark: passed
When I run the command to create a user | :white_check_mark: passed
Then a user should be created | :white_check_mark: passed
And a folder should be created for that user | :x: failed
And the log should be updated with an entry about the user creation | :white_check_mark: passed
And the console output should show the created user details | :white_check_mark: passed
When I log out | :heavy_minus_sign: skipped
And I enter the new user's credentials | :heavy_minus_sign: skipped
Then I should be logged in as the new user | :heavy_minus_sign: skipped
And I should have access to the new user's folder | :heavy_minus_sign: skipped

Benefits

In the proposed behavior, we know that the user creation system is correctly:

• creating the user
• writing to the log file
• outputting the results on the console

On the other hand, in the current behavior, we just know the user folder wasn't created, but there may be more issues in that system.

The workaround for that would be splitting the scenario and testing one outcome on each, but that would also make the test suite much more verbose and slower.

BC break

If there's any concern with BC breaking (maybe some formatters and extensions assume no steps will run after a failed one) this feature could be enabled by a config entry, and disabled by default

[brunowowk]

For anyone interested, I made an extension that does just that:
Behat Overlook Extension

[everzet]

This should stay in third-party extension :) I don't want to diverge Behat behaviour too much away from other Cucumber implementations.