Cucumber-js: The exception object of an error that caused a step to fail is no longer accessible in custom formatters as of version 7

[oleduc]

In version 4 of cucumber it was possible to access the exception which caused a step to fail in a custom formatter.

{
  "sourceLocation": {
    "uri": "features\\tests\\ecom\\family\\get-family-from-member.feature",
    "line": 7
  },
  "actionLocation": {
    "uri": "features\\tests\\ecom\\family\\steps_definitions\\get-family-from-member.js",
    "line": 8
  },
  "result": {
    "duration": 10065,
    "exception": {
        ... Exception data ...
    }
  }
}

In version 7, out of the box there is only a "message" property which contain a serialized stacktrace but that's it.

Here is an example test step object:

{
  "attachments": [],
  "keyword": "Given ",
  "result": {
    "status": "FAILED",
    "message": "Error: Request failed with status code 500\n    at createError (C:\\dev\\slp-api-consumers\\node_modules\\axios\\lib\\core\\createError.js:16:15)\n    at settle (C:\\dev\\slp-api-consumers\\node_modules\\axios\\lib\\core\\settle.js:17:12)\n    at C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:87:29\n    at new Promise (<anonymous>)\n    at $If_1 (C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:86:30)\n    at C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:97:18\n    at new Promise (<anonymous>)\n    at responseInterceptor (C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\interceptors\\response.js:21:10)\n    at C:\\dev\\slp-api-consumers\\node_modules\\axios-cookiejar-support\\lib\\index.js:130:67\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)\n    at Object.exports.create (C:\\dev\\slp-api-consumers\\typescript-client\\src\\api\\slp\\test\\organization.api.ts:15:5)\n    at createPopulatedOrganismeOptions (C:\\dev\\slp-api-consumers\\typescript-client\\src\\common-procedures\\create-populated-organisme.ts:95:27)\n    at CustomWorld.<anonymous> (C:\\dev\\slp-api-consumers\\integration-tests\\step-definitions\\groups.steps.ts:50:25)",
    "duration": {
      "seconds": "10",
      "nanos": 73000000
    }
  },
  "actionLocation": {
    "uri": "step-definitions\\groups.steps.ts",
    "line": 41
  },
  "sourceLocation": {
    "uri": "features\\my-desk\\groups\\create-group-students.feature",
    "line": 27
  },
  "text": "a class with a teacher, 30 seats and 0 students exists"
}

We are currently porting our test suite from version 4 to version 7 and we had a custom formatter which displayed some information about our system based on request responses which we attached to the exception that our http client triggered upon failure. It was very useful to have access to the exception in the formatter for this reason.

Is there a different way to access exceptions now? Would you accept a pull request adding it to the result object of the test step object?

[aslakhellesoy]

It wouldn’t hurt to bring this back would it @charlierudolph @davidjgoss ?

[aslakhellesoy]

That said, we’re encouraging our community to build formatters around the standardised message protocol, which allows formatters to be used by all cucumber implementations. This api is also more stable.

Here is an example : https://github.com/cucumber/cucumber-js/blob/master/src/formatter/html_formatter.ts

[oleduc]

That said, we’re encouraging our community to build formatters around the standardised message protocol, which allows formatters to be used by all cucumber implementations. This api is also more stable.

Here is an example : https://github.com/cucumber/cucumber-js/blob/master/src/formatter/html_formatter.ts

Oh that is fine so long as the exception is accessible somewhere along the way. Any pointers as to what needs to be done to pass exceptions up to the formatter constructor or the event collector? I'd be willing to give this a try.

[charlierudolph]

Hmm, I think for this we may want a new field on TestStepResult in cucumber messages. Maybe details and it could be a json string that can have more structured error information?

[jan-molak]

@charlierudolph - I agree with @oleduc, the formatted stack trace makes it very difficult for custom formatters to understand what error has been thrown and with what properties. It also makes it impossible to deserialise such an error to extract its details (i.e. the original expected and actual properties of an AssertionError), call methods on it, etc.

However, I'm not sure if adding a new details field on TestStepResult itself would help to resolve the issue.
To my understanding, anything that's passed via Cucumber messages needs to be serialised. Cucumber, of course, has no way of knowing how to serialise the errors so that they could be reliably de-serialised into valid objects afterwards.

So what I think could work here is to:

• return the original error from StepRunner.run method, alongside the TestStepResult:

// cucumber-js/src/runtime/step_runner.ts 
export async function run({
  defaultTimeout,
  hookParameter,
  step,
  stepDefinition,
  world,
}: IRunOptions): Promise<{ 
    result: messages.TestStepResult, 
    error: any,            // "any" because it could be Error but also anything else people might feel like throwing...
}>{ 
  // ...
}

• extend ITestStepHookParameter to include error, which would allow afterStep and after hooks to receive such error without breaking backwards compatibility:

export interface ITestStepHookParameter {
  gherkinDocument: messages.GherkinDocument
  pickle: messages.Pickle
  result: messages.TestStepResult
  error?: any
  testCaseStartedId: string
  testStepId: string
}

• do the same with ITestCaseHookParameter:

export interface ITestCaseHookParameter {
  gherkinDocument: messages.GherkinDocument
  pickle: messages.Pickle
  result?: messages.TestStepResult
  error?: any
  testCaseStartedId: string
}

I think this approach would:

• allow custom formatters to have access to the original error object
• maintain backwards compatibility
• avoid polluting Cucumber messages with duplicate information

@aslakhellesoy, @charlierudolph, @davidjgoss - Please let me know if you're happy with the approach and I can prepare a PR

[davidjgoss]

Sorry for the late reply @jan-molak.

Having the original exception on the param for After and AfterStep makes sense to me.

I think we could do something with messages as well though. From what I've seen, JS assertion libraries are pretty consistent about having actual and expected properties on the error, so we _could_ look for those and add their JSONified value to a message on a best effort basis. I'm not sure how straightforward this would be for other langs but it would enable things like showing a GitHub-style diff in the HTML formatter for an assertion error.

[jan-molak]

Having the original exception on the param for After and AfterStep makes sense to me.

Fab, that would be of great help! Should I send you a PR?

I think we could do something with messages as well though. From what I've seen, JS assertion libraries are pretty consistent about having actual and expected properties on the error, so we could look for those and add their JSONified value to a message on a best effort basis. I'm not sure how straightforward this would be for other langs but it would enable things like showing a GitHub-style diff in the HTML formatter for an assertion error.

Potentially, yeah. Serenity/JS AssertionError also captures the actual and expected. We also capture the cause, and other libraries might have additional custom properties.
Speaking of properties, actual and expected could be instances of objects, so serialisation of those would need to capture their type (i.e. CustomerRecord rather than Object). I think it can all get quite tricky.

🤔 But maybe it's the right thing to do to format the diff on the Cucumber side before creating a message. Maybe the issue is that the representation of the error in the message is trying to be two things at the same time - a stack trace and a diff. Maybe they should be represented in the message separately?

[davidjgoss]

Having the original exception on the param for After and AfterStep makes sense to me.

Fab, that would be of great help! Should I send you a PR?

Go ahead!

The potential messages change for more granular info on a failed result is now raised under https://github.com/cucumber/common/issues/1638.