Cucumber-js: Result is missing scenario & tags

[manu7691]

Any way to access to scenarioResult.scenario.tags in the new version v3? Is it possible to override somehow test-case-finished event to pass old model scenario_result? Thanks

[charlierudolph]

Nope. You can not easily access the tags of the test case. There is no plan to return to the old scenario result. What is your use case for needing the tags?

[manu7691]

In my case i have dependency on tag system. In order to get some data in the afterhooks.

Something like this:

@SuiteName @SuiteSectionName <- These tags tell the suite
Feature: 

@TC1563697 <- This tag identifies the testcase in the test management tool @New
Scenario: 
    Given  
    When 
    Then 

[charlierudolph]

You can create hooks that run only for scenarios with specific tags: https://github.com/cucumber/cucumber-js/blob/v3.0.1/docs/support_files/hooks.md#tagged-hooks

[gd46]

I also have several use cases for knowing the scenario tags. Given the following sudo example(missing version syntax a bit as I have test cases in 1.3.1, and im looking at the latest 3.0.1):

@set_video @youtube
Scenario: User should see youtube video

@set_video @vimeo
Scenario: User should see vimeo video


this.After({tags: @set_video}, function (testCase) {
  let tags = testCase.scenario.tags;

_.forEach(tags, (function (tag) {
 if(tag === '@youtube') {
   setVideo('youtube');
 }
if(tag === '@vimeo') {
 setVideo('vimeo');
}
});

}

I have one tag that says when the hook should be run and I have other tags that act as data to make the hook more dynamic to work for other use cases. Otherwise I find myself creating the same hook with the same logic and only changing the data I pass it. I find knowing the tags to be very useful and a very powerful tool to making hooks more flexible.

Can I also ask here I cant seem to get the result object in the After hook in 3.0.1. I tried testCase, scenarioResult, and scenario. Am I missing something?

[pawelus]

We have a case where we keep our test cases in TestRail and we download them and save as feature files before the run.

Then in the after hook we send detailed test results to our SQL database. Those results include:

• feature ID from TestRail - taken from a tag (each feature has automatically generated tag with feature ID)
• exception that was thrown - taken from scenario.getException() in cucumber 1.x
• all tags the feature is tagged with
• steps that failed - we use stepResult hook to get the result of every step
• a bunch of other info grabbed from TestRail using feature ID taken from tags

So with the current change in cucumber 3.x we will never be able to switch as it completely breaks our infrastructure.

[yaronassa]

@pawelus My infrastructure does exactly the same thing.
Seems like since you don't lose anything from performing these actions asynchronously (i.e. your actual test infrastructure doesn't "care" about the TestRail update), you can move that code into a custom formatter, and use the information coming from the event protocol to build and send the reports.

Personally I have a wrapper script that launches cucumber.
It download the TestRail scenarios before cucumber is launched, so it wasn't a huge problem for me to move the TestRail report code out from cucumber hooks and into the wrapper script.
After cucumber completes, the script reads the outputted cucumberResults JSON, and compiles all the information it needs from there.

Sounds like you'll have to rearrange your code either way.
I think moving all pre/post actions to a wrapper script is a nice solution that restores some of the control we've lost in V3.
It's still somewhat a pain, as I have to serialize important context information to enrich the outputted results (as my infrastructure state is destroyed by the time the relevant code runs), but it's doable.

[gd46]

I think losing the ability to know tags in hooks would be a shame. It was the only way to make a little more configurable since you can't pass it parameters I was adding extra tags and then see which were applied to the current scenario to call functions with different inputs.

[pawelus]

@yaronassa we run our tests through Protractor, we do not launch cucumber directly so there's yet another layer of abstraction to count in here.

We do download features before Protractor starts as you do but sending results to the database is a different story.

With sharding & tests running on Selenium grid and reruns of failed features it will be quite a hassle and complex logic to get the results in the right order in the database. Quite an amount of work to restore the capabilities we had in cucumber 1 and 2.

Also creating a custom formatter only to get step results just doesn't sound right.

[yaronassa]

Hey, I'm with you.
I would have wanted unrestricted access to cucumber's current state - feature, scenario and step, with it's entire property collection, and write access (I would even want access to the future "call stack" of the scenarios, with the ability to manipulate it beforehand).

Seeing as cucumberJS is deliberately moving away from this kind of "inner-visibility", I'm offering the kind of solutions I think can work in the foreseeable future. Personally, I think there'll come a time where tinkerers like us would have to resort to overriding inner methods in Cucumber's internal to retain this sort of access.
And that's OK - I guess there're a dozen of us, and thousands of more casual users.

[bnadim]

If possible i would also back the name of the scenario. Indeed i am adding a snapshot capability and i need tu know the name of the scenario to name my snapshots.

[charlierudolph]

@gd46 you could do the following:

this.After({tags: "@set_video and @youtube"}, function () {
  setVideo('youtube');
})

this.After({tags: "@set_video and @vimeo"}, function () {
  setVideo('vimeo');
})

That doesn't have any duplication aside from the @set_video flag.

---

@pawelus

Then in the after hook we send detailed test results to our SQL database.

Do you need to do this in a After hook? Could you do this after your tests finish by parsing the json formatter results? You could use the event protocol formatter in order to continue to this is as results occur although that would require some more processing. A by product of the 3.x changes is that parsing results moves out of the support files and to standalone processes. I think this is a better separation of things and ideally how things were built originally.

---

@bnadim

You could add screenshots with the attach function to have them get outputted in the event potocol / json formatters and then do some post processing to save those to files based on the scenario name. Side not: the scenario name is not guaranteed to be unique whereas the scenario uri and line number are.

[gd46]

@charlierudolph I suppose that is a solid replacement solution for my needs. It also removes the need of analyzing the current running scenario. It just might lead to defining several combinations of hooks that account for the different combinations a given function could handle within a hook.

So for instance I have similar examples where I get the current running scenarios tags split it based on a pattern and use part of the match as the parameter to a function. In these cases all the steps that need to happen are the same and the only thing that changes is the parameter. So this would cut down on several setup steps in the hook so it could work for multiple cases.

One such example:

tags: @clear_w2OnlyUser, @clear_w2OnlyArcotEnableUser

I split based on @clear_ and grab the second half as the parameter. tagName coming from the old scenario result of getTags, getName. 

let profileToClear = tagName.split('@clear_')[1];

browser.waitForAngularEnabled(false);
browser.get(url);
login();
navigate();
deleteProfile(profileToClear);

Let me know your thoughts on this? I do believe your example would be an easy replace in some cases. But in others where there are several extra steps that need to happen you could end up potentially duplicating all these steps.

[justusromijn]

My use case is that I setup a mockserver for each scenario, based on the tag I know how it should behave. Adding hooks for specific tagged scenarios is very maintainance-intensive, because I need to add a hook for each scenario that I support...

[Jordyderuijter]

We are also using the scenario.name in our Before and After hooks in order to log the name of the scenario. This way we can see when a certain scenario starts and ends when analyzing test results. This change breaks that feature.

[leggebroten]

I am using Cucumber-js to drive Selenium. Both local and Browserstack

I was using test state (feature, scenario, tags, results, etc) in the hooks for many essential things.

• to have Scenario-specific alterations of the URL
• dynamically skip tests based on the current browser configuration
• Use scenario-specific URL to record test results back to the Browserstack
• Use Feature and Scenario names to generate session names in Browserstack
• Parse tags to identify and set scenario specific browser resolution

All these were wrapped in apps to enable concurrent instances to reduce overall run time.

Why were these dropped?
Are they never coming back?

[charlierudolph]

@gd46

Does the scenario being run involve that type of profile? Maybe you could have the scenario save what type of profile is interacting to the world and then you have a single clear tag which prompts the removal of the saved profile?

---

@justusromijn

For you mock server could you move the setup from tag based to the using a step that defines what the setup is? Then you can parameterize easily.

---

@Jordyderuijter

You can log the scenario line and uri (which is actually unique and prevent you from having to search for the scenario name).

---

@leggebroten

to have Scenario-specific alterations of the URL

You can use a step for this instead or use unique tags

dynamically skip tests based on the current browser configuration

How were you dynamically skipping tests? #912 adds support for this

Use scenario-specific URL to record test results back to the Browserstack

You can use the line and uri instead of the name. As noted before, the line and uri are guaranteed unique while the name is not. I also suggest parsing the json formatter or event protocol formatter formatter for test results and using attachments if you need to add anything.

Use Feature and Scenario names to generate session names in Browserstack

You can use the line + URI

Parse tags to identify and set scenario specific browser resolution

You can use a step for this.

---

From all of these examples, I have yet to a use case that convinces me we should add back tags or name to the hook. Having the tags and name appears to have made certain things easier to do, but I think there are other ways to accomplish them that make more sense to me

[justusromijn]

@charlierudolph thanks for the reply. For the mockserver I already had a quick chat with some coworkers and using a shared "background" was in consideration for a solution, so yes you can cross that one of the list.

[leggebroten]

@charlierudolph Thanks for the reply.

While your suggestions of using Steps and parsing output could be made to work, it’s far inferior to prior versions where callbacks have access to test State.

0) Inconsistent with the Observer pattern. Callbacks should be provided the data they need for the behavior they’re supporting

1) Induces extreme fragility. Tests should be reliable or they're not going to be trusted. Line numbers and file names change. Simply adding or removing a carriage return will break tests.

2) Using Steps to alter State violates DRY. Consider the frequent case of adding A/B tests that are marked with a Tag. Unless you add command line extension to execute tests based on the Steps they include, tests will require BOTH Tag and their supporting, state-altering Steps.

3) Requires the developer to do extra work. Parsing output to find State is needless, tedious, error prone, binds tests to output formats (poor Coupling), and violates DRY.

4) Inconsistent with Cucumber. Adding Steps (a semantic part of the test) to alter the Feature's state is counter to intent of Cucumber of isolating test writer. If the behavior hasn't changed, the test shouldn't.

5) It's not declarative. Tags are metadata ABOUT the test, it is semantically consistent to use them to alter the state in which the test will run. Whereas Steps are embedded within the test. You identify a set of tests by their Tags … not the Steps they use.

Oh, and I wasn't actually "Skipping" tests, I was using callback( null, 'pending' ) to avoid running the test. The new 'skip' feature is superior solution.

[leggebroten]

@charlierudolph, I was thinking, another solution ...
when constructing the World object, provide reference to the Scenario object used for the test.

Since the World is available as "this" to all callbacks it would achieve full parity with V2 (provided AfterAll callback were given the Results)

[charlierudolph]

@leggebroten

1. I never heard the observer pattern being a design goal for cucumber.
2. Scenario names are also fragile as a single character change also changes it. I agree file / line number break when taking what feels like unrelated actions. I was suggesting the use in reporting of results and not in running tests and thus the fragile part doesn't have much negative effect.
3. Using Steps to alter State violates DRY I don't understand this. The tag is used for selecting features and the step is used for setup/actions/expectations. These are different purposes. We have minimal support for using the tag for setup with tagged hooks but it is not built to handle complexity.
4. I was only suggesting parsing formatter output for reporting. That should not be involved in running tests.
5. Adding Steps (a semantic part of the test) to alter the Feature's state is counter to intent of Cucumber of isolating test writer I don't understand this either. If you can't use steps to perform actions and set state, how are you running tests?
6. Tags are metadata ABOUT the test, it is semantically consistent to use them to alter the state in which the test will run. I disagree about that being consistent. Tags to me are only for identifying tests. There is support for minimal altering of state but not for complex altering of state with parameters.

There is no more scenario object in the system. I don't think parity with v2 should be a goal. I'm glad this conversation came up and am good looking for other solutions but I don't want to go back to what we had previously as I believe that was making users rely much more on the implementation of cucumber and not on its interface.

[leggebroten]

@charlierudolph,

Thank you for taking the time to discuss this issue. I truly appreciate the work you've done to provide us with Cucumber-js. But as it stands I cannot upgrade to V3.

I don't intend to be combative. I'm just worried and it may come out in my responses.

Using only the documented Event interface from the prior version, I have built a framework that dramatically reduces overall test run time by allowing Features to be run concurrently over a huge, configurable test matrix with includes Operating systems/Versions, Browsers/Versions, Desktop/Mobile, and Optimizely A/B tests.

I CAN'T use Steps. It's too late. The device type, OS/version and browser/version MUST be determined before the test begins or every Scenario will be forced to have a "set up OS" Step and a "set up Browser" Step. This is in direct conflict with Cucumber's goals.

Alternatively, as I mentioned earlier, you could provide the Scenario information (Feature URI, name, line, and Tags) to the World constructor. This is semantically consistent with the intent of the World object and simpler to encapsulate and extend. Not only will this eliminate most of my Event Handlers, but because the Handler's "this" is the World object, it is State consistent with the Observer Pattern.

1. I didn't say the Observer Pattern was a design goal for Cucumber. Merely that Hooks and Events are Observers and that, as such, they should be given the state they need to do their job. This Pattern allows separation between the caller’s implementation and event handlers.

2. I have no dependency on the actual names. But since they're unique, they are the means of building the associated user-meaningful session name in Browserstack.

3. Having access to the set of Tags during Before, allowed me to build reusable constructs for altering State (like setting Optimizely URL parameters) without modifying the test (adding Steps). This meant there was a 1-1 correlation between tests I wanted to run (Tags) and their setup. DRY.

4. You've expressed concern that developers were becoming dependent on implementation, yet your solution of removing necessary Observer State is to make the developer dependent on other implementation (file format)? How can forcing a user to do WAY more tedious, slow, and error-prone work parsing files be better than accessing a State property?

5. Obviously Steps are setting test State. But one of Cucumber’s design goals was that the person writing the Steps should be isolated from the underlying implementation. E.G. test semantic behavior without getting bogged down in the underlying details. Adding Steps to alter query parameters, and defining device/browser/os, seems counter to this goal.

6. For you Tags are only about identifying tests. But Cucumber Wiki has many uses for tags.

7. Organizing and grouping
8. Filters (running sets via command line)
9. Links to related documents (such as Optimizely flags)
10. Indicating where in the process the feature is

Having access to the Tags during Before callback allows me to ensure appropriate state based on the browser configuration that is being used to drive the test. Using Tags makes that Declarative. Using Steps obfuscates the Intent

The calls, and parameters of the Event handlers in prior versions WERE interface. If devs coupling their code with your implementation leaking out, pass a read-only “Scenario” Proxy to the event handlers.

[Jordyderuijter]

@charlierudolph thanks for your reply

Scenario names are also fragile as a single character change also changes it. I agree file / line number break when taking what feels like unrelated actions. I was suggesting the use in reporting of results and not in running tests and thus the fragile part doesn't have much negative effect.

True, scenario names are also fragile, but it is a lot less fragile way to reference scenario's than referencing them by file and line. We are using the scenario names in the logging daily to analyze failures in the run and see what went wrong. If the logging only displays the file and line number it will cause way too much overhead to trace back the scenario. Besides that, if someone made changes in the feature file or moved the scenario somewhere else in the meantime this will even get more complicated.

[charlierudolph]

If the logging only displays the file and line number it will cause way too much overhead to trace back the scenario.

How is it more overhead? You know the file and line number and can go directly there. If you have the scenario name, you have to search for that string to be taken to that file and line.

[Jordyderuijter]

How is it more overhead? You know the file and line number and can go directly there. If you have the scenario name, you have to search for that string to be taken to that file and line.

Like I said, if the feature file changed in the meantime and the scenario is not positioned on that line anymore. Especially when I am looking through logs of an older test run (for example 1 week ago). In this case I would have to go through git and see which scenario was on that line at that given time.

[mattwynne]

FWIW, cucumber-ruby has two extension points that I think satisfy the principle of "easy things should be easy, hard things should be possible" and might be ideas that could help with these use-cases.

First of all, we pass an instance of a RunningTestCase to each hook. This is an immutable value object that contains all the information about the Gherkin source (scenario / outline / tags etc) plus the current result status of the scenario. Being immutable means we're protected from having to consider cases where the user might try and change things on us, but equally it gives them transparency over what the current state of play is.

Second, we have filters. Filters allow you to manipulate the stream of test cases before they're executed. We use filters to, for example, remove test cases from the stream that don't match the tag pattern specified on the CLI. Filters are a power-user feature.

HTH

[mattwynne]

I just realised that the documentation behind that link for filters is terrible! Sorry. No time to fix it now, but there are some more examples here: http://www.rubydoc.info/github/cucumber/cucumber-ruby/Cucumber/Filters

[gd46]

Hi @charlierudolph,

Sorry its been awhile since I have follow up on this conversation. Can you elaborate on your statement:

"Does the scenario being run involve that type of profile? Maybe you could have the scenario save what type of profile is interacting to the world and then you have a single clear tag which prompts the removal of the saved profile?"

Im not sure I follow.

And I understand there maybe other ways of obtaining the same solution without the old test result. But from what I have seen so far in this conversation is that all alternative seem more complex then they need to be. The old test result had plenty of descriptive data about the scenario that was about to run and it was the only way to have "parameterized" hooks. Protractor as it stands from what I have seen and tried doesnt currently support running a feature from a scenarios line number. So the new testCase result doesnt have much I find useful other than the status.

I would like to see testCase match the the result returned when using getTestCasesFromFileSystem with PickleFilter. I have used this to do some interesting parallel work to support filtering of scenarios by tag to pass to protractor as shards. The information returned from that result is far more useful and I think would be extremely useful to have in testCase result.

Example result from PickFilter:

{ pickle: 
     { tags: [Object],
       name: 'Github test with page object',
       language: 'en',
       locations: [Object],
       steps: [Object] },
    uri: 'test/features/examples/github-example.feature' }

Im not saying it needs to match exactly but I see a lot in this result that others here and I would seem to benefit from.

If your interesting in my example, Im using it here: https://github.com/gd46/dibellag-automation-framework/blob/master/configuration.js#L91

[gd46]

@charlierudolph Is this where testCase is being set?

https://github.com/cucumber/cucumber-js/blob/fbff6b0fae54d2e341ee247addc60a9f05753f1d/src/formatter/helpers/event_data_collector.js#L22

From what I can tell there is a reference to pickle along side testCase. So why not return a few more bits from the pickle result onto the testCase result?

[charlierudolph]

@gd46 okay, lets add pickle to the object that gets passed to the hook replacing sourceLocation. That needs to be updated in this function: https://github.com/cucumber/cucumber-js/blob/master/src/runtime/test_case_runner.js#L153

[gd46]

Hi @charlierudolph, I just saw your comment. That would be great! I'll try to take a look at this soon. I'd be happy to make the contribution.

[gd46]

@charlierudolph Just wanted to clarify the change. Are we ok with the difference that pickle contains uri without the line number directly on it as sourceLocation does. And if whomever wishes to consume the uri with the line number they can use the line numbers returned from the pickle object? I see no issue with this just wanted to confirm.

[charlierudolph]

I guess let’s leave the source location object as is (instead of removing it) and add the pickle object.

[gd46]

Ok that works for me as well. So Im new to contributing to cucumber, Im trying to understand the structure still.

As you have pointed out the line that needs to be updated, I believe we can just add something like this next to sourceLocation:

pickle: this.testCase.pickle

Then anyone who wishes to consume it in the hook can access it as follows:

testCase.pickle.tags
testCase.pickle.name
etc. 

I have made the updated but Im not 100% sure how to update all the related tests. Would you be able to provide some guidance?

[gd46]

I was able to test the change by linking my fork with the changes to one of my local projects. The complete testCase result will look like this:

{
  "sourceLocation": {
    "uri": "test\/features\/examples\/example.feature",
    "line": 4
  },
  "pickle": {
    "tags": [
      {
        "name": "@example",
        "location": {
          "line": 3,
          "column": 3
        }
      }
    ],
    "name": "Custom Transform should take belly and capitalize it",
    "language": "en",
    "locations": [
      {
        "line": 4,
        "column": 3
      }
    ],
    "steps": [
      {
        "text": "I have cucumbers in my belly",
        "arguments": [

        ],
        "locations": [
          {
            "line": 5,
            "column": 10
          }
        ]
      }
    ]
  },
  "result": {
    "duration": 7,
    "status": "passed"
  }
}

After doing some more testing I noticed that pickle doesnt contain uri at the time we have access to it in test_case_runner. So I think its fine to keep source location as is.

This is what PickleFilter returns pickle as:

{
    "pickle": {
      "tags": [
        {
          "name": "@example",
          "location": {
            "line": 3,
            "column": 3
          }
        }
      ],
      "name": "Custom Transform should take belly and capitalize it",
      "language": "en",
      "locations": [
        {
          "line": 4,
          "column": 3
        }
      ],
      "steps": [
        {
          "text": "I have cucumbers in my belly",
          "arguments": [

          ],
          "locations": [
            {
              "line": 5,
              "column": 10
            }
          ]
        }
      ]
    },
    "uri": "test\/features\/examples\/example.feature"
  }

So everything will be the same minus pickle not having uri in it.

[gd46]

Opened a PR to highlight the changes so far. Still need to update tests.

[gd46]

Working on updating tests. I have this thing setup to change my local java version and realized thats why I wasnt able to run some the feature tests :/.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.