Cucumber-js: Run specific scenario in v3

[mmuller99]

Hi there,

Since v3 you are unable to run a specific scenario within a feature file. It will run all the scenarios.

ex. helloworld.feature:46

Thanks,
Marco

[charlierudolph]

We have features to verify this is working: https://github.com/cucumber/cucumber-js/blob/v3.0.1/features/target_specific_scenarios_by_line.feature (and I just verified it still works on cucumber-js own test suite). Any more detail on the matter you can give? (sample project structure and the full command you ran) If you supply a file and line, it should not run all scenarios. It should be running just the matching scenarios (or none if nothing matches).

[mmuller99]

Hi Charlie,

Please find attached test project
test-cucumber.zip

Am running Windows 10, using zsh, but same happens in PS.

[mmuller99]

Running the https://github.com/cucumber/cucumber-js/blob/v3.0.1/features/target_specific_scenarios_by_line.feature tests pass

[charlierudolph]

Downloaded your test project and it works for me (on a Mac). We have appveyor CI and are able to run cucumber there and the same test.

This is very odd. Sorry I don't think I can be of much help here. Its especially odd if that test passes for you locally but then this isn't working properly.

Could you possibly debug further by adding some console.log statements to the installed node module? Some places to look at printing out the argv passed to cli/argv_parser and what feature paths are passed to the pickle_filter

[mmuller99]

Found the issue. Running in PS:

The featureUriToLinesMapping object has the paths exactly as what is being passed when running the tests:

featureUriToLinesMapping: { '.\features\bar.feature': [ 3 ] }

where the pickle filters uri has been normalized to:

uri: features\bar.feature

Thus not matching on this.featureUriToLinesMapping[uri];

[mmuller99]

Also in cygwin, the pickle filter is normalizing the path to Windows format instead of UNIX:

featureUriToLinesMapping: { 'features/bar.feature': [ 3 ] }
uri: features\bar.feature

[gd46]

@charlierudolph I was having a similar issue with getting this feature to run a specific scenario by line number. Im working on Mac, and I have this project setup:

https://github.com/gd46/dibellag-automation-framework

You can try running an example with:

npm test -- --browserName chrome --specs test/features/cucumber/transform.feature:14

And I am still seeing all scenarios in the feature being executed.

[charlierudolph]

@gd46 Taking a quick look at your example in appears your framework uses --features not --specs

[gd46]

After further analysis I found the issue to be a side effect of this defect here and in protractor-cucumber-framework it appears when you pass it a spec file it is resolving it by absolute path. And since it doesnt run just the one scenario when you have the "./" in front it doesnt work. The moment I try to hack protractor-cucumber-framework to have just this pass:

test/features/cucumber/transform.feature:14

It ran just the one scenario.

So Im not sure why the spec path is absolute if it needs to be or if it can match what the consumer passes in as specs. And Im not sure where the spec pattern gets converted to an absolute path.

[gd46]

From what I can tell so far it appears that this line:

https://github.com/cucumber/cucumber-js/blob/master/src/cli/index.js#L67

Returns featurePaths as absolute paths. So if I write my specs as:

test/features/cucumber/transform.feature:14

or

./test/features/cucumber/transform.feature:14

It comes out to be:

/Users/dibellag/dibellag-automation-framework/test/features/cucumber/transform.feature

Which doesnt allow the run spec file line number to work for me.

[charlierudolph]

@gd46 The configuration builder expands the feature paths in order to read the files but the unexpanded paths are kept for the filter options. https://github.com/cucumber/cucumber-js/blob/master/src/cli/configuration_builder.js#L47

[gd46]

Hmmm ok. Do you think the issue is in protractor? I see inside protractor cucumber framework index.js exports.run spec is the absolute path.

[gd46]

@charlierudolph I tried what @mmuller99 was talking about in regards to featureUriToLinesMapping. In my case I do not see it matching because it contains the absolute path and its comparing it against the relative path of the spec.

[gd46]

@charlierudolph I am able to test my example if I update this line:

https://github.com/cucumber/cucumber-js/blob/master/src/pickle_filter.js#L50

to the following:

let path = require('path')
 const lines = this.featureUriToLinesMapping[path.resolve(uri)]

This was my short hack to see it working within my test setup.

[mmuller99]

I had to do a path.normalize on both getFeatureUriToLinesMapping and matchesAnyLine to get both my scenario as well as the cucumber-js lib tests to pass. Not pretty, just a quick test

pickle_filter.js

import _ from 'lodash'
import path from 'path'
import { TagExpressionParser } from 'cucumber-tag-expressions'

const FEATURE_LINENUM_REGEXP = /^(.*?)((?::[\d]+)+)?$/
const tagExpressionParser = new TagExpressionParser()

export default class PickleFilter {
  constructor({ featurePaths, names, tagExpression }) {
    this.featureUriToLinesMapping = this.getFeatureUriToLinesMapping(
      featurePaths || []
    )
    this.names = names || []
    if (tagExpression) {
      this.tagExpressionNode = tagExpressionParser.parse(tagExpression || '')
    }
  }

  getFeatureUriToLinesMapping(featurePaths) {
    const mapping = {}
    featurePaths.forEach(featurePath => {
      const match = FEATURE_LINENUM_REGEXP.exec(featurePath)
      if (match) {
        const uri = path.normalize(match[1])
        const linesExpression = match[2]
        if (linesExpression) {
          if (!mapping[uri]) {
            mapping[uri] = []
          }
          linesExpression
            .slice(1)
            .split(':')
            .forEach(function(line) {
              mapping[uri].push(parseInt(line))
            })
        }
      }
    })
    return mapping
  }

  matches({ pickle, uri }) {
    return (
      this.matchesAnyLine({ pickle, uri }) &&
      this.matchesAnyName(pickle) &&
      this.matchesAllTagExpressions(pickle)
    )
  }

  matchesAnyLine({ pickle, uri }) {
    const lines = this.featureUriToLinesMapping[path.normalize(uri || '')]
    if (lines) {
      return _.size(_.intersection(lines, _.map(pickle.locations, 'line'))) > 0
    } else {
      return true
    }
  }

  matchesAnyName(pickle) {
    if (this.names.length === 0) {
      return true
    }
    return _.some(this.names, function(name) {
      return pickle.name.match(name)
    })
  }

  matchesAllTagExpressions(pickle) {
    if (!this.tagExpressionNode) {
      return true
    }
    return this.tagExpressionNode.evaluate(_.map(pickle.tags, 'name'))
  }
}

[gd46]

@mmuller99 Thanks for your findings, I was trying to resolve why this wasnt working for me a couple weeks ago when I realized your findings was a similar issue to mine. I think we need to do more than just path.normalize or path.resolve. I think the end solution should do both, because in the case where the spec path input is absolute we are always trying to compare it with a relative path. I think these two should be normalized to be the same and then use path.normalize to make sure its windows safe. Thoughts?

[mmuller99]

@gd46 Thanks for your input. The normalize won't be necessary if we are always going to use absolutes, as the paths will get normalized internally during the path.resolve call. So then only path.resolve is required at both locations

[gd46]

I see, thanks @mmuller99. @charlierudolph What are your thoughts on doing a path.resolve on both the input spec path and the sourceLocation uri when doing these comparisons to make sure there always accurate?

[gd46]

I see it work correctly on my end if I update your path.normalize to path.resolve. This should resolve the Windows path issue, and the comparing absolute path to relative path issue. I think we should probably add more tests here:

https://github.com/cucumber/cucumber-js/blob/master/src/pickle_filter_spec.js

In order to capture these scenarios.

[mmuller99]

@charlierudolph any thoughts?

[charlierudolph]

I'm good with using path.resolve in both places

[gd46]

Great @mmuller99 would you like to create the PR since you found the issue? Otherwise I can create one if you need.

[mmuller99]

@gd64 I'll add the PR, is there any specific test scenario(s) we need to look at?

[charlierudolph]

If you can create a failing test in pickle_filter_spec that is fixed by using path.resolve that would be great

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.