Cucumber-js: implement --retry

[ericyliu]

it would be nice if there was a feature that retries the test suite X times, and passes a test if that test passes on any of the runs

[westlakem]

@ericyliu It's coming in cucumber 3. Not sure why Cucumber 2 is still in RC though... it's been out for 2 years

[chrisui]

@charlierudolph or anyone if you had any pointers I would be happy to pick this up and give it a go.

My presumptive approach would be that at the end of a scenario running we can tell if it failed or not and re-run it (keeping some state for the retried count). Where can I hook into this?

As well as this I would expect at the end of the whole process to be able to access a list of all scenarios, the number of times they retried and their final statuses so this can be a) logged and b) dumped to disk if desired.

[charlierudolph]

Closing as I personally don't want to implement this as I disagree with it. I don';t like providing a feature that can be used to deal with a flickering test instead of fixing the cause of the flickering.

[aslakhellesoy]

👏👏👏

[westlakem]

@charlierudolph sometimes it's impossible to deal with flickering tests. For example, in our environment we have 23 different services that our application hits, and as we don't own those services any of them can be down at any given time. the --retry flag allows us to rerun the test so a temporary service outage (when they restart the service) doesn't cause our test suite to fail.

[chrisui]

With respect to personal disagreement with this feature (this argument can be as fruitless as tabs vs spaces in the engineering world), if Cucumber as a tooling spec is implementing this we should also support this in the JS version of this tool.

It is also in the real world with broad tests suites and complex applications impossible to defend 100% against flakeyness and therefore becomes incredibly expensive. Absolutely you can have an engineering journey to improve it but I doubt there are many companies with the luxury of resources to support such an effort with any decent ROI.

In the real world being able to retry failed tests with the addition of good tracking to identify flakeyness for pro-active actioning is super valuable.

I'd appreciate if we reconsider the decision to close this issue and as stated before I am more than happy to work on this with some guidance.

[chrisui]

To add to this cucumber-js already has rerun functionality which is basically just a retry but less efficient as it has to be run as a separate process (this is providing im understanding it correctly). Retrying is a much nicer solution.

cc @charlierudolph

[charlierudolph]

It is also in the real world with broad tests suites and complex applications impossible to defend 100% against flakeyness and therefore becomes incredibly expensive.

I absolutely hate flakey tests. On my current projects, I am trying to move away from running any flakey test on CI and saving those for manual runs where if something fails due to flakiness you can rerun it or check it manually. This is dependent on finding your source of flakiness and limiting the number of tests that have to deal with it.

To add to this cucumber-js already has rerun functionality which is basically just a retry but less efficient as it has to be run as a separate process (this is providing im understanding it correctly). Retrying is a much nicer solution

Rerun was built so you can easily focus on the tests that need to fixed. Yes, you can use it to retry just the failed scenarios.

[localjo]

I agree 100% with flaky tests being something to avoid. That being said, the flakiness of a test is often something that's not under a team's control. Here's my situation;

I'm using Cucumber, Nightwatch, Selenium and Browserstack to run end-to-end tests on a web app. Often, a scenario will fail due to the inherent flakiness of Selenium, Browserstack or the browser I'm testing in. Interactions involving mouse movements are notorious for this. It's rare for me to be able to run through all of my scenarios without at least a handful being flaky.

I can't make Selenium, Browserstack or browsers less flaky, that's just the nature of the tools I have to work with. The solution I need is to be able to retry my scenarios a couple of times if they fail.

[aslakhellesoy]

I think we should reconsider this. Cucumber-Ruby has it now.

[localjo]

Agreed. Here's the code for the Ruby implementation. Could be useful for a JS implementation. https://github.com/cucumber/cucumber-ruby/pull/920/files

[hurrikam]

Hi everybody, is the 'retry' feature now available in Cucumber-JS? As much as I agree flaky tests should be fixed, experiences shows it's not always possible when working in large projects with many external dependencies involved.
At the end of the day, if such a flaky test case couldn't be covered by automation because of the missing retrial logic, then it would have to be manually tested anyway.
I believe using the retry logic should be left as a choice to the Cucumber user.
I'm aware of the 'rerun' feature but using it adds extra complexity as tests have to be separately re-run opposite to have the Cucumber runner transparently do it for you.

[hurrikam]

Hi @charlierudolph and @aslakhellesoy. Any progress with this feature?

[aslakhellesoy]

Nobody is working on it to my knowledge. If anyone provides a pull request I'd consider adding it though.

[hurrikam]

Hi @aslakhellesoy thanks for replying so quickly. This is something we're really in need at the moment as one of the devices we need to automate is a bit unstable and in such a case there are no really practical alternatives to just re-running a failed scenario.
My idea is to use a per scenario tag to specify the retry count but at least as a starting implementation that could be just a command line parameters.
I'll try have a look at the code in the next few days.

[aslakhellesoy]

That would be great! Please try to make it behave in the same way as in Cucumber-Ruby. If it diverges from that we may not merge it, as consolidation/consistency is something we're striving for.

[hurrikam]

Hi @aslakhellesoy, I've implemented the base retry functionality as the CLI --retry option. I've tested it at work with our project and it works well. I'd need permissions to push my working branch and open a PR to discuss the implementation. Let me know, thanks.

[jain-neeeraj]

@hurrikam Great work.. 👏👏👏
I am also looking forward to use this. I am using nightwatch-cucumber and they are waiting this to be fixed in cucumber js. Ref # https://github.com/mucsi96/nightwatch-cucumber/issues/213

[hurrikam]

@charlierudolph could you please help with this? I can't submit my PR at the moment.

[charlierudolph]

@hurrikam you can create a fork and push your code there then submit a PR from your fork into this repo. https://help.github.com/articles/fork-a-repo/

[hurrikam]

@charlierudolph please see PR-1114
As mentioned in the title this is the minimalist implementation (very few code changes indeed) to get the feature working without breaking existing formatters and reports.

I went through some old conversations in the Cucumber Ruby repo in regard on whether each run of a flaky test should be reported, possibly introducing a 'flaky' result status for the previous runs. Not sure if that was done and again, the draw back would be to get all the attempts of the same test case into the logs.

Let's discuss it.

[Nick-Lucas]

@hurrikam my team have taken a fork of the PR so we can get this functionality. It's working really well so far. I hope you can get it merged soon as the flag is a huge help!

[thomaswmanion]

@Nick-Lucas That's awesome - Do you have that in?

[Nick-Lucas]

@thomaswmanion yep we're actively using the feature. We have a backend that can sometimes get a little choked up flushing through job queues, so it's a been lifesaver.

For anyone wanting to use the PR you can point your package.json at my fork (or fork it off me):

  "dependencies": {
    "cucumber": "https://github.com/Nick-Lucas/cucumber-js.git#feature/issue-727-retry",
  },

Cucumber is pre-built so it can just be installed into node_modules. It should go without saying that there's definitely no warranty or support as the feature isn't even pre-alpha yet

EDIT:

I have now left my company and handed over the fork to their github: https://github.com/wonderbill/cucumber-js.git#feature/issue-727-retry

[thomaswmanion]

Nice!!!! Thanks @Nick-Lucas - appreciate the understanding that environments of systems can be flaky in nature!

[robert-fahey]

This needs to be merged into Master, this isn't closed.

[ricardgarcia]

@thomaswmanion yep we're actively using the feature. We have a backend that can sometimes get a little choked up flushing through job queues, so it's a been lifesaver.

For anyone wanting to use the PR you can point your package.json at my fork (or fork it off me):

  "dependencies": {
    "cucumber": "https://github.com/Nick-Lucas/cucumber-js.git#feature/issue-727-retry",
  },

Cucumber is pre-built so it can just be installed into node_modules. It should go without saying that there's _definitely no warranty or support_ as the feature isn't even pre-alpha yet

EDIT:

I have now left my company and handed over the fork to their github: https://github.com/wonderbill/cucumber-js.git#feature/issue-727-retry

How can I run the retry feature when running tests?

[individual-it]

@ricardgarcia just use the --retry option see https://github.com/owncloud/phoenix/pull/1207/files
We are using the branch from my PR here https://github.com/cucumber/cucumber-js/pull/1205 till it get merged hopefully one day

[ricardgarcia]

Hi @individual-it , many thanks for the reply. Should it work if I'm using webdriverIO to run my tests on cucumber-js?
I've tried the "--retry 1" command but no retries are being done.

[jain-neeeraj]

@ricardgarcia As described in https://github.com/cucumber/cucumber-js/pull/1205, to use this function you can use

"cucumber": "cucumber/cucumber-js#issue-727-retry"

as the changes are not yet merged in the main cucumber repo.

[ricardgarcia]

Yes @jain-neeeraj I've used this version in the package.json and it did not work, but as I mentioned I'm running tests using cucumber on webdriverIO, so the following framework "wdio-cucumber-framework". Should it work somehow? How do I run the tests in order to have a retry if I use cucumber-js framework?

[jain-neeeraj]

@ricardgarcia lets collaborate at gitter https://gitter.im/cucumber/cucumber-js, I should be able to help you in solving this issue.

[hardvolk]

@charlierudolph any chance this could be merged into master?