Protractor: Enable jasmine's stopSpecOnExpectationFailure feature for protractor

[machtyn]

(See also: http://stackoverflow.com/questions/37334069/protractor-jasmine-and-stopping-test-on-first-fail )

Jasmine 2.3.0 added an option stopSpecOnExpectationFailure that when set to true will stop a test on the first failure. However, attempting to set this option with jasmineNodeOpts failed.

/*
 * conf.js
 */
exports.config = {
    framework: 'jasmine',
    specs: ['search-spec.js'],
    useAllAngular2AppRoots: true,
    jasmineNodeOpts: {
        stopSpecOnExpectationFailure: true
    }
};

Additional information from user alecxe:

From what I see in the protractor and jasmine-npm (the jasmine runner that protractor uses) source code, it is not as simple as adding the stopSpecOnExpectationFailure or stopOnFailure jasmine node option to the configuration. There has to be changes applied to protractor to support the new stopSpecOnExpectationFailure jasmine node option. Please create an issue/feature request in Protractor issue tracker.

I've also tried to call the throwOnExpectationFailure function, set the stopSpecOnExpectationFailure value throw jasmine.getEnv() in onPrepare() but none of that worked. Perhaps, I am missing something and there is a way to enable the setting.

[JS31096]

Is there an ETA for this ?

[JS31096]

Hi @mgiambalvo ,
Is this feature included in the new version (4.0.0) , if yes pleases provide any documentation link.

Thanks in advance.

[juliemr]

It's non-trivial to add Protractor support for stopSpecOnExpectationFailure because it doesn't work with asynchronous tests. You can see this with plain old Jasmine and a test that looks like:

describe('async foo', function() {
  it('should bar', function(done) {
    setTimeout(function() {
      expect(1).toEqual(2);
    }, 10);
  });
});

You will see that instead of completing nicely, this tests will error without giving the reporter a chance to complete or letting the test shut down nicely.

We could, potentially, set up a global handler for uncaught exceptions and try to parse out when one is coming from Jasmine, but I'm very hesitant to add this infrastructure for one specific framework, especially since I wonder if Jasmine will be changing its support in the future.

I filed an issue with Jasmine at https://github.com/jasmine/jasmine/issues/1213, let's see if there's an update there. Putting this in the 'blocked' milestone for now.

[juliemr]

Still pending Jasmine fixes.

[gustavomick]

@juliemr any workaround while this is fixed by js team?

currently as workaround I am decorating expectation meths with this:

`
checkFailAfterExpect = () => {

    if ( jasmine.getEnv().failedExpectations.length > 0) {
        return promise.rejected('IT finished with checkFailAfterExpect()');
    }
    return when(true);

}
`
then using (kindof) fail fast. dont know if is best think to do thou, thanks.

[mariuszbeltowski]

@gustavomick I use "protractor-fail-fast": "^2.0.0", for few months and works fine

in config:

const FailFast = require('protractor-fail-fast');
export const config = {
    onPrepare: () => {
        jasmine.getEnv().addReporter(FailFast.init());
    },

    onCleanUp: () => {
        FailFast.clean();
    }
}

[gustavomick]

@Belciu fail fast is not enough for me because at least at my code
1 - doesn't stop after first failed expectation but after first IT, at least i tried and having those results
2 - ff is disabling all suites, what to my biz doesn't make sense, we agreed primary "test development design rule" where we consider suite as "one isolated user interaction path/test case". with that then we need to check all other paths (suites) because means totally different navigation flow.

btw, i used ff and added/modified failing by suite options.. code similar to this .. to finish one suite instead of all suites

function disableCurrentSuite(refs) {
  let env = jasmine.getEnv();
  (<any[]>refs.suites).find((suite) =>
 {
    let spec = suite.children.find((spec) => {
      let fs = spec.id === env.currentSpec.id; // if this is current spec then disable all suite
      if (fs) {
        let csuite = suite;

        // look for root suite to disable all tree top->.spec-> multilevel suites
        while (csuite.parentSuite && csuite.parentSuite.description !== 'Jasmine__TopLevel__Suite') {
          csuite = suite.parentSuite;
        }
        disableSuiteAndDescendants(csuite);
        // 
      }
      return fs;
    });
    return !!spec;
  });
}

let disableSuiteAndDescendants = (suite) => {
  suite.children.map((child) => {
    if (child.children) { // child is suite      
      disableSuiteAndDescendants(child);
    } else { // child is spec      
      child.disable();
    }
  })
  suite.beforeFns = [];
  suite.afterFns = [];
  suite.beforeAllFns = [];
  suite.afterAllFns = [];
}

[Ne-Ne]

Is there any traction on this?

I am writing big singular tests. There are very much dependant on a user creating a form before the form gets tested. If the form creation fails. Then the tests needs to fail at the point a.s.a.p

Many thanks.

[TWiStErRob]

@juliemr Your reported issue was duplicated against a more generic error handling implementation,
which was fixed in https://github.com/jasmine/jasmine/commit/1042c9a2ddd0f454192f91e27c17cc54158b06d2 in March which landed in ^2.6.x, I'm suggesting to remove the Blocked milestone.

[khitrenovich]

Are there any expectations for this feature to be supported in Protractor too?

[nbabii]

Also looking for stopSpecOnExpectationFailure feature to be supported by protractor.

[ppitonak]

Any updates?

[demisx]

Please enable stopSpecOnExpectationFailure for Protractor. It will save a bunch of time.

[cnishina]

Going to say I might have fixed it with https://github.com/angular/protractor/pull/5102. Does upgrading to Jasmine 3.3.0 resolve some of these issues? If so, I would like to also close this. I haven't had the time to try out this issue.

[vickyru]

@cnishina this bug is still there i tried it with jasmine 3.3.0.
Please provide this feature to end users. Much awaited feature.

[Joralf]

It looks like https://github.com/Updater/protractor-fail-fast is the only working solution to stop a test in case an error is thrown. It seems however one of it's dependencies (https://github.com/Updater/jasmine-fail-fast) is no longer maintained and this poses a security threat, because of a dependency to an old lodash version.

Here's details on the lodash issues:

• https://snyk.io/vuln/SNYK-JS-LODASH-450202 (current issue)
• https://nvd.nist.gov/vuln/detail/CVE-2018-3721 (last year's issue)

The jasmine-fail-fast repo is not under the communities control, so I think it would be great to implement stopSpecOnExpectationFailure.

From the implementation it looks like we can already pass nodeOpts but it doesn't seem to work: https://github.com/angular/protractor/blob/master/lib/config.ts#L621. Any advice/suggestions?

[Regenhardt]

@juliemr The blocking issue you opened with jasmine seems to have been solved some time ago, do you have a status on this one? We'd love to see this feature.

[autTestUser]

@juliemr Please update on this. Enabling 'stopSpecOnExpectationFailure' for Protractor helps a lot.