Newman: Exit status 0 even with failing test

[jhenaoz]

Actually if a test fail it should exit with status code -1 or diferent to 0.
When i run CI with travis or Jenkins if the result of npm test 0 the operation execute successfuly, even if there are test failing in the postman collection.

1. Newman Version: 3.4.2;
2. Linux, travis:
3. Newman library
4. recently
5. Expected behaviour: exit with status code -1 when test fail
6. Command / script used to run Newman: runing npm test in this repo:
   https://github.com/jhenaoz/MeetupPostman/blob/master/src/app.js
7. Sample collection, and auxilliary files:
   https://github.com/jhenaoz/MeetupPostman/blob/master/src/collection.json
   https://github.com/jhenaoz/MeetupPostman/blob/master/src/environment.json
8. Screenshots (if applicable):
   
   

Steps to reproduce the problem:

1. Download the repo: https://github.com/jhenaoz/MeetupPostman
2. npm install
3. npm test

[czardoz]

@jhenaoz, when used as a library, Newman does not exit at all, since most users who use it as a library need the flexibility of handling the error themselves.

If you use the CLI directly, Newman will exit with a non-zero code on errors. You can either do that, or do it in your script by calling process.exit() :)

[iampeterbanjo]

How would I determine that there was an error from the event summary?

From the docs I have failing tests but the sample code doesn't work - it always logs "collection run completed".

newman.run({...
}).on('done', function (err, summary) {
    if (err || summary.error) {
        console.error('collection run encountered an error.');
    }
    else {
        console.log('collection run completed.');
    }
});

[manodupont]

That is not closed i think! I also have that error!

newman run $myColl -e $myEnv --bail
echo $?

0

BUT, i do have an assertions!

What the hell !??!

myname@myMac test (master) $ newman -v
4.3.1
myname@myMac test (master) $ 

[shamasis]

Will have a look at it soon and reopen. Meanwhile, if you find the root cause, please feel free to send a Pull Request @manodupont

[Thor07]

@shamasis any update on this issue?I'm still facing this issue. Please share pointer to any duplicate thread in this resolution (if any)

[codenirvana]

When using Newman as a library, the following can be done to detect an execution failure:

newman.run({
    collection: '...'
    // ...
}, function (err, summary) {
    if (err || summary.run.error || summary.run.failures.length) {
        process.exit(1);
    }
});

For CLI usage, as mentioned in https://github.com/postmanlabs/newman/issues/883#issuecomment-275703436, Newman CLI will exit with a non-zero code on errors.

If you are facing any issues regarding this, please share a scaled-down collection that we can use to reproduce this issue?

[avicohh]

When using Newman as a library, the following can be done to detect an execution failure:

newman.run({
    collection: '...'
    // ...
}, function (err, summary) {
    if (err || summary.run.error || summary.run.failures.length) {
        process.exit(1);
    }
});

For CLI usage, as mentioned in #883 (comment), Newman CLI will exit with a non-zero code on errors.

If you are facing any issues regarding this, please share a scaled-down collection that we can use to reproduce this issue?

There is problem with adding process.exit(1), because then the process exits before the reports are done.
For now the only solution I found, is to add some timeout before the process.exit(1)

[pa-eps]

Same issue here. Using process.exit(1) quits too early without letting reports to be done. I'm not even using any additional reports; just cli reports. The test summary is not printed if use process.exit(1).

Please re-open this issue.

[pa-eps]

In case it helps anyone else here's how I worked around this issue:

newman.run(
  config,
  function (error, summary) {
    if (error || summary.error || summary.run.failures.length) {
      console.error('collection run encountered errors or test failures')
      process.exit(1)
    }
  })
  .on('start', function (err, args) {
    console.log('running postman collection with following configurations...\n',config)
  })

Maybe this is the intended usage but the readme in the repo has instructions for using done event, which seems to interfere with report generation.