Newman: How to Stop NEWMAN when any assertion fails

[Sripathi1983]

1. Newman Version (can be found via newman -v):
2. OS details (type, version, and architecture):
3. Are you using Newman as a library, or via the CLI?
4. Did you encounter this recently, or has this bug always been there:
5. Expected behaviour:
6. Command / script used to run Newman:
7. Sample collection, and auxilliary files (minus the sensitive details):
8. Screenshots (if applicable):

[Sripathi1983]

Please let me know any command line option is available so that the postman collection stops getting executed when any in-between collection fails

[kunagpal]

@Sripathi1983 You can use the --bail option to stop collection runs on the first failure, like so:

newman run coll.json --bail

[Sripathi1983]

That works for only one collection, suppose two collections are getting executed by two newman commands one after the other and if first one fails then the second collection also should n't get executed as their is a failure in the first collection, is it possible

[Sripathi1983]

eg- first collection : newman run firstcoll.json --bail
second collection: newman run secondcoll.json --bail

the second collection shouldn't run as first collection there is assertion failure , so that overall result should be a failure

[kunagpal]

This can be accomplished via the CLI, or via programmatic usage:

CLI

newman run firstcoll.json --bail && newman run secondcoll.json --bail

Programatically

var async = require('async'), // https://www.npmjs.com/package/async
    newman = require('newman');

async.series([
  function (next) {
    newman.run({
      collection: 'firstcoll.json',
      bail: true
    }, next);
  },
  function (next) {
    newman.run({
      collection: 'secondcoll.json',
      bail: true
    }, next);
  }
], process.exit);

[Sripathi1983]

I'm integrating the newman with jenkins for build validation, cli part i got it, but when I want code to do the same functionality how can I execute the code when integrated with Jenkins as a javascript file or anything

[Sripathi1983]

can you please give complete list flags that is used for newman cli

[kunagpal]

To use Newman programmatically in your Jenkins setup, one of the steps in your test pipeline must include:

node newman.test.js // newman.test.js contains the sample code specified above ^

The complete list of flags is here: https://github.com/postmanlabs/newman#command-line-options

[kunagpal]

Closing issue on our end, feel free to reach out for any other suggestions or issues :smile:

[svenkat1]

Hi i am having an issue with newman integration. I am running newman parallely which will execute 5000+ test conditions, when i am using junit,html,cli reporter it running fine but in case i use json reporter means it throwing this error

1347905 ms: Mark-sweep 1226.7 (1422.5) -> 1216.6 (1424.5) MB, 539.5 / 0.0 ms [allocation failure] [GC in old space requested].
1348451 ms: Mark-sweep 1216.6 (1424.5) -> 1216.6 (1423.5) MB, 546.1 / 0.0 ms [allocation failure] [GC in old space requested].
1349035 ms: Mark-sweep 1216.6 (1423.5) -> 1216.6 (1378.5) MB, 583.8 / 0.0 ms [last resort gc].
1349607 ms: Mark-sweep 1216.6 (1378.5) -> 1216.6 (1363.5) MB, 572.2 / 0.0 ms [last resort gc].

If i execute with increased heap size it working fine by using the command --max-old-space-size=4000,
but this is not a good solution, in case if testcondition increased to some amount means then i want to increase the size while running every time it makes my system slower.

I tried garbage collector cleaning but it also failed. I tried to find the root cause in my code their is no issue,
the issue occurs only when i am using newman json reporter as it produces bulk files and its object size inceases. so kindly suggest me the solution to solve the issue.

[kunagpal]

@svenkat1 Although using --max-old-space-size is a quick workaround, a more lasting solution is to stream reports to files, as has been discussed here:

https://github.com/postmanlabs/newman/issues/935#issuecomment-284329866
https://github.com/postmanlabs/newman/issues/564#issuecomment-242437859

[svenkat1]

@kunagpal
I am declaring newman script like this
Example:-
newman.run({
collection: collectionFileName,
globals : globalsEnv,
environment : runtimeEnv,
reporters: ['json','junit'']
});

Here i declared json and junit in my reporters. Error occuring only when i give json in reporters, so while in runtime, newman produces json reports which is a bulk file and i execute newman parallely more than 20.

How can I add stream report in newman json reports?

[kunagpal]

@svenkat1 I was referring to streamed reports as an improvement within Newman, not something that can be enabled with an option. This is an important change, and we will keep you updated on any progress that happens on this.

[svenkat1]

@kunagpal Thank you for your response. once you made the changes let me update.

[mchippag]

Hi, I am trying to execute a Jenkins pipeline in which one stage is dedicated for newman run(using collection). I want to know if I can abort the build if my newman assertions are failed.

[michaelajr]

IMO there needs to be a newman verify command that can look at a summary report and exit with an error if there test run failures. We have newman run happening and we do not fail if there were test failures (need to do other things like dump a coverage report and stop the server) - so we need a way to programatically fail a pipeline after the test run exits. I suppose we could use the exit code of the run command. But that is hard to do in Maven if you want to bind to the verify goal. Anyway just a thought.