Newman: Failing tests exit successfully

[bishtawi]

I have newman v3.0.0 installed globally. I am executing newman from the commandline using:
newman run ContractsAPI.postman_collection -e DEV_QA_User.postman_environment

I get this assertion failure:

  #  failure                         detail                                                                                                                       

 1⠄  AssertionFailure                Response time is less than 200ms                                                                                             
                                     at assertion:1 in test-script                                                                                                
                                     inside "Contracts - Valid contract commit." of "Contracts APIs"   

But newman exits with the successful exit code of 0. I am not passing in -x, --suppress-exit-code, or --bail. I would expect a failing exit code (like 1).

[mordi34]

@kunagpal Do you have any idea when your fix will be released?

[kunagpal]

Hey @mordi34 @bishtawi, a fix has been created here, it will become available in the next release. I'll update this thread with the ETA for the same when ready.

[tojagrut]

I faced the same problem. As an additional feature, I am generating xml report (junit one) and using Junit parser to parse it which fails the build in case of any failed test cases. This is pretty useful when you want to know only the failed test cases out of thousands of total test cases.

[kunagpal]

This issue should have been fixed in Newman v3.0.1, do post here if that is not the case.

[bishtawi]

Works for me, thanks!

[mordi34]

Me too, thanks

[kunagpal]

Marking the issue as resolved.

[arashidkargo]

Currently Running newman 3.6.1 . When executing newman run https://www.getpostman.com/collections/265d94c75d9bc5cf8976

I get following assertion error: 1. AssertionFailure Response time is less than 1500ms
at assertion:4 in test-script
inside "DEV-Request / DEV:Post-Create Brand" of "DEV-Request"

1. AssertionFailure Response time is less than 500ms
   at assertion:4 in test-script
   inside "DEV-Request / DEV:Put-Change Brand" of "DEV-Request"

2. AssertionFailure Response time is acceptable
   at assertion:5 in test-script
   inside "DEV-Request / DEV:Put-Change Brand" of "DEV-Request"

[kunagpal]

@arashidkargo Please test this out with Newman v3.8.0 and create a new issue here if this behaviour persists.

[arashidkargo]

Hey Kunagpal It's working in V3.8.0. Thanks....