Newman: Random test failures between runs: getaddrinfo ENOTFOUND for request leads to child test failures and double counting.

[dojones1]

1. Newman Version (can be found via newman -v): 4.2.2-alpine

2. OS details (type, version, and architecture): Centos x86 7.6.1810

3. Are you using Newman as a library, or via the CLI?: Via docker container

   3a. Did you encounter this recently, or has this bug always been there:
   Recently. CI driven test runs were stable (Ran every 5 minutes for a week without issue)
   Now persistently and randomly unstable between runs.

   • No changes to test cases or environment.
   • Tests can be run from desktop without issue.

4. Expected behaviour:
   All tests pass

5. Command / script used to run Newman:
   docker run --rm --volume="${PWD}":/etc/postman -t /postman/newman:4.2.2-alpine run "/etc/postman/api_endpoints.postman_collection.json" --environment="/etc/postman/prod.postman_environment.json" --reporters junit --report-junit-export="/etc/post/api_endpoints.xml" --insecure --suppress-exit-code
   docker run --rm --volume="${PWD}":/etc/postman -t /postman/newman:4.2.2-alpine run "/etc/postman/proxy_endpoints.postman_collection.json" --environment="/etc/postman/prod.postman_environment.json" --reporters junit --report-junit-export="/etc/post/proxy_endpoints.xml" --insecure --suppress-exit-code

6. Sample collection, and auxiliary files (minus the sensitive details):
   Can't provide these unfortunately. However, the tests are all very simple and of the form:
   Request: GET
   pm.test("title"), function() {
   pm.response.to.have.status(401);
   });

pm.test("title"), function() {
pm.expect(pm.response.responseTime).to.be.below(200);
});

I have 113 tests between the two collections, and different tests fail randomly between runs.
Jenkins reports:
62 failures out of 133
45 failures out of 129
21 failures out of 122

Note: The number of tests has not changed between runs, so there is an issue with the junit XML generation - see Scenario 1 below.

Several scenarios seen:

1 - dns failure)
Always seen in: ./
Iteration: 0
RequestError: Error: getaddrinfo ENOTFOUND :
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:57:26)

The number of these *. failures account for the over counting in the xml output*

For cases where 1 occurs then the child tests will all fail with either scenario 2 or scenario 3 below:

2 - response UNDEFINED)
Failed 1 times.
Collection JSON ID:
Collection name:
Request name: / / .
Test description: Response must be valid and have a body
Error message: expected response to have status reason 'OK' but got 'UNDEFINED'.
Stacktrace: AssertionError: expected response to have status reason 'OK' but got 'UNDEFINED' at Object.eval sandbox-script.js:1:3).

3 - response time undefined)
Failed 1 times.
Collection JSON ID:
Collection name:
Request name: / / .
Test description: Response time is less than 200ms.
Error message: expected undefined to be a number or a date.
Stacktrace: AssertionError: expected undefined to be a number or a date at Object.eval sandbox-script.js:2:3).

[dojones1]

Issue found to be due to rogue entries in my /etc/resolv.conf file

[shamasis]

@codenirvana - maybe a debug mode in future that outputs where the DNS was resolved from. 😛😛