Newman: Incomplete junit report when having nested requests/tests

[gkaiser85]

1. Newman Version (can be found via newman -v):
   3.9.3
2. OS details (type, version, and architecture):
   macOS 10.13.3
3. Are you using Newman as a library, or via the CLI?
   CLI
4. Did you encounter this recently, or has this bug always been there: -
5. Expected behaviour:
   The junit reporter should include all assertions in a report (including those of nested requests). The CLI reporter and the postman app are able to show all assertions.
6. Command / script used to run Newman:
   newman run nestedRequests.postman_collection.json -r cli,junit
7. Sample collection, and auxiliary files (minus the sensitive details):
   https://www.getpostman.com/collections/71794c3e8d7dcc3c1782
8. Screenshots (if applicable): -

Steps to reproduce the problem:

1. Download & run the collection (see above 7. & 6.)
2. Check the junit result stored in ./newman

Details:
For a project I have to execute a POST request to queue a job (where I get a UUID as response). Then I have to send a GET request, using the UUID, to retrieve more details on the queued job.
I don't want to copy/paste the same GET request over and over in my collection and therefore I have implemented a nested request within the test script of a folder (and therefore for multiple requests). This works perfectly but when I try to get junit reports via newman, the xml does not include the assertions for nested requests.

The sample collection has two requests:

• The first is a simple request having a pm.test() defined as test-script.
• The second has a nested request pm.sendRequest() and a related pm.test() as test-script.

[kunagpal]

@gkaiser85 Thanks for reporting this, I was able to reproduce the described condition with the collection you provided. I'll update this thread when any progress is made. Cheers! 😄

[decu69]

Are there any news when this would be fixed? I lost couple of hours finding that my script is correct but junit reporter can`t interpret such request in a correct way.

[decu69]

The true is that the issue is not about nested request. The issue could be easily repeated by using pm.sendRequest in Tests - in that case, junit reporter would ignore the result, even if you have pm.test() somewhere in your test definition.

[joshnorwich]

On top of JUnit not reporting pm.test calls nested inside pm.sendRequest, it also seems to ignore reporting on pm.test in Collection or Folder level scripts on the collection. Rather than have my "test for status 200" script on each individual request I use a Collection level script. Now my JUnit report in Jenkins seems to think there are only 5 tests in the collection, despite the CLI report showing over 100 assertions being tested.

[musanas]

+1 for fixing this issue
Until this is fixed, there is no point in hitting on the same request with multiple test data if you are relying only on junit report ( as that can be accomplished only by doing postman.setNextRequest and then passing the same request name as the argument to it).

and if the second set of data supplied to the request has resulted in failure, the jUnit report will silently ignore it and will not show up, (though the newman cli report/html report will show it up). This is bad.

[musanas]

@joshnorwich
I do not face the issue that you have mentioned above, the tests added at the folder level is being considered in the Junit report as expected.
Newman version used 3.9.6
Thanks

[ThisIsABug]

The true is that the issue is not about nested request. The issue could be easily repeated by using pm.sendRequest in Tests - in that case, junit reporter would ignore the result, even if you have pm.test() somewhere in your test definition.

Actually this is exactly the issue. Every request which holds a nested request created with pm.sendRequest fails to output a result. If the nested request is in a folder, every test in the whole folder will end up creating an empty testsuite <testsuite/>. To work around that problem we currently have a dedicated request without nesting, but it messes up our collections pretty much, additionally an entry in the list cannot run on its own now, it has to be executed in the right order eg. request1=open session, request2=read data, request3=close session (request1 and 3 should be in the pre-script/test of a folder above request2).