Newman: Duplicate Tests results displayed in the HTML report, when using workflow to iterate through same request n times

[kharod]

Hey Guys,

Scenario and Request Setup:
I have written my postman request in a way that each request is running 4 times with different test scenarios. Using the workflow of the requests.

Request1 : Post
it has Test1
Test2
Test3
Test4
Test5 in it
at the end it calls itself to run with different set of data values (this happens for 4 values)

And I am using the Newman to run the tests from Teamcity. Test execution is fine, but the report that is generated makes things confusing.

Problem:
In the report that is generated, the requests that have looped on itself are shown that many times, but all the tests and their results for each loop are clubbed together and displayed in each test execution request.

So to say if the Request1 has 5 test in it and it looped 4 times on itself, then the report shows it 4 times, but each time, it shows 5 tests execution status repeated every time.

Report generated is like this:
Request1 :
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5

Request1 :
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5
Test1
Test2
Test3
Test4
Test5

...
it is displayed like this 5 times. Which is very lengthy and confusing.

Is there any way i can change the way the grouping is done for the report template file i am using, that will fix this issue? Either we can split it to display 5 times, but dont display each runs test status's in every request. Or we can make it club together all tests but display only once instead of showing it 5 times.

Thanks in advance.

[kunagpal]

@kharod Could you also share:

• The command / script used to run the collection
• The collection that causes this.
• The generated HTML report.
• The Newman and Node versions.

[kharod]

Hi @kunagpal,

Command Used:
newman run C:\Test\Test.postman_collection.json -e C:\Test\Dev.postman_environment.json --reporters cli,teamcity,html --reporter-html-template C:\Test\Custom_template.hbs --reporter-html-export C:\Test\tests-report.html

Collection:
Test.postman_collection.zip

HTML Report:
tests-report.zip

Newman & Node Version:

Let me know if you need any other information.

Thanks.

[Jignesh28]

I am having the exact same issue. I ran the collection which consists of 5 steps and ran 6 iterations with different data. All the 5 steps were repeated 6 times with the exact same stats in the html report.

[cchretien]

template-html-with-responses.zip
+1
newman 3.5.2
nodejs 6.11.3 or 8.1.4

Here's my sample template.

[shamasis]

I suggest creating a new reporter and/or a new template for the built in HTML reporter. After all, this was the whole purpose of exposing the easy to use reporter api. 😊

Closing this ticket.

[cchretien]

It's not a template issue. It's an issue with the aggregator. That's unfortunate that you won't support reporter fixes

[joshualyon]

@shamasis thanks for your feedback and taking the time to respond. I have to agree with others that this seems to be a bug. It fundamentally doesn't make sense for the report output to iterate n times over X requests and then show each request n times with the same aggregated data everytime. Was this really the expected output for the aggregations in the JSON object?

At the very least, could the default report show some examples that show how one would achieve something that makes more sense by default:

• Output iterates n times but shows data unique for each execution
• Output iterates 1 time for each request but shows data aggregated

My concern is the default JSON object makes either of these more difficult to achieve than they should be which begs the question why an aggregation would show n iterations in the first place? .

[peushan]

Has anyone found any alternative for duplication in the report? I also have one end point with different data (which I am passing from csv) but in report shows only the first data set in all 10 iterations. Thanks in Advance

[kunagpal]

@peushan The HTML reporter is meant to aggregate statistics across all iterations, multiple iterations showing up in generated reports is a bug that was fixed in https://github.com/postmanlabs/newman/pull/1553. This will become available with the next Newman release.

[shamasis]

@cchretien @joshualyon - I understand the disappointment. And I am completely with you on your points. But, we need a way to not keep on adding features to the internal HTML reporter. Over the course of time, newman has ended up being HTML reporter upgrades.

With such popularity of HTML reporter, I suggest we port out the HTML reporter and create a new one for advanced HTML reporting that is more community maintained.

Thoughts?

[cchretien]

Looks like this was addressed in the end #1553

[peushan]

@kunagpal Thanks heaps for the update.

[praveenkannambadi]

I am using newman Version 3.10.0 . The HTML reporter is not aggregating statistics across all iterations, multiple iterations. Can you please let me know if this is the expected behavior

[evgenyshandrik]

Hi @kharod!

Can you share a template (*.hbs) for report?