Newman: HTML report generated shows the same value for all iterations when using a data file with multiple data sets

[rohitparethodi]

1. Newman Version : 3.9.3

   1. OS details : Windows 8 64 bit

   2. Are you using Newman as a library, or via the CLI? Node module and CLI. Both have the same issue

   3. Did you encounter this recently, or has this bug always been there: I encountered recently. Could have always been there

   4. Expected behaviour: the results for different data iterations on a collection should show up in the html report generated

   5. Command / script used to run Newman: newman run collectionname -e environment -d data -k

   6. Sample collection, and auxiliary files (minus the sensitive details): any collection would do as long as we use a data file with multiple data sets

   7. Screenshots (if applicable):N/A

[chrisron95]

I believe I have the same issue. I have to reuse the same call multiple times throughout my run, and if on one iteration of that call there's an error, each iteration of that call will show that it failed in the report. I am using the Newman library but yes, this applies to that as well as directly in the CLI. This has been occurring as long as I've been using Newman (a couple weeks).

[kunagpal]

@rohitparethodi @chrisron95 This is intended behaviour, as the HTML report is meant to aggregate statistics across multiple runs. As for the request displayed in the markup, this is picked up from the first occurrence of that request, which would be the one from the first iteration.

[rohitparethodi]

@kunagpal be that as it may, the report has separate entries for those iterations and the end result is misleading. This behaviour has made the html report almost unusable for my data driven tests

[Patrik9]

Yes, the template is not usable for multiple iterations. Do you plan to add tabs for example for multiple iterations of a same test? Thaaaat would be amazing. Otherwise good job on the template but I need to find something else because I need report with multiple iterations.

[kunagpal]

@rohitparethodi That was due to a bug, which was fixed in https://github.com/postmanlabs/newman/pull/1553. This will be out with the next Newman release.

@Patrik9 That is a good suggestion, we could consider adding that in a future release.

[kunagpal]

@rohitparethodi This has been fixed in Newman v3.10.0 😄

[chrisron95]

In my opinion it's now worse than it was before the fix. Now each iteration is just combined into the first iteration of the call. So in my case, when I'm checking the status of the order and using delays and repeating the order status call until it's complete, if one of those order status calls has an error it's all combined into one because now any call that's the same as another gets mashed into one entry in the report. This gets confusing as there are multiple orders in my tests and I use the same order status call, but it only shows in the first spot. If there's an error, this makes it very difficult to tell which area the error occurred. At least before it would show each iteration of the calls where they were in the run. There should be an iteration node for each call or something so that we can have each iteration shown separately with their own pass/fail status and all that good stuff.

[kunagpal]

@chrisron95 We're looking at ways to improve the arrangement of information in line with your suggestions above, which will be taken up as a part of the Newman v4 line of releases.