Newman: Html report is not generated with the new version 3.9.X

[EstherSalinas]

Html report is not generated with the new version 3.9.X
Since I've update to the new version, the report in html is not generated. Whith the version 3.8.3 it works fine.

Version and environment information:
-->

1. Newman Version: 3.9.X
2. OS details (type, version, and architecture): Windows
3. Are you using Newman as a library, or via the CLI? as a library
4. Did you encounter this recently, or has this bug always been there: recently, since I've install the new version of newman 3.9.X. With the version 3.8.3 it works fine.
5. Expected behaviour: create report
6. Command / script used to run Newman:

#!/bin/bash
ENVIRONMENT=$1
TEST=$2
FILE=$3
newman run collections/${FILE} --disable-unicode --color --insecure -x -e environments/${ENVIRONMENT}.json --timeout-request 80000 --reporters cli,html,json,junit --reporter-json-export output/report-${ENVIRONMENT}-${TEST}.json --reporter-junit-export output/report-${ENVIRONMENT}-${TEST}.xml --reporter-html-export output/report-${ENVIRONMENT}-${TEST}.html
cat output/report-${ENVIRONMENT}-${TEST}.html |grep Assertions | sed -e 's/<[^>]*>/ /g' | awk '{print $1" " ($2-$3)*100 / $2 "%  "$2" "$3 }'

[kunagpal]

@EstherSalinas I was unable to reproduce this with Newman v3.9.1 on Node v6.12.2. Could you share the following details as well?

1. The node version used (node -v)
2. The underlying operating system.
3. The generated report at output/report-${ENVIRONMENT}-${TEST}.html, as well as the result of your grep Assertions command chain.

[EstherSalinas]

Hello @kunagpal,

I've tested in my local machine (Windows) and in the CI server where I run the tests.
Below is the information.

1. The node version: 6.9.1 (local), 7.5.0 (docker container node:7.5.0)
   Note: I've tested in CI server with newman v3.9.1 on Node v6.12.2 as you have mentioned and the report was not generated.
2. Operating system: Windows 8.1 Enterprise in local and Debian “wheezy" in the docker container where I run the test.
3. The generated report (With newman 3.8.3):
   report-int-3rd.zip
   I also tested without the grep assertions but the report was not generated.

I also tested another collection with 3.9.X and the result was generated properly with node 6.9.1. This is the result of the collection that is generated properly with both versions of newman:
report-int-backend.zip
I used the same bash script.

Thank you in advance,

Esther.

[benjaminmmartin]

@kunagpal I have been able to reproduce this issue with the following environment:
Docker 17.09.1-ce
Base alpine:latest
Node v6.10.3
Newman v3.9.1 and v3.9.0
Neither html or json reporters appear to be working correctly.

Switching back to Newman v3.8.3 is a current workaround.

[termleech]

I wanted to add a little more information to this. I just upgraded from Newman 3.8.3 to 3.9.1 as I had written tests and using the junit reporter along with the cli, the result.xml file reported 0 tests for each of the items in the collection. Reverting back to 3.8.3 and everything is working and the result.xml is generated properly. I'm using the latest Node LTS and have verified this both on my Mac as well as our Jenkins instances running on RHEL 7.

[Mohammad-sabbagh]

@kunagpal I am able to reproduce this issue using HTML and Junit reporters but it happens intermittently with the following environment:
Node v8.11.1
Newman v3.9.3
OS: Windows 10

[kunagpal]

@EstherSalinas @Mohammad-sabbagh @termleech Could you check if this persists with Newman v3.9.4?

@benjamin-martin987 Your issue appears to be related to extracting reports from within a Docker container to outside of it. Could you take a look at https://github.com/postmanlabs/newman/issues/1522#issuecomment-393125766?

[Mohammad-sabbagh]

@kunagpal yes still can replicate this on Newman v3.9.4

[benjaminmmartin]

@kunagpal #1522 is not related. The reports are to be written to a directory local to the Docker container. No volume mounting.

Newman v3.9.4 still contains the issue.

[Kal6]

I am also facing this issue, xml report is getting generated with no details and 1kb size of file. mentiond below the file output. however html report is getting generated.
Reports.zip

[benjaminmmartin]

I had a thought, checked the documentation, and identified the cause (in my case anyway).

As per the Newman documentation, all options are to be listed after the collection:
newman run <collection-file-source> [options]
Stating the desired reporters before the collection will be ignored by versions after v3.8.3.

Works with Newman v3.8.3 and earlier:
newman run -r cli,junit my_collection.postman_collection.json

Works with Newman v3.9.4:
newman run my_collection.postman_collection.json -r cli,junit

The findings above have been tested in the following environments:

macOS High Sierra v10.13.4
Node v8.9.1
Newman v3.9.4

Docker 18.03.1-ce
Base centos7
Node v7.10.1
Newman v3.9.4

[Kal6]

@benjamin-martin987 - Thanks for your reply, I have used the below command line in the same way as you mentioned. But still xml is getting generated with no results (1kb file). I have multiple folders for different API tests, So i used to folder as param in the command line to pick that particular folder tests to run.

newman run C:\Users\kkaku\Documents\Fulfillment\Framework\OOF_V1.postman_collection.json --folder "OOF_WO_008_PrepLocations_AcknowledgeOrder" -r html,cli,junit -g C:\Users\kkaku\Documents\Fulfillment\Framework\MyWorkspace.postman_globals -e C:\Users\kkaku\Documents\Fulfillment\Framework\Cert.postman_environment.json -d C:\Users\kkaku\Documents\Fulfillment\Framework\OOF_WO_008_PrepLocations_AcknowledgeOrder_TestData.csv.

i noticed that i had newman 3.9.3 older version and i did update to 3.9.4 but still no LUCK, I am not sure where i commit to mistakes.

OS: Windows 8.1
newman -v
3.9.4
node -v
v8.11.1

Greatly appreciated for your help!

[p00j4]

I'm using with command line and HTML report is not getting generated for me as well
Is the fix broken now? What are the compatible versions?

I'm using following
Version info

newman 4.3.1
node v11.13.0
OS: ubuntu 14.x

[DannyDainton]

Hey @p00j4

Just wanted to get some more information on what is happening for you.

• What is the command that you're running?
• Are you seeing an error in the console when running the Newman command?
• Are you able to post a screenshot of what's you're seeing?
• Have you specified a location for the report or is it going to the default /newman directory?

[p00j4]

Thanks @DannyDainton for quick response. I have got 3.10.0 version working and moving back to it. However the problem is reproducible in 4.3.1 version.

Here is a sample to reproduce

1. Install newman version 4.3.1 (sudo npm install -g [email protected])
2. Run on commandline

newman run https://www.getpostman.com/collections/0c1d3a3a42e6cd3d25cd --reporters cli,html,junit

1. check newman folder, it has only the xml. Html is missing.
   

[DannyDainton]

Did you also install the html reporter globally with V4 of Newman?

npm i -g newman-reporter-html

The html reporter was separated out into its own project in V4 so you would also need to install this in order to use that reporter.

Would you be able to try that please and see if that works?

[p00j4]

Oh yes, this works. Thanks @DannyDainton
What was the right way to figure this out on my own apart from needing to read the release notes?

[DannyDainton]

Awesome, glad it solved the problem.

There is also a section on the Newman README that references then need to install this reporter separately. This doesn't, however, mention that you would need to do this for V4. That's something that can be updated.

https://github.com/postmanlabs/newman#html-reporter

Just as a personal promotion thing and it's obviously your choice - There is another version of the HTML reporter that I maintain, which has a few additional features in there. 😄

https://github.com/DannyDainton/newman-reporter-htmlextra