Nightwatch: Regression: Custom reporter not working for multiple test files execution

Created on 28 Sep 2016 · 21Comments · Source: nightwatchjs/nightwatch

This seems broken since v0.9.0 release. Custom reporter works with just one test file execution but doesn't work when executing a set of test files. This is for both approaches, --reporter and reporter method in external globals file.

I tried with v0.8.18 and it works fine for any test executions.

Its surprising that no one even reported this issue (does that mean nobody is using this feature? )

bug

Source

mdharamadas

👍3

Most helpful comment

@lreading we have a workaround in place that uses the global afterEach hook. The client parameter contains a currentTest property with which we update a json.

This is a rough example (might need some readaption to work correctly).

const filePath = 'some/path/to.json';

afterEach: (client, done) => {
    const { currentTest } = client;
    let results;

    try {
        results = require(filePath);
    } catch (e) {
        results = {};
    }

    const data = {
        name: currentTest.module,
        success: currentTest.results.errors === 0 && currentTest.results.failed === 0,
        errors: Object.entries(currentTest.results.testcases).reduce((errors, [testcaseName, testcase]) => {
            if (testcase.errors !== 0 || testcase.failed !== 0) {
                errors.push({
                    testcase: testcaseName,
                    assertions: testcase.assertions
                });
            }

            return errors;
        }, [])
    };

    results[data.name] = data;

    try {
        fs.writeFileSync(filePath, JSON.stringify(results), "utf8");
    } catch (e) {
        console.log(`An error occured while saving ${filePath}:`);
        console.log(e);
    }

    done()
};

aberonni on 17 Sep 2018

🎉2 ❤1

All 21 comments

@mdharamadas - How is the custom reporter broken? What is the command line or settings you're using? What happens in 0.8.18? What happens in 0.9.0?

bitwiseman on 29 Sep 2016

@bitwiseman pls see the required details below...

nightwatch.json

{
  "src_folders": ["tests"],
  "output_folder": "reports",
  "custom_commands_path": [],
  "custom_assertions_path": [],
  "page_objects_path": "lib/page/",
  "globals_path": "./globalsModule.js",
  "live_output": false,

  "test_workers": {
    "enabled": true,
    "workers": "auto"
  },

  "test_settings": {
    "default": {
      "skip_testcases_on_fail": false,
      "end_session_on_fail": true,
      "silent": true,
      "screenshots": {
        "enabled": true,
        "on_failure": true,
        "path": "screenshots"
      },
      "selenium": {
        "log_path": "log/"
      },
      "desiredCapabilities": {
        "browserName": "chrome",
        "javascriptEnabled": true,
        "acceptSslCerts": true
      }
    },
    "chrome": {
      "launch_url": "http://localhost",
      "selenium_port": 4444,
      "selenium_host": "localhost",
      "selenium": {
        "start_process": true,
        "server_path": "",
        "host": "127.0.0.1",
        "port": 4444,
        "cli_args": {
          "webdriver.chrome.driver": ""
        }
      }
    }
  }
}

In globalsModule.js, I have reporter funtion defined as follows...

...
  reporter: function (results, done) {
    console.log('custom reporting');
    done();
  }
...

When I run single test file using following command line, I see that it prints 'custom reporting' in the test execution logs.
./nightwatch --config ./nightwatch.conf.js -e chrome --test tests/sample.js

But, when I run all the test files using following command line, I DO NOT see 'custom reporting' printed in the test execution logs.
./nightwatch --config ./nightwatch.conf.js -e chrome

Both of these commands work as expected (prints 'custom reporting' in logs) with Nightwatch v0.8.18.

Same behaviour if I have separate reporter module defined in a separate file and call it using command line option --reporter.

Let me know if you need any further details or if anythings not clear.

mdharamadas on 29 Sep 2016

@beatfactor can you take a look at this one pls? thanks!

mdharamadas on 3 Oct 2016

Same issue for me, that'd help a lot! Thanks!

Bobu77 on 19 Dec 2016

I can confirm this is still an issue on 0.9.16

termleech on 29 Jul 2017

xeppen on 10 Aug 2017

Was browsing through issues to find another issue and stumbled upon this.

Guess mine is a duplicate, but I turned it into a feature request instead of a bug report as I was not aware this is a regression.

https://github.com/nightwatchjs/nightwatch/issues/1557

StephanBijzitter on 28 Aug 2017

We have digged into the code a little and found out the following:

The Reporter-class is instantiated in two places: lib/runner/run.js and lib/runner/cli/clirunner.js
The "options"-parameter is different in the two places and we assume that the lib/runner/cli/clirunner.js is called when running tests in a test-worker.
Our custom reporter uses data from the globals.test_settings, which are only part of the options if the reporter is not called from a test-worker. This leads to an exception and the report-file is not written.

The correct solution in my opinion would be, not to call the reporter at all from the test-workers. Instead the worker should report back the results to the main-process. The main-process should merge the results and then call the reporter.

nknapp on 13 Oct 2017

+1. Able to reproduce in 0.9.19

Rajesh-Narayanappa87 on 11 Jan 2018

👍1

I am able to write to a file from custom reporter but console.log doesn't work if custom reporter is defined in globalsModule.js.
nightwatch version 0.9.20

ankiit on 11 Apr 2018

Hi @beatfactor can you give an update on this?

aberonni on 6 Jul 2018

@aberonni have you tested it in the latest v1.0.6?

beatfactor on 6 Jul 2018

@beatfactor reproduced also with v1.0.6

You can see the output with only 1 test result instead of 2 here, and reproduce in this repo.

aberonni on 12 Jul 2018

@beatfactor could you add the v1.0 tag to this issue?

Also let me know when the latest code is pushed and I will test again against the latest code.

aberonni on 17 Jul 2018

Has anyone found a workaround for the interim?

lreading on 16 Sep 2018

@lreading we have a workaround in place that uses the global afterEach hook. The client parameter contains a currentTest property with which we update a json.

This is a rough example (might need some readaption to work correctly).

const filePath = 'some/path/to.json';

afterEach: (client, done) => {
    const { currentTest } = client;
    let results;

    try {
        results = require(filePath);
    } catch (e) {
        results = {};
    }

    const data = {
        name: currentTest.module,
        success: currentTest.results.errors === 0 && currentTest.results.failed === 0,
        errors: Object.entries(currentTest.results.testcases).reduce((errors, [testcaseName, testcase]) => {
            if (testcase.errors !== 0 || testcase.failed !== 0) {
                errors.push({
                    testcase: testcaseName,
                    assertions: testcase.assertions
                });
            }

            return errors;
        }, [])
    };

    results[data.name] = data;

    try {
        fs.writeFileSync(filePath, JSON.stringify(results), "utf8");
    } catch (e) {
        console.log(`An error occured while saving ${filePath}:`);
        console.log(e);
    }

    done()
};

aberonni on 17 Sep 2018

🎉2 ❤1

Thanks for the detailed response!

lreading on 17 Sep 2018

This issue has been automatically marked as stale because it has not had any recent activity.
If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.

stale[bot] on 16 Dec 2018

Not stale

aberonni on 17 Dec 2018

@beatfactor looks like this can be closed now that the PR has been merged and released

aberonni on 10 Feb 2019

Still experiencing this on v1.3.2. The results object that is passed into the write() method inside of the file specified with the --reporter flag is empty when running more than 1 test file.