Nightwatch: Test run is aborted and report not saved on selenium server error.

[michal-ra]

Description

Test execution is aborted and XML report is not generated on HTTP 500 error on POST to selenium server.

Tests are runnig fine unltil click() command, which returns HTTP 500 (element not interactable).
Even if aborted test run is expected behavor, I would still expect to see a test report.

It looks like nightwatch quits unexpectedly as even retry is not perfomed - you can see that log ends at (retry 1) and there is nothing more in output.

Sample test

sampleTest.js

module.exports = {
  'sample': function test() {
      // arrange
      this.unitUnderTest = page();
      this.unitUnderTest.navigate();
      this.unitUnderTest.api.expect.url().to.contain(this.unitUnderTest.url);

      // act
      this.unitUnderTest.section.TopTabs.click('@menu');
      this.unitUnderTest.section.sectionName.click('@SelectAll');

      // assert
      this.unitUnderTest.section.sectionName.expect.element('@element1').to.be.selected;
      this.unitUnderTest.section.sectionName.expect.element('@element2').to.be.selected;
      this.unitUnderTest.section.sectionName.expect.element('@element3').to.be.selected;
      this.unitUnderTest.section.sectionName.expect.element('@SelectAll').to.have.attribute('data-is-active').equals('false');
      this.unitUnderTest.section.sectionName.expect.element('@Download').to.be.enabled;
  },
}

Run with command
No custom args used.

$ nightwatch test/sampleTest.js

Verbose output

debug.log

Request POST  /wd/hub/session/909f8f3189148525809814cf63951b8e/element/7ec37ff1-8cb6-45bd-8206-01281620ace6/elements  

   { using: 'css selector', value: 'label[for=elemLabel]' }

   Response 200 POST /wd/hub/session/909f8f3189148525809814cf63951b8e/element/7ec37ff1-8cb6-45bd-8206-01281620ace6/elements (49ms)

   {

     sessionId: null,

     state: 'success',

     value: [

       {

         'element-6066-11e4-a52e-4f735466cecf': '4dd993fd-b884-42ae-be50-b7f116d1d801'

       }

     ],

     status: 0

}

   Request POST  /wd/hub/session/909f8f3189148525809814cf63951b8e/element/4dd993fd-b884-42ae-be50-b7f116d1d801/click  

{}

   Response 500 POST /wd/hub/session/909f8f3189148525809814cf63951b8e/element/4dd993fd-b884-42ae-be50-b7f116d1d801/click (27ms)

   {

     sessionId: null,

     state: 'element not interactable',

     value: {

       systemInformation: "System info: host: 'xxxx-xxxx-6x4qw-nnp6r', ip: 'xxx.xxx.xxx.xxx', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.0-11-amd64', java.version: '1.8.0_275'",

       message: 'element not interactable',

       error: [

         '  (Session info: headless chrome=87.0.4280.88)',

         "Build info: version: '4.0.0-beta-1', revision: '1f4909f59c*'",

         "System info: host: 'xxxx-xxxx-6x4qw-nnp6r', ip: 'xxx.xxx.xxx.xxx', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.0-11-amd64', java.version: '1.8.0_275'",

         'Driver info: driver.version: unknown'

       ]

     },

     status: 60

}

   Request POST  /wd/hub/session/909f8f3189148525809814cf63951b8e/element/4dd993fd-b884-42ae-be50-b7f116d1d801/click  (retry 1) 

{}

Done in 116.47s.

Configuration

nightwatch.conf.js

const path = require('path');
const chromedriver = require('chromedriver');
require('dotenv').config({ silent: true });

const remote = {
  selenium: {
    start_process: false,
    host: 'localhost',
    port: 4444,
    ssl: false,
    keep_alive: {
      enabled: true,
      keepAliveMsecs: 4000
    },
    timeout_options: {
      timeout: 15000,
      retry_attempts: 5
    },
  },
  desiredCapabilities: {
    chromeOptions: {
      args: ['no-sandbox', 'headless', 'window-size=1920,1080', 'disable-gpu']
    }
  },
  chrome: {
    desiredCapabilities: {
      chromeOptions: {
        args: ['no-sandbox', 'headless', 'window-size=1920,1080', 'disable-gpu']
      }
    }
  },
};

const env1 = {
  ...remote,
  launch_url: `URL`,
  loginPath: `LOGIN`,
};

const env2 = {
  ...remote,
  launch_url: `URL`,
  loginPath: `LOGIN`,
};

module.exports = {
  test_settings: {
    default: {
      src_folders: ['scenarios/'],
      page_objects_path: ['pages'],
      custom_assertions_path: 'customAssertions',

      skip_testcases_on_fail: false,
      globals: {
        waitForConditionTimeout: 5000,
        abortOnElementLocateError: false,
        abortOnAssertionFailure: false,
      },

      test_workers: {
        enabled: true,
        workers: 'auto',
      },

      desiredCapabilities: {
        browserName: 'chrome',
        w3c: false,
        acceptInsecureCerts: true,
        chromeOptions: {
          args: ['no-sandbox', 'ignore-certificate-errors']
        }
      },
      chrome: {
        desiredCapabilities: {
          browserName: 'chrome',
          w3c: false,
          acceptInsecureCerts: true,
          chromeOptions: {
            args: ['no-sandbox', 'ignore-certificate-errors']
          }
        }
      },

      screenshots: {
        enabled: true,
        on_failure: true,
        on_error: true,
        path: './tests_output/screenshots',
      },
      output_folder: './tests_output'
    },

    local: {
      loginPath: `LOGIN`,
      launch_url: 'URL',

      webdriver: {
        start_process: true,
        server_path: chromedriver.path,
        port: 4444,
        cli_args: ['--port=4444']
      },
    },

    env1,
    env2
  },
};

Environment

Tests are executed on Kubernettes using Jenkins.
Container hosting browser is spawned from image from selenium - standalone-chrome:87.0

| Executable | Version |
| ---: | :--- |
| nightwatch --version | 1.5.1 |
| npm --version | 6.13.4 |
| yarn --version | 1.22.4 |
| node --version | v12.14.1 |

| Browser driver | Version |
| --- | --- |
| NAME | VERSION |
| selenium-server | selenium/standalone-chrome:87.0 |

[AndreiVasileUTM]

I have this issue also, any ETA for the fix?

[beatfactor]

It is very difficult to reproduce this error and any help with a specific case would be appreciated. The sample test which was provided is not very helpful as it is referencing various page objects and/or sections.

[michal-ra]

It all boils down to forcing the HTTP 500 from selenium server, so the easiest way to reproduce it is to p.ex. click some disabled control.
Then you should get this:

Response 500 POST /wd/hub/session/2aa7b0d1100a8c93dde5f54fb17d40b0/element/286b6ce8-c1e2-48a8-9c1f-d3877b6d4932/click (28ms)

   {

     state: 'element not interactable',

     sessionId: null,

     value: {

       systemInformation: "System info: host: 'e2e-tests-xx1cw-7vb18', ip: 'x.x.x.x', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.0-11-amd64', java.version: '1.8.0_275'",

       message: 'element not interactable',

       error: [

         '  (Session info: headless chrome=87.0.4280.88)',

         "Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'",

         "System info: host: 'e2e-tests-xx1cw-7vb18', ip: 'x.x.x.x', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.0-11-amd64', java.version: '1.8.0_275'",

         'Driver info: driver.version: unknown'

       ]

     },

     status: 60

[beatfactor]

I'm afraid I can't reproduce this. I can easily simulate the error by clicking on the disabled input at this demo url hosted on the nightwatch website, but the error is handled accordingly (test runner exits, report is written etc.). I have tried using both a local selenium server and browserstack, with chrome and firefox.

Can you provide more details and a full debug log? I don't have Kubernetes, but I could try using docker locally if you can provide some details on which containers to run.

[michal-ra]

Yup, locally there is no problem at all.
I didn't try to reproduce it on docker locally, but hope it would work. So my setup is two containers:

1. official node image which has Nightwtch
2. selenium server - standalone

First container has tests and Nightwatch on it and connects to second one to open tested application which is hosted on separate server.
Hope it helps 🤞