Nightwatch: protocol error while running .isElementDisplayed() with chrome 75 on W3C mode

[individual-it]

Reporting a bug

Since running selenium with Chrome 75 the W3C is set as default.
This causes nightwatch not being able to get the display state of the elements

seems to be the same issue in https://github.com/nightwatchjs/nightwatch/issues/2109#issuecomment-499691751

Error while running .isElementDisplayed() protocol action: TypeError: Error while trying to create HTTP request for "/wd/hub/session/c23595696c27f9c2e67f604a4499f188/element/[object Object]/displayed": Request path contains unescaped characters
[webui-acceptance-tests:L24:4s]     at new ClientRequest (_http_client.js:137:13)
[webui-acceptance-tests:L25:4s]     at Object.request (http.js:39:10)
[webui-acceptance-tests:L26:4s]     at HttpRequest.createHttpRequest (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/http/request.js:112:55)
[webui-acceptance-tests:L27:4s]     at HttpRequest.send (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/http/request.js:191:29)
[webui-acceptance-tests:L28:4s]     at Promise (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/transport/transport.js:189:15)
[webui-acceptance-tests:L29:4s]     at new Promise (<anonymous>)
[webui-acceptance-tests:L30:4s]     at Selenium2Protocol.sendProtocolAction (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/transport/transport.js:187:12)
[webui-acceptance-tests:L31:4s]     at Selenium2Protocol.runProtocolAction (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/transport/jsonwire.js:61:17)
[webui-acceptance-tests:L32:4s]     at Object.isElementDisplayed (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/transport/actions.js:54:10)
[webui-acceptance-tests:L33:4s]     at Selenium2Protocol.executeProtocolAction (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/transport/transport.js:235:48)
[webui-acceptance-tests:L34:4s]     at WaitForElementVisible.executeProtocolAction (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/element/command.js:201:27)
[webui-acceptance-tests:L35:4s]     at WaitForElementVisible.protocolAction (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/api/element-commands/_waitForDisplayed.js:5:17)
[webui-acceptance-tests:L36:4s]     at WaitForElementVisible.elementFound (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/element/command.js:168:17)
[webui-acceptance-tests:L37:4s]     at action (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/api/element-commands/_waitForDisplayed.js:40:36)
[webui-acceptance-tests:L38:4s]     at PeriodicPromise.perform (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/util/periodic-promise.js:79:29)
[webui-acceptance-tests:L39:4s]     at PeriodicPromise.runAction (/var/www/owncloud/phoenix/node_modules/nightwatch/lib/util/periodic-promise.js:48:33)
[webui-acceptance-tests:L40:4s] 

Config:

const chromedriver = require('chromedriver')
const LOCAL_LAUNCH_URL = process.env.SERVER_HOST || 'http://localhost:8300'
let LOCAL_BACKEND_URL = process.env.BACKEND_HOST || 'http://localhost:8080'
const BACKEND_ADMIN_USERNAME = process.env.BACKEND_USERNAME || 'admin'
const BACKEND_ADMIN_PASSWORD = process.env.BACKEND_PASSWORD || 'admin'
LOCAL_BACKEND_URL = LOCAL_BACKEND_URL.startsWith('http') ? LOCAL_BACKEND_URL : 'http://' + LOCAL_BACKEND_URL
const SELENIUM_HOST = process.env.SELENIUM_HOST || ''
const SELENIUM_PORT = process.env.SELENIUM_PORT || 4445
const START_PROCESS = (SELENIUM_HOST === '')
const FILES_FOR_UPLOAD = process.env.FILES_FOR_UPLOAD || require('path').join(__dirname, '/tests/acceptance/filesForUpload/')

module.exports = {
  page_objects_path: './tests/acceptance/pageObjects',
  custom_commands_path: './tests/acceptance/customCommands',
  test_settings: {
    default: {
      globals: {
        waitForConditionTimeout: 10000,
        waitForConditionPollInterval: 10,
        filesForUpload: FILES_FOR_UPLOAD
      }
    },
    local: {
      launch_url: LOCAL_LAUNCH_URL,
      globals: {
        backend_url: LOCAL_BACKEND_URL,
        backend_admin_username: BACKEND_ADMIN_USERNAME,
        backend_admin_password: BACKEND_ADMIN_PASSWORD
      },
      selenium_host: SELENIUM_HOST,
      webdriver: {
        start_process: START_PROCESS,
        server_path: chromedriver.path,
        port: SELENIUM_PORT,
        cli_args: ['--port=' + SELENIUM_PORT]
      },
      desiredCapabilities: {
        browserName: 'chrome',
        javascriptEnabled: true,
        acceptSslCerts: true,
        chromeOptions: {
          args: ['disable-gpu']
        }
      }
    },
...

[beatfactor]

Please have a look at this conversation on the Mailing List.

[vlad-vinogradov]

Please have a look at this conversation ...

Yes, the legacy mode can be used:

'chrome-dev': {
   webdriver: { 
      use_legacy_jsonwire: true ,
      host: ...,
      port: ...,
      default_path_prefix: '/wd/hub'
   },
   desiredCapabilities: {
      browserName: 'chrome',
      'goog:chromeOptions': {
          w3c: false,
          prefs: { 'profile.managed_default_content_settings.notifications': 1 }
       },
       ...
   }
}

But the latest chromedriver version has a bug: it rejects empty POST body in any mode.

Source code:

if (request.data.length()) {
  ...  
} else if (kW3CDefault && iter->method == kPost) {
  // Data in JSON format is required for POST requests. See step 5 of
  // https://www.w3.org/TR/2018/REC-webdriver1-20180605/#processing-model.
  PrepareResponse(trimmed_path, send_response_func,
                  Status(kInvalidArgument, "missing command parameters"),
                  nullptr, session_id, kW3CDefault);
  return;
}

(kW3CDefault is a constant with true value in the latest version)

Workaround solution:
replace line 80 in the nightwatch/lib/transport/actions.js:

opts.data = opts.data || '';

with:

opts.data = opts.data || {};

[beatfactor]

So is that all that's needed? There is also #2122 open.

[vlad-vinogradov]

This is enough for legacy JsonWire requests be handled successfully in the latest chromedriver version. I mean legacy POST-requests for which JsonWire protocol doesn't specify any data in the request body, for example, click-endoint.

At least, my tests are performed without problem, including those in which displayed-endpoint is involved.

And, as stated in #2109, you should not use selenium_host and selenium_port settings (only webdriver.js and jsonwire.js should be used in transport.js).

In the latest chromedriver versions the value of the kW3CDefault constant has been changed.
As a result, some logical bugs pop up associated with the goog:chromeOptions.w3c mode.

As for #2122, I don’t think that chromedriver requires something in the request body that isn’t
in the protocol specification. The message "missing command parameters" is returned simply because of the above error.

About chromedriver in W3C mode i have said in the #2109 (use_legacy_jsonwire: false).
'goog:chromeOptions':{w3c:true} is default mode in the latest versions.
In W3C mode POST body must have JSON-value. See Webdriver Processing Model

[vlad-vinogradov]

It should also be said that in the future chromedriver versions JsonWire support may be removed.

See: Chromedriver Issue 2596: Remove legacy wire protocol support

So, it makes sense to insist on displayed-endpoint support in w3c mode.

[OndraM]

Hi, many thanks @vlad-vinogradov for digging up the cause and providing a suggestion for the workaround. I can confirm sending basically anything inside POST body solves the problem for Chromedriver 75 and as per our testcases it didn't break anything else :-) . (However just empty dictionary {} does not solve it - there must be at least one element inside the POST body data.). So actually no ID is required as in #2122.

This is how we solved it in php-webdriver: https://github.com/facebook/php-webdriver/pull/640/files#diff-1ee156e0c33356b17f4dc2b3faec69eeR280

And we may hope Chrome team solves the bug 2943 ASAP.

[vlad-vinogradov]

One of the most effective ways of testing programs is reading the source code by other programmers.
Only this method allows to detect logical errors :)

[individual-it]

I can confirm that the fix makes it possible to run Chrome 75 with JSONWire protocol, but my reported error message actually happened with the W3C protocol. That still does not work doesn't nightwatch support the W3C protocol fully?

[vlad-vinogradov]

... That still does not work doesn't nightwatch support the W3C protocol fully?

displayed endpoint is re-enabled in chromedriver w3c-mode. See:

• Bug: chromedriver:2943:

  The Is Element Displayed command is not part of W3C spec, but is still
  used by some APIs, and its functionality can be difficult to replicate
  in those APIs. This CL re-enables this command in W3C mode to ease
  transition to W3C mode.

• chromedriver source code fix

So, Nightwatch's Visible-methods should work fine with future chromedriver versions in any mode.
Wait a little...

[vlad-vinogradov]

https://bugs.chromium.org/p/chromedriver/issues/detail?id=2943#c9:

The bug that incorrectly rejected POST request with empty body in non-W3C mode has been fixed, and has been released in ChromeDriver 75.0.3770.90 and 76.0.3809.25

In addition, ChromeDriver 76.0.3809.25 and beyond will accept Is Displayed command in both W3C and OSS modes.

See also: https://chromedriver.storage.googleapis.com/index.html

[beatfactor]

@vlad-vinogradov Thanks for the update. So should we remove the work-around?

[vlad-vinogradov]

So should we remove the work-around?

You can remove it a little bit later, when most of the users switch from ChromeDriver 75.0.3770.8 to 75.0.3770.90.

In reality i think that you can use any of the POST body values. But if some specific choice is needed, it makes sense to follow the behavior of Selenium-Webdriver API.

JsonWire Protocol doesn't specify the strict value of the POST body in cases where there are no JSON parameters. JsonWire specification is incomplete in that regard (the same applies to W3C Webdriver specification). So, you need to consider the implementation of specific JsonWire Servers and Clients.