Sitespeed.io: Error / Crash when testing multiple urls, and tracking cpu usage, with version 17+

[psyked]

Overview

Since version 17, I am seeing the following error when using a text file of URLs (with aliases) where there was previously no error (up to 16.10.3)

Error details

Google Chrome 90.0.4430.72 
Mozilla Firefox 87.0
Microsoft Edge 89.0.774.14 dev
INFO: Versions OS: linux 5.10.25-linuxkit nodejs: v12.16.2 sitespeed.io: 17.0.0 browsertime: 12.0.0 coach: 6.3.3
INFO: Will run Lighthouse tests after Browsertime has finished
INFO: Running tests using Chrome - 3 iteration(s)
INFO: Use Chrome trace categories: -*,disabled-by-default-lighthouse,v8,v8.execute,blink.user_timing,devtools.timeline,disabled-by-default-devtools.timeline,disabled-by-default-devtools.timeline.stack
INFO: Testing url https://www.comparethemarket.com/ iteration 1
ERROR: Failed waiting on page https://www.comparethemarket.com/ to finished loading, timed out after 300000 ms WebDriverError: unknown error: session deleted because of page crash
from tab crashed
  (Session info: chrome=90.0.4430.72)
from tab crashed
  (Session info: chrome=90.0.4430.72)
    at Object.throwDecodedError (/usr/src/app/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/usr/src/app/node_modules/selenium-webdriver/lib/http.js:642:13)
    at Executor.execute (/usr/src/app/node_modules/selenium-webdriver/lib/http.js:568:28)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Driver.execute (/usr/src/app/node_modules/selenium-webdriver/lib/webdriver.js:731:17)
INFO: URL failed to load, trying 5 more time(s): Failed waiting on page https://www.comparethemarket.com/  to finished loading, timed out after 300000 ms 
ERROR: Failed waiting on page https://www.comparethemarket.com/ to finished loading, timed out after 300000 ms NoSuchSessionError: invalid session id
    at Object.throwDecodedError (/usr/src/app/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/usr/src/app/node_modules/selenium-webdriver/lib/http.js:642:13)
    at Executor.execute (/usr/src/app/node_modules/selenium-webdriver/lib/http.js:568:28)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Driver.execute (/usr/src/app/node_modules/selenium-webdriver/lib/webdriver.js:731:17)

Testing only a single URL at a time does not elicit this response.

Replication steps

This is using the Docker images, so the steps to replicate would be:

1. Text file containing the following (urls.txt):

https://www.comparethemarket.com/ Home
https://www.comparethemarket.com/customer-rewards/ Rewards

1. Config file / CPU enabled via:

{
    "cpu": true
}

1. Run command:

docker run \
        --rm \
        -v "$(pwd)":/sitespeed.io \
        sitespeedio/sitespeed.io:17.0.0-plus1 \
        --config "config.json" \
        --plugins.add analysisstorer \
        --plugins.add /lighthouse \
        ./urls.txt

Additional information / debugging

• Testing a single URL does not create this error
• Testing without tracing CPU usage does not create this error
• Testing with < 17.0.0 does not create this error

[soulgalore]

Hi @psyked thank you for creating the bug and writing exactly how you run, that helps a lot!

I could reproduce it with that URL when I enable the CPU options but only in the Docker container, not when I run on Mac OS. I've seen more problems with Chrome/Chromedriver 90 (see for example https://github.com/sitespeedio/browsertime/issues/1548) so I think there has been undocumented (or un-communicated) changes). Let me have a deeper look and see if I can fix it.

[soulgalore]

Checking the Chromdriver log:

[1619102730.775][INFO]: [ddc7b3aa7e74a6df7f37391b4b32dae9] RESPONSE ExecuteScript ERROR unknown error: session deleted because of page crash
from tab crashed
  (Session info: chrome=90.0.4430.72)

I'm gonna see if I can debug further tonight and then create a upstream bug in Chromedriver and try to make smaller test case.

[soulgalore]

Let me roll out an alternative docker container tomorrow with Chrome 89 and latest sitespeed.io and then I'll create the issue for Chromedriver/Chrome asap.

[soulgalore]

I did some more testing this morning at it seems this only happens in our Docker container, I tried with Ubuntu and it works fine there. Also could reproduce with https://www.sitespeed.io. I reverted to Chrome 89 (and Chromedriver 89) and I get it there too, so it must be something that has changed on our side. We changed the start flags for Chrome, I think that could be the thing.

[soulgalore]

I reverted the updated settings for Chrome and that fixed the problem. I wanna find out exact which setting caused the breakage and then I will roll out a new version.

[soulgalore]

@psyked I've released 17.1.1 that at least fixes the problem for me, please let me know if it works for you!

[psyked]

@soulgalore that fix works for us - thank you 😄