Docker-selenium: Frequently getting errors like DevToolsActivePort missing or Chrome is not reachable in the grid

[rnemeth1980]

🐛 Bug Report

I get errors for every build but for always for a different job:
org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Chrome version in the pod and in the framework are in sync: 75.0.3770.90. Selenium and docker image versions are also in sync: 3.141.59
I have 10 nodes with 3 chrome instances each on k8s.
options.addArguments("--headless") options.addArguments("--no-sandbox") options.addArguments("--disable-dev-shm-usage") do not help either.

If I remove the headless option the Chrome not reachable error occurs frequently:
https://gist.github.com/rnemeth1980/3f45ea3407be1b6e9d6804734a0a8836

Grid configuration:

To Reproduce

Error occurs frequently every day by running the build on bamboo.

Expected behavior

Grid should be stable and should not bring errors.

Environment

Kubernetes version: 1.8.7
Docker-Selenium image version: selenium/node-chrome:3.141.59
Docker image id: docker-pullable://selenium/node-chrome@sha256:d85b6ccf584edb4e198d7e9b3f05507aa3b4a1a417d5f544fb62c314544637e8

[ddavison]

This might be a reminder to us that it might be worth looking into; making new node-chrome-headless container(s) that do not have Xvfb and other superfluous stuff (VNC...).

[baflQA]

Hello. Struggling with the same issue.
How do We know what was the root cause of this exception? The pods are orchestrated by the K8S (or even apps like zalenium), the pod is a long time gone before I notice this exception.
Might that be that START_XVFB=false would help for that?

[diemol]

This might be a reminder to us that it might be worth looking into; making new node-chrome-headless container(s) that do not have Xvfb and other superfluous stuff (VNC...).

@ddavison this is already possible by passing the env var START_XVFB=false.

[diemol]

@rnemeth1980 chrome not reachable normally means that Chrome crashes due to lack of resources. It might be more stable when running headless since in theory it should use a bit less resources, but nevertheless if things like taking a screenshot is done while running headless, the browser needs to render the whole page and it will be slower and use more resources.

Bottom line is that resource monitoring should be in place to determine the proper cause.

It would be great if you can provide a way to reproduce the issue, including:

• How everything is deployed (k8s manifests or Helm charts)
• Commands used to deploy the images
• A link to a GitHub project that can be executed to reproduce the issue
• Please scope this to use the k8s engine of Docker desktop, so no big deployments are needed.

[baflQA]

@diemol how do I pass the START_XVFB=false flag when running my grid with zalenium?

[diemol]

@baflQA this is a different project, Zalenium uses elgalu/selenium, which does not have that option.

[baflQA]

Ah, ok, I thought it's built on top of the docker selenium. Thx!

[ddavison]

This might be a reminder to us that it might be worth looking into; making new node-chrome-headless container(s) that do not have Xvfb and other superfluous stuff (VNC...).

@ddavison this is already possible by passing the env var START_XVFB=false.

true, but xvfb will still reside on the image

[pawelusfr]

@rnemeth1980 chrome not reachable normally means that Chrome crashes due to lack of resources. It might be more stable when running headless since in theory it should use a bit less resources, but nevertheless if things like taking a screenshot is done while running headless, the browser needs to render the whole page and it will be slower and use more resources.

Bottom line is that resource monitoring should be in place to determine the proper cause.

It would be great if you can provide a way to reproduce the issue, including:

• How everything is deployed (k8s manifests or Helm charts)
• Commands used to deploy the images
• A link to a GitHub project that can be executed to reproduce the issue
• Please scope this to use the k8s engine of Docker desktop, so no big deployments are needed.

@diemol I quite often get DevToolsActivePort missing error.
Nodes are deployed on AWS c5.large spot instances (1 container per instance) with Amazon Linux started through an Ansible script.
Startup command is:

sudo docker run -v /dev/shm:/dev/shm --restart=always -d --name selenium-chrome -p 5555:5555 -e REMOTE_HOST=\"http://$(curl http://169.254.169.254/latest/meta-data/local-ipv4):5555\" -e HUB_HOST=$HUB_IP -e HUB_PORT=4444 -e LANG=en_GB.UTF-8 -e LANGUAGE=en_GB.UTF-8 selenium/node-chrome:3.141.59-selenium

I'm NOT using Chrome headless mode.
Usually there won't be more than 3 tests running on 1 node.

Let me know if you need more info

[ClemannD]

Also seeing (unknown error: DevToolsActivePort file doesn't exist) when trying to run an instance of protractor on our grid.

Current setup:
I have an ECS cluster with 4 EC2 instances, one of them running selenium-hub and i have 4 selenium/chrome-node containers running

Using selenium/3.141.59-vanadium for hub and nodes.

My grid configuration:

My EC2 instances have 16gb ram, I have given the nodes 2gb ram, and am mounting to dev/shm of the host. (confirmed the mount is actually working by creating test file in dev/shm of host and am able to see it in dev/shm of docker container).

The Issue:
When I start protractor and point it to my selenium hub, I am getting

All the suggestions when I google (unknown error: DevToolsActivePort file doesn't exist) say to try different combinations of --no-sandbox, --disable-dev-shm-usage and so on, tried them all, no luck.

Was wondering if this issue has been encountered by anyone and what possible solutions may be,
Thank you

[cgoldberg]

@ClemannD

upgrade chromedriver to match your chrome browser version and try again. You are running chromedriver version 74 with chrome version 77.

[ClemannD]

@cgoldberg

Thank you for the response, but I realize i accidentally uploaded the wrong screenshot, I was testing with an older version of the selenium/chrome-node docker image to see if it was an issue with the newest version. No luck.

So the issue is the same if my chrome and chormedriver versions match, both at 74 and at 77.

Any other suggestions?

[ClemannD]

I ended up solving this.

Turns out we were setting chromeOptions: instead of goog:chromeOptions:.
Not sure what the difference is, since chromeOptions: always worked for us in the past.

[diemol]

Apologies for not replying to this issue in a while, is this still happening?

[ddavison]

Closing. Go ahead and re-open if this is still happening!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.