Docker-selenium: Selenium grid not respecting timeout values

[stenver]

Hello

I have tried to increase Selenium node timeout values, but the nodes do not respect the values given. I have a long-running job that requires long uptime from Selenium, but the nodes shut down browser within a short time limit.

Here is what I have tried to do:

JAVA_OPTS="-Xmx512m -Dwebdriver.server.session.timeout=-1 -Dwebdriver.server.browser.timeout=-1"
docker run -d -p 4444:4444 -e JAVA_OPTS="$JAVA_OPTS" -e GRID_TIMEOUT=86400 -e GRID_BROWSER_TIMEOUT=86400 --name selenium-hub selenium/hub

I also tried using the SE_OPTS to increase timeout, but then nodes fail to start

docker run -d -p 4444:4444 -e JAVA_OPTS=-Xmx512m -e SE_OPTS="-timeout=86400 -browserTimeout=86400" --name selenium-hub selenium/hub

I have also set the client timeout to be longer:

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 86400000
conf[:http_client] = client
conf[:desired_capabilities] = Selenium::WebDriver::Remote::Capabilities.firefox

Capybara.register_driver driver_name do |app|
    Capybara::Selenium::Driver.new(app, conf)
end

[stuffandthings]

I'm having this same issue, using 2.50.0 and not via Docker

[rubytester]

I think in SE_OPTS you may not use = signs.

SE_OPTS="-timeout 86400 -browserTimeout 86400"

You can also provide your own config.json override for your particular customization.

[slugics]

any way to pass in maxSession and maxInstances to SE_OPTS. I tried following -browser "maxSession=2,maxInstances=5" but didnt work

[abacaj]

@slugics FYI maxSession=2 will overwrite maxInstances=5, max session at 2 means you can only run 2 total browsers on that node regardless of their type.

Here is what you want in order to set max session, it is set as an environment variable:

docker run -d -p 4444:4444 -e GRID_MAX_SESSION=2 --name selenium-hub selenium/hub:2.50.0

[slugics]

Thanks for the prompt reply. I guess that didn't seem to solve my problem. My problem is that tests run and than just get stuck at "starting chrome driver 2.20" and nothing happens afterwards. Thought maybe something to do with sessions but thats not the case it seems. Any help would be appreciated.

[abacaj]

So your HUB works and you are able to send a test to it but it is stuck starting chrome?

How are you initializing your chrome driver in your code - would need to see some sample code.

My guess is the hub is not reachable from your code - if you are hosting your hub locally then you should be able to see http://localhost:4444/grid/console. Are you able to see that url and if you do see it, what do you see... post a screenshot.

Another issue is to make sure the selenium node is the same version as the hub i.e with this command:
docker run -d --link hub:hub selenium/node-chrome:2.50.0

[slugics]

My grid is in EC2 and test do run against the grid. However after like 10
tests or less it gets stuck. Last msg i see in the output is that chrome
is starting and thats were its stuck till i kill the tests.
On Feb 9, 2016 12:27 PM, "Anton Bacaj" [email protected] wrote:

So your HUB works and you are able to send a test to it but it is stuck
starting chrome?

How are you initializing your chrome driver in your code - would need to
see some samples.

My guess is the hub is not reachable from your code - if you are hosting
your hub locally then you should be able to see
http://localhost:4444/grid/console. Are you able to see that url?

—
Reply to this email directly or view it on GitHub
https://github.com/SeleniumHQ/docker-selenium/issues/150#issuecomment-182046953
.

[abacaj]

@slugics Are you calling driver.quit() at the end of your test? Your browsers could be hanging and not closing.

Run your grid with:
docker run -d -p 4444:4444 -e GRID_MAX_SESSION=2 -e GRID_BROWSER_TIMEOUT=30000 --name hub selenium/hub:2.50.0

This will ensure the browser closes if it hangs for >= 30 seconds, by default it is set to 0 (which is infinite) - thus giving you that output because the browser is never closed and the node is never freed.

[stenver]

I tried overriding the conf.json and also injected environmental variables like GRID_BROWSER_TIMEOUT=30000, but when I have a test which sleeps for 10 minutes, then the browser still closes

[abacaj]

What is the issue you are having @stenver ? A test that sleeps for 10 minutes isn't really a normal use case here.

[slugics]

Thanks for all your help. One more question. I am not able to add more browser instances. I would like to have more than 1 but whatever I tried it keeps showing only one. I want to see something like in this img attached. Thx

[slugics]

Ok thanks. Is there a limit of number of nodes I can run? I tried to run 15 and after period of time my docker just crashed. I couldn't even restart the service. I had to kill the entire VM.

[slugics]

Its large VM on EC2. Instance Type r3.4xlarge

[abacaj]

I do not use amazon to host my grid - but here are some performance details from my hub:

• 2 core machine, with at least 7gb of memory: 4 docker containers running Chrome.
• 2 core machine, with at least 7gb of memory: 4 docker containers running Firefox.
• 2 core machine, with at least 7gb of memory: 1 docker container running Hub.

Total of 8 browsers and 3 machines, you can use smaller machines for less money.

Selenium grid runs better scaled outward this means more machines, not more cpu or ram.

Your machine is extremely large and will not perform well with many containers on it regardless of how big it is. I know because I have tried very large D4 machines.

I recommend, scaling down to smaller machines and get more instances. It will perform much better.
Make sure the HUB is running on its own instance - this is key for performance.

Remember each docker container is running a browser - browsers are heavy in terms of CPU. The more browsers you run - the heavier the CPU is being used and the less time each node gets to run.

[slugics]

Thanks for all your help. i will give it try with more smaller vm's

[abacaj]

Your welcome - I had to go through this so I learned the hard way haha.

But my Production HUB has been active for days now without any issues in terms of performance all because I have scaled it outward.

[slugics]

We have over 2k tests running against our grid which consists of 19 VM's. No issues at all for past two years but wanted to see how I can use docker to reduce the number of VM's

[abacaj]

Cool - let me know how it turns out by spreading the docker containers.

FYI: Our 8 node grid (3 VMs) handles 450 automated requests an hour at about 1minute per automation, this is roughly 10,000 automation requests per day.

[slugics]

I have trouble connecting nodes to my hub when i put them on different vm. What command did you use to run each of them. Here is what i tried.

VM 1
docker run -d -p 4444:4444 -e GRID_MAX_SESSION=2 -e GRID_BROWSER_TIMEOUT=30000 -e SE_OPTS="-timeout 30000" --name selenium-hub selenium/hub:2.50.0

VM 2
docker run -d -p 4444:4444 -e REMOTE_HOST="http://:4444" -e HUB_PORT_4444_TCP_ADDR=“” -e HUB_PORT_4444_TCP_PORT="4444" selenium/node-chrome:2.50.0

Anything missing here? I do see one of them connect but its showing following error on the grid/console
DefaultRemoteProxy unknown version,server response code : 500

Thx

[slugics]

docker run -d -p 4444:4444 -e REMOTE_HOST="http://hubip:4444" -e HUB_PORT_4444_TCP_ADDR=“hubip” -e HUB_PORT_4444_TCP_PORT="4444" selenium/node-chrome:2.50.0

Not sure why < ip > was removed from my post up there

[rubytester]

@slugics by default node listens on port 5555 (unless you override it). so to connect your node to hub try this

docker run -d -p 5555:5555 -e REMOTE_HOST="http://YOURIP:5555" -e HUB_PORT_4444_TCP_ADDR=HUBIP -e HUB_PORT_4444_TCP_PORT=4444 selenium/node-chrome

[rubytester]

Is the original issue fixed for you@ @stenver ? I see other comments that are valid but unrelated to the actual issue topic

[abacaj]

I would highly recommend using something like ansible for automating your grid deployments - my ansible scripts help avoid the problems you describe, @slugics - but the right command is to expose the default port which is 5555:5555 as @rubytester mentioned.

Also do not forget to share the hosts memory when running your chrome container by adding -v /dev/shm:/dev/shm to your docker run command. This is a work around for node-chrome crashing.

[slugics]

Thanks. I had no luck running the hub separately on different machines. So tried to run hum with 5 nodes on the same machine which is not too much and it worked fine for about 45 min than hub crashed again. Trying to start the hub again I got this message.

Error response from daemon: database or disk is full

What is the log location for the docker? What could be taking up all the disk if that message is correct?

Thx

[stenver]

Sorry for the late response, I didnt have a chance to work with this.

I wanted to increase the timeout time to practically infinite, so I set it to 86400 seconds(24 hours). This could be used for debugging, developing a site and on some very special cases.

When I used Selenium Grid dockers and used the SE_OPTS or GRID_TIMEOUT, then these timeout values were not respected, the browser shut down inside the containers.

I proceeded to create my own container and manually set the timeout values with:
entrypoint.sh

#!/usr/bin/env bash

function shutdown {
    echo "shutting down selenium.."
    kill -s SIGTERM $NODE_PID
    wait $NODE_PID
    echo "shutdown complete"
}

echo "Starting xvnc and webdriver:"

export DISPLAY=:1
Xvfb :1 -shmem -screen 0 1600x1440x16 &
sleep 1
x11vnc  -rfbport 5900 -passwd vagrant -display $DISPLAY -N -forever -shared &
sleep 1
fluxbox -display $DISPLAY &
sleep 1

echo "starting selenium with configuration:"
cat /opt/selenium/config.json

java -jar -Djava.security.egd=file:///dev/urandom \
  /opt/selenium/selenium-server-standalone.jar \
  ${SE_CONF} \
  ${SE_OPTS} &\

NODE_PID=$!

trap shutdown SIGTERM SIGINT
wait $NODE_PID

I started the container with the command:

$ docker run -i -t -p 4444:4444 -v /dev/urandom:/dev/random --privileged stenver/seleniumhub /bin/bash
/$ SE_OPTS=" -browserTimeout 86400 -timeout 86400" ./opt/bin/start_container.sh

This time, the timeout values were respected.

In summary, Selenium dockers dont seem to respect the SE_OPTS, GRID_TIMEOUT, or GRID_BROWSER_TIMEOUT values. Currently I got around the limitation by creating my own dockers. I might investigate the issue futher and try to fix it on selenium as well in the coming weeks

[dgangwar]

I was able to get the timeout and browsertimeout value working for my grid setup. I tried using docker-compose file.

hub:
image: selenium/hub
ports:
- "4444:4444"
privileged: true
environment:
- GRID_BROWSER_TIMEOUT=100000
- GRID_TIMEOUT=90000
firefox:
image: selenium/node-firefox
ports:
- "5900"
links:
- hub
chrome-debug:
image: selenium/node-chrome-debug
links:
- hub
ports:
- "5900"
volumes:
- /dev/shm:/dev/shm

[Zechtitus]

Just putting this in here as I couldn't get the selenium grid to work with any of the above solutions. (time of writing it was using 2.53.0 version for :latest)

Docker-compose file:
version: "2"

services:
hub:
image: selenium/hub:latest
ports:
- "4444:4444"
environment:
- GRID_BROWSER_TIMEOUT=300000
- GRID_TIMEOUT=300000
- SE_OPTS=-browserTimeout 100 -timeout 100

You can check this by going into the grid and verifying the values http://localhost:4444 - they should correspond to the values you set (in seconds). I have a feeling the GRID_BROWSER_TIMEOUT and GRID_TIMEOUT settings can disappear as well. If your playing around with other settings and want to figure out the right syntax - put them into the docker-compose SE_OPTS var and then do the following:

1. docker-compose run hub /bin/bash
2. once inside the docker container run the /opt/bin/entry_point.sh
3. if there is an error it should be obvious now... if it looks like it succeeded then you can go the route of looking in your http://localhost:4444 and see if the changes were applied

Hope this helps somebody with debugging the docker containers from selenium. I was running into stupid issues with using quotes or not, having : to split args and such.

[diemol]

I'll close this one since the last two comments show the proper way to setup the timeouts in docker-selenium, which addresses the initial issue.