Docker-selenium: Use Docker networking instead of links

[logankoester]

Docker links are a one-way, single-host communication system. They should now be considered deprecated, and you should update your app to use networking instead.

See https://docs.docker.com/compose/networking/

There is a condition in NodeBase that causes node containers to exit when networking is enabled, because it causes links to be ignored.

[golonzovsky]

+1

[SpencerMalone]

You can temporarily get around the condition check like this:
docker network create SE
docker run -d -p 4444:4444 --net=SE --name selenium-hub selenium/hub:2.48.2
docker run -d --net=SE -e HUB_PORT_4444_TCP_ADDR=selenium-hub selenium/node-firefox:2.48.2

[golonzovsky]

@SpencerMalone thanks! Was looking for this kind of param :)

[rubytester]

Up to now there were two ways to run selenium grid configurations with hub and nodes.

You either 1) --link the node to hub given giving the hub link name hub or you 2) tell the node 2 pieces of vital info to connect to the hub, hub address and hub port in the form of environment variables HUB_PORT_4444_TCP_ADDR and HUB_PORT_4444_TCP_PORT

Case 1 --link named hub

Why such names for env variables? The selenium node container consumes the env variables that get created in that container when --link to hub named hub is used. The link has to be named hub for convention to work.

For example:

docker run -d -p 4444:4444 --name selenium-hub selenium/hub
docker run -d --link selenium-hub:hub selenium/node-chrome

The node is a recipient of env variables because port 4444 has been exposed in the hub. The hub has Dockerfile entry EXPOSE 4444.

There are several env variables created in the recipient (node) to provide information about that port in the target container (hub)

The node receives information about the target of the link with this format <name_of_link>_PORT_<exposed_port_number>. The variables created are:

HUB_PORT
HUB_PORT_4444_TCP
HUB_PORT_4444_TCP_ADDR
HUB_PORT_4444_TCP_PORT
HUB_PORT_4444_TCP_PROTO

We use this convention and consume the two env variables that get created to register the node to the hub.

(fyi: I made a youtube video on this a while back trying to explain links in a lighting talk: https://www.youtube.com/watch?v=nRZ9aYizUEo)

Case 2: hub and node on separate machines

In the case where a simple --link worked when you are on the same machine not the same is true when your hub and node sit on different machines and they don't know about each other's existence. You simply can't connect the node to the hub with a --link. Because of this we tell the node what the address and port is for the hub by running the node with something like this. The env variables don't get created with the link. We have to create them as env vars when starting a node.

docker run -d -e HUB_PORT_4444_TCP_ADDR=<hub-ip-address> -e HUB_PORT_4444_TCP_PORT=<4444-usually> selenium/node-chrome

And the entry_point.sh gets the info to register the node to the hub. There is an extra problem here that is solved with -e REMOTE_HOST=<outside-ip-of-your-node-vm> So really 3 piece of info are needed.

Case 3: networking instead of link

Now we have docker-compose and networking. Basically how do we deprecated Case 1 but still make Case 2 work.

Thoughts? Do we want to abandon --link now and move to --net ? I would not want to add new env vars unless needed. (PR #136 add new var to bypass a guard when address is not set).

I have not tested running containers with networing. I use Case 2 and some marathon mesos.

[rubytester]

I have played a bit with the network instead of a link and in this case I can see we just use the existing env variables (and perhaps we can make them more human readable like HUB_HOST and HUB_PORT)

Basically the example provided by @SpencerMalone shows how docker network is used instead of --link. The node knows how to register with the hub becuase the hub name is entered in node's /etc/hosts file, I can see it there twice with and gain as .

Using example like this:

docker network create grid
docker run -d -P --net=grid --name hub selenium/hub
docker run -d --net=grid -e HUB_PORT_4444_TCP_ADDR=hub -e HUB_PORT_4444_TCP_PORT=4444 selenium/node-chrome

Usign docker-compose.yml it may look like this:

hub:
  image: selenium/hub
  ports:
    - "4444:4444"
  container_name: hub
node:
  image: selenium/node-chrome
  environment:
    - HUB_PORT_4444_TCP_ADDR=hub
    - HUB_PORT_4444_TCP_PORT=4444

Where I run it as docker-compose --x-networking -d -p grid up -d it will create a network named grid and stick to a static container name hub for the hub so we can refer to it in the env var for the node

The above command starts a selenium grid with experimental networking instead of using --link

$ docker-compose ps
   Name               Command           State           Ports
----------------------------------------------------------------------
grid_node_1   /opt/bin/entry_point.sh   Up
hub           /opt/bin/entry_point.sh   Up      0.0.0.0:4444->4444/tcp

Notice above I have force the project name to be grid but it does not have to be. it will by default become a name of a folder your docker-compose.yml is located in.

I would hold off right now on PR #136 in its present form.

For those of you using docker-compose can you please provide your usage examples in your projects? I would like to see the use case scenarios that work or don't work for you?

Thanks.

[SpencerMalone]

Whatever we do, supporting links would be very nice. Docker decided to stop support of RHEL 6 after 1.7.1, so I'm sure there are people other than me who can't actually use the new networking stuff. I'm down to hold off on the PR. What would you think about this implementation?:

Use the same setup that was used with linking. Assume that the hub will have a certain container name, and assume that the nodes will be on the same network as the hub. Verify that either the HUB_PORT_4444 variables are filled (by people manually entering them or by linking), or that /etc/hosts contains a host entry that fits with our predetermined container name. Not really sure why this didn't cross my mind when I whipped up that PR...

[alexkogon]

Hi folks,

I was just about to make an issue about this but searched through them and found this.

I wanted to connect my nodes to an existing Selenium Grid Hub, and even dug deep enough to find the HUB_PORT_4444_TCP_ADDR and HUB_PORT_4444_TCP_PORT and set them appropriately, but was not able to get that working. I was very pressed for time just before having to stop working for the end of year holidays, so tried running with the Docker hub as outlined in the normal docs, and that worked fine.

I'll give this a shot again next couple days based on the info here and come back if I can't get it working...

[alexkogon]

This works fine, sorry there were networking issues on my side which I didn't realize until I researched further. Thanks for the great work folks!

[sudokai]

+1 to keeping the env variables. I'm facing the same situation as @SpencerMalone, running this on RHEL6 and can't upgrade Docker beyond 1.7.1.

[defender]

Hi

There is another problem that currently is not solved in current solution. In case you want to run hub and nodes on different machines but also several nodes containers per machine.

It means that you should manage node external ports by yourself. In case you will run it with -p :xxx we will have connectivity problem. Hub will not be able connect node because his port is unknown.

My suggestion is

1. Mount docker to node container we will get docker inside docker.
2. Now we can run docker inspect inside node entry_point.sh to retrieve external port and send it to hub with -port parameter

Now you can run several nodes on same machine and hub on another one

WDYT ?

Thanks

[alexkogon]

Interesting @defender I did run multiple containers on a single machine but it seemed to figure it out maybe I wasn't looking closely enough?

Will take a look again and let you know

[alexkogon]

Hi folks (and @defender)

When running multiple Docker nodes on the same machine as the hub, exposing 80:4444 for the hub and giving the local IP address for HUB_PORT_4444_TCP_ADDR and 80 for HUB_PORT_4444_TCP_PORT, it works great and the internal callback mapping for 5555 (or whatever it is being loaded up on) is done auto-magically without me doing anything.

It does seem there is a problem with using nodes on different machines though. At the least perhaps we need to open a port with -p for Docker and tell the hub where to call them back. @elgalu do you have a solution for this? Should I create another ticket? I haven't been able to get a remote node working at all so far, but that may have been due to weird networking issues within our environment I've been trying to resolve...

[VolkerK]

When using docker-compose 1.6 the network is automatically created and linking is not needed.
Still you need to specify the linking environment variables, since they are used in the entrypoint.sh
My docker-compse.yml (version 2) looks like this:

version: '2'
services:
  hub:
    image: selenium/hub:${SEL_VERSION}
    ports:
      - "4444:4444"

  firefox:
    image: selenium/node-firefox:${SEL_VERSION}
    volumes:
      - /dev/urandom:/dev/random
    depends_on:
      - hub
    environment:
      - HUB_PORT_4444_TCP_ADDR=hub
      - HUB_PORT_4444_TCP_PORT=4444

  chrome:
    image: selenium/node-chrome:${SEL_VERSION}
    volumes:
      - /dev/urandom:/dev/random
    depends_on:
      - hub
    environment:
      - HUB_PORT_4444_TCP_ADDR=hub
      - HUB_PORT_4444_TCP_PORT=4444

I can then start my grid and scale it with docker-compose very handy.

docker-compose up -d
docker-compose scale firefox=4 chrome=2

Which will start a grid with four firefox nodes and two chrome nodes.

[alexkogon]

@VolkerK thanks, that is a good idea; I've not been using Docker compose or Swarm because I prefer direct control and it doesn't really add a lot of value compared to what I'm already doing with automation to bring Docker nodes up and down.

One issue I had was when using Nodes on multiple hosts, the REMOTE_HOST had to be defined to callback to the Docker host on the port specified instead of the default behavior where the node tells the Hub where it is, which does not work unless the Hub is on the same machine. Does Docker Compose resolve this issue somehow on multiple machines in a Swarm or are you running it all on one host so it is not an issue?

[VolkerK]

@alexkogon I only use compose and not swarm, so everything runs on one host.
But I think in newer versions you have something called multi-host networks. Then it should be possible to start nodes on multiple hosts and they act like they are in the same network. So every node should be able to see the hub.
But you need a key-value store and a swarm cluster for that (see above link)

[alexgorbatchev]

May I suggest to add docker compose information to the readme? I spent a whole evening trying to get a node to connect to the hub with compose using various methods. The details in this issue resolved my issue with node not seeing the hub.

[alexkogon]

@VolkerK thanks for the info. We're still wiring it all together so we take it in steps, keep it working and automate what works :)

[wfwheele]

+1 for adding current docker-compose instructions to README. +1 to this issue overall

[hurrycaine]

I also set up swarm and then ran into issues connecting.
@SpencerMalone

docker run -d --net=SE -e HUB_PORT_4444_TCP_ADDR=selenium-hub selenium/node-firefox:2.48.2

This did not work for me, it connected to the selenium-hub ok but the node checked in with its eth1 ip which was internal to docker host only. It was not using the ip that the --net was using, which for me was eth0.

I had to do as suggested @rubytester and add -e REMOTE_HOST=<outside-ip-of-your-node-vm>
but that was not quite right as the hub returned a 500 with Not a correct url to register a remote :
I had to add http:// in front.

So my command looked like something like this
docker run -d --name safefoxy4 --net=selenswarm -e HUB_PORT_4444_TCP_ADDR=safe-selenium-hub -e REMOTE_HOST=http://safefoxy4:5555 selenium/node-firefox

All swarm did for me in this exercise was allow me to spin up nodes without worrying about externalizing the port, at first it did not seem worth it but it is nice to be able to spin up nodes (sometimes on same servers) and not have to worry about ports.

[Z3r0Sum]

@rubytester My use case is leveraging docker swarm across multiple EC2 instances. I have a project I wrote - that mimics the essence of docker compose; however, allows me to dynamically scale the project itself. This way each "build" we perform can spinup its own grid infrastructure based on its needs without manipulating a yaml file and running into the state issues that it presents. Right now, I dynamically gather information about the hub invocation and use link. I want to use swarm networking across multiple EC2 instances to more effectively leverage the resources I have at my disposal. Is this actively being worked on? It seems like Case 2 should work for now and I can test it tomorrow. Just to verify:

docker run -d -e HUB_PORT_4444_TCP_ADDR=<hub-ip-address> \
 -e HUB_PORT_4444_TCP_PORT=<4444-usually> \
 -e REMOTE_HOST=<outside-ip-of-your-node-vm> selenium/node-chrome

[iamluc]

A little hack to use docker-compose scale and swarm multi-hosts

version: '2'

services:
    seleniumhub:
        image: selenium/hub

    selenium:
        image: selenium/node-firefox-debug
        environment:
            HUB_PORT_4444_TCP_ADDR: "seleniumhub"
            HUB_PORT_4444_TCP_PORT: "4444"
        entrypoint: 'bash -c "REMOTE_HOST=\"http://$$HOSTNAME:5555\" /opt/bin/entry_point.sh"'

[symehmoo]

How can i achieve running docker selenium hub on a instance and multiple other instances that are nodes communicate to my hub
ideally i want to have 1 Hub and 5 node machines (that have 50 chrome browsers & 50 firefox browsers)

the yaml file i use with docker compose is
hub:
image: selenium/hub
ports:
- "4444:4444"
firefox:
image: selenium/node-firefox
links:
- hub
chrome:
image: selenium/node-chrome
links:
- hub

how can i have multiple hosts

[VAdamec]

With new docker swarm there is problem with scaling, for proper proxy communication you need to pass port, but than there is an with exposing port and scale. This configuration works (1 replica):

docker network create --driver overlay --subnet 10.0.50.0/24 --opt encrypted seleniumnet

docker service create --network seleniumnet --name selenium-hub -p 4444:4444 \
--constraint 'node.role == manager' \
-e JAVA_OPTS="-Xmx512m -DPOOL_MAX=512" -e GRID_TIMEOUT=600000 \
-e GRID_BROWSER_TIMEOUT=600000 selenium/hub

docker service create --network seleniumnet --name selenium-node-chrome-300 \
--constraint 'node.role == worker' \
-p 5556:5556 \
--mount type=bind,source=/dev/shm,target=/dev/shm \
-e HUB_PORT_4444_TCP_ADDR=swarm.service.dc1.consul -e HUB_PORT_4444_TCP_PORT=4444 -e NODE_MAX_INSTANCES=10 -e NODE_MAX_SESSION=10 \
--replicas 1 selenium/node-chrome:3.0.0-cerium bash -c 'SE_OPTS="-host swarm.service.dc1.consul -port 5556" /opt/bin/entry_point.sh'

docker service create --network seleniumnet --name selenium-node-firefox-300 \
--constraint 'node.role == worker' \
-p 5557:5557 \
--mount type=bind,source=/dev/shm,target=/dev/shm \
-e HUB_PORT_4444_TCP_ADDR=swarm.service.dc1.consul -e HUB_PORT_4444_TCP_PORT=4444 -e NODE_MAX_INSTANCES=10 -e NODE_MAX_SESSION=10 \
--replicas 1 selenium/node-firefox:3.0.0-cerium bash -c 'sudo systemd-machine-id-setup; SE_OPTS="-host swarm.service.dc1.consul -port 5557" /opt/bin/entry_point.sh'

# swarm.service.dc1.consul is DNS record for all servers in Swarm.
swarm.service.dc1.consul:4444 - grid console
swarm.service.dc1.consul:5556 - node console for chrome
swarm.service.dc1.consul:5557 - node console for firefox

When you run scale it stops responding to remote calls. Still finding where is an issue. Anybody run Selenium in new docker swarm setup ?

[haonumen]

@VAdamec try this code. it worked.

docker service create --network seleniumnet --name selenium-node-chrome-300
--constraint 'node.role == worker'
-p 5556:5556
--mount type=bind,source=/dev/shm,target=/dev/shm
-e HUB_PORT_4444_TCP_ADDR=swarm.service.dc1.consul -e HUB_PORT_4444_TCP_PORT=4444 -e NODE_MAX_INSTANCES=10 -e NODE_MAX_SESSION=10
--replicas 1 selenium/node-chrome:3.0.0-cerium bash -c 'SE_OPTS="-host $HOSTNAME -port 5556" /opt/bin/entry_point.sh'

[saifsysim]

@VAdamec Does this work with the Se3 hub

[tparikka]

Related to this info, the use of -e REMOTE_HOST no longer seems to work as of version 17.03.1-ce, throwing the following error.

REMOTE_HOST variable is DEPRECATED in these docker containers. Please use SE_OPTS="-host -port " instead!

[derek-adair]

Adding the version: "2" config for docker-compose.yml in the demo's would be quite nice.

Thanks @VolkerK, I only wish I found this BEFORE the hours of debugging and learning how grid works in docker... damnit.

[diemol]

Hi all,
In https://github.com/SeleniumHQ/docker-selenium/releases/tag/3.8.1-chlorine we have introduced HUB_HOST and HUB_PORT, in addition the README has been updated on its usage. Therefore now you can also use docker networking.

Please give it a try, any feedback is appreciated.

I'll close this issue since the PR sent for it was reused to produce the current release.