Docker-selenium: Incorrect Remote Host Reported to Hub

[desertaxle]

Meta -

Image(s):
node-chrome

Docker-Selenium Image Version(s):
3.4.0

Expected Behavior -

Setting remote host in /opt/selenium/config.json would enable the hub to reach a node running on a different EC2 instance on a port other than 5555.

Example config.json:

{
  "capabilities": [
    {
      "version": "58.0.3029.81",
      "browserName": "chrome",
      "maxInstances": 1,
      "seleniumProtocol": "WebDriver",
      "applicationName": ""
    }
  ],
  "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
  "maxSession": 4,
  "port": 5555,
  "register": true,
  "registerCycle": 5000,
  "nodePolling": 5000,
  "unregisterIfStillDownAfter": 60000,
  "downPollingLimit": 2
  "remoteHost": http://10.205.80.202:20005
}

Actual Behavior -

The hub comes up with a remoteHost that is the docker IP and port 5555 (e.g. http://172.17.0.7:5555). This can be seen in the hub console:

This mismatch results in the node being marked as dead by the hub as it tries to reach the node on the incorrect remoteHost value.

[mach6]

In Se3, support for remoteHost in the node's config.json was removed.

Try using host and port

[desertaxle]

That was my first approach, but I found that when I set host and port then the hub wasn't able to connect to the nodes. I think it's because port mapping that I'm using when starting the nodes (e.g. 20005:5555) and when I set port to the port number on the EC2 instance then the port inside the container also changes. If that's the case then the container would no longer be listening on 5555.

Is that line of thinking correct?

[McSnowflake]

I use a selenium docker grid, with the hub being not a docker container for various reasons. I needed the -remoteHost param to tell the node which port to tell to the hub when registering. Because the exposed port is not the same as the port used inside the container (default 5555).
To use the -port param as supstitute, I would have to make sure that exposed and internal port are the same, which I cannot because the exposed port is randomly chosen by the orchestration tool.

Does somebody know away around this?
And why has the -remoteHost param been removed in the first place?

[tparikka]

I previously logged issue #432 for what I think is the same thing you're describing here and at #307 and at that time the information I was given was that if you're running multiple Selenium node containers manually with docker run, you'll have to assign each container a unique publicly facing port that is the same as the internal Selenium node listening port, like so:

docker run -d --name node101 -p 5595:5595 -e HUB_PORT_4444_TCP_ADDR=192.168.2.103 -e HUB_PORT_4444_TCP_PORT=4444 -e SE_OPTS="-host 192.168.2.103 -port 5595" selenium/node-chrome:3.4.0

You'd need to do the above as opposed to having multiple public ports exposed that port forward to 5555 on each individual Selenium node. The rationale given by @diemol was that custom port forwarding was out of scope for docker-selenium.

[diemol]

Yeah, you would need a proxy on top of that to do some port forwarding...
@desertaxle, if it is possible for you, you could set both ports to be the same ones (exposed and internally mapped one).

@McSnowflake, is there a way for you to know the port that the orchestration tool assigned before starting the container? Perhaps an env var that you can pass?

[BruceDone]

@McSnowflake
after i looked deep into the source code ,if you use the -port in the SE_OPTS , the NODE_PORT in the node docker env will be replaced by this value , so if you really need to deploy the selenium grid in different machines , here is the way .

Machine A ip:10.10.2.38
docker-compose

hub:
    image: selenium/hub:3.3.1
    ports:
      - 4444:4444
    environment:
      - GRID_TIMEOUT=40

Machine B ip:10.10.2.52

node:
    image: selenium/node-chrome:3.3.1
    ports:
          - "5556:5556"
    environment:
          - NODE_PORT=5556
          - HUB_PORT_4444_TCP_ADDR=10.10.2.38
          - HUB_PORT_4444_TCP_PORT=4444
          - SE_OPTS=-host 10.10.2.52 -port 5556

and it works well.

[discobean]

I am having the same issue.

If the selenium agents could provide a way to tell the server a host/port it is using (rather than using the provided -host and -port values), then it would solve the problem in both ECS and other docker orchestration tools.

By bringing back the -remoteHost or REMOTE_HOST variable this issue would be resolved.

[braindonor]

I've just hit this issue too.
I agree with @discobean on this: "By bringing back the -remoteHost or REMOTE_HOST variable this issue would be resolved."

[McSnowflake]

@diemol As I said: the orcestration tool (rancher) randomly chooses the externel (ie exposed) port in the moment before the container is created. I know of no possiblilty of passing that into the container.

@BruceDone Thx for the Hint. I have to use

ports:
  - "5556"

in order to make rancher assign a random port. Does the _SE_OPT-NODE_PORT-option_ the same as -remoteHost did, ie telling the node to give the exposed port to the hub when registering, instead of the internal (5556)?

[McSnowflake]

Hi there,
are there any updates available on the remoteHost topic?

I will try to rephrase the problem:

SE_OPTS is not an adequat substitute for the REMOTE_HOST because:

• SE_OPTS overwrites the node port settings. Whereas

• REMOTE_HOST allowed different internal and external ports. (ie selenium starts on 5555; portmapping that to 34567; node tells hub to use the latter)

Greetings

[McSnowflake]

@braindonor Are you using docker version 3?

Because I thought REMOTE_HOST param has been removed?
And when I try it REMOTE_HOST is ignored.

[McSnowflake]

Selenium bug report created: #4501

[braindonor]

I'm using Docker 1.11.2 & Selenium 3.4.0

[McSnowflake]

When I passed REMOTE_HOST into the container, I can't find it afterwards:

In entry_point.sh I do: export REMOTE_HOST="http://$NODE_HOST_IP:$EXT_NODE_PORT"
Inside the containers shell via env, I can't find that environment variable, which propably means that it get's cleared...

[braindonor]

apologies. your'e correct, I posted the old script. Here's the V3 script:

docker run -d -e GRID_MAX_SESSION=1 -e GRID_TIMEOUT=250000 -e GRID_UNREGISTER_IF_STILL_DOWN_AFTER=250000 -P -p $HUB_PORT:$HUB_PORT --name selenium-hub serverxxx:5000/selenium-hub:3.4.0

docker run -d -e "SE_OPTS=-host $(hostname -s) -port 555${COUNT}" -e HUB_PORT_4444_TCP_ADDR=$HUB_HOST -e HUB_PORT_4444_TCP_PORT=${HUB_PORT} -P -p 555${COUNT}:555${COUNT} --dns=10.232.217.1 --dns-search=xxxxxx.com --name selenium-ff-node-${SUFFIX}${COUNT} serverxxx:5000/selenium-ff-node:3.4.0

[McSnowflake]

@braindonor Hi, thx for the reply. You are using the same internal and external port, which is exactly the restriction, that I'm struggeling with.

[aairey]

I am also facing this issue on AWS ECS.
But it goes for any orchestration tool.

Basicly if you want to spawn more containers (which is the docker philosophy) running firefox or chrome nodes using the same image on one or multiple docker hosts, it is not possible anymore.

I have managed to create a separate entrypoint to set the host and port (SE_OPTS=-host $HOST -port $EXPOSED_PORT), but Selenium does not seem to like this. It reports it's host and port fine to the hub but it does not allow the hub to connect because it is actually listening on 5555.
And thus you get a Connection refused.

We should be able to set the host:port combo towards the "outside world" as an ENV var again, or similar.

[diemol]

Hi, I've spent some time checking this, and here is what I found:

The images deprecated the use of REMOTE_HOST because it was used in the JSON config for the node, and that value is not supported in the JSON node config anymore, but it is supported as a flag in the command line.

Bottom line is that you can still use -remoteHost, but via SE_OPTS. Here is an example, where I start a hub normally and a node that uses the port 5555 internally but maps to the external 5595. The -remoteHost passes the url the hub will use to reach the node, and it just works the way it worked before when it was an env var.

Start hub:

docker run -p 4444:4444 --name selenium-hub selenium/hub:3.7.1-beryllium

Start node:

docker run --name node101 -p 5595:5555 \
    -e HUB_PORT_4444_TCP_ADDR=192.168.1.7 \
    -e HUB_PORT_4444_TCP_PORT=4444 \
    -e SE_OPTS="-remoteHost http://192.168.1.7:5595" \
    selenium/node-chrome:3.7.1-beryllium

Please give it a try and report back if the problem was solved, so we can close this issue.

[McSnowflake]

Hi @diemol ,
thanks for your afforts.

I tried what you suggested, but unfortunately it didn't work.

Selenium is reporting this:
Exception in thread "main" com.beust.jcommander.ParameterException: Unknown option: -remoteHost at com.beust.jcommander.JCommander.parseValues(JCommander.java:742) at com.beust.jcommander.JCommander.parse(JCommander.java:282) at com.beust.jcommander.JCommander.parse(JCommander.java:265) at com.beust.jcommander.JCommander.<init>(JCommander.java:210) at org.openqa.grid.selenium.GridLauncherV3$3.setConfiguration(GridLauncherV3.java:267) at org.openqa.grid.selenium.GridLauncherV3.buildLauncher(GridLauncherV3.java:155) at org.openqa.grid.selenium.GridLauncherV3.main(GridLauncherV3.java:75)

[diemol]

Hi @McSnowflake,

Can you show me the exact command you used? You can change the IP you used of course :)

[McSnowflake]

Hi again @diemol ,

this is the command I used:

java -cp "selenium-server-standalone.jar:another.jar" org.openqa.grid.selenium.GridLauncherV3 -role node -nodeConfig /opt/selenium/config.json -remoteHost http://ip:port -proxy anotherAccessProxy

I upgraded to selenium 3.7.1 and I don't see the error mentioned above anymore. So I guess, support for remoteHost param moved back in along the way?

[diemol]

Oh, I thought you were using the docker containers. Anyway, if it works for you now with the latest release I am happy :)
I am not sure about the recent history of the parameter since I don't use it, I just saw that it is working after checking the latest version and reading what the params do.

@desertaxle @discobean @aairey @tparikka
Please try again based on the latest comments and let us know if it worked for you. If you find any problems still, please feel free to reopen this issue.

[McSnowflake]

@diemol,
I am actually running that in docker, but have to attach the remoteHost param in the entrypoint-script because only after starting the container is the actual portmapping available.

I use rancher to spawn selenium nodes and rancher maps its ports to a random exposed port, which is only known after the container started. So I tweaked the original selenium docker images to get the exposed port after starting and pass it to selenium with the command I posted above.

But this broke after upgrading to selenium 3.

Thanks again for looking into it and
Have a great weekend,
McSnowflake

[aairey]

I need to do the same for AWS ECS. So I have custom images with custom
entrypoint scripts based on Selenium's upstream ones.

On Sat, 25 Nov 2017, 09:31 McSnowflake, [e-mail] wrote:

@diemol https://github.com/diemol,
I am actually running that in docker, but have to attach the remoteHost
param in the entrypoint-script because only after starting the container is
the actual portmapping available.
I use rancher to spawn selenium nodes and rancher maps its ports to a
random exposed port, which is only known after the container started.
That's why I need the remoteHost param.

Thanks again for looking into it and
Have a great weekend,
McSnowflake

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SeleniumHQ/docker-selenium/issues/481#issuecomment-346926740,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHSjvjYFVHl1zfss6YJdFb0JLOoFYAARks5s59BFgaJpZM4NW_0h
.

[ArSn]

@aairey do you mind sharing your custom entrypoint for other poeple that have the very same problem (like me) :)

[aairey]

Sure, I got it from here: https://groups.google.com/d/msg/selenium-users/PYX1fYqNGqc/zOuFIF4JBgAJ

Here is our Dockerfile for Firefox (s/firefox/chrome/ for a chrome node):

# Image that runs Firefox node to connect to a Selenium Grid instance
# See: https://groups.google.com/d/msg/selenium-users/PYX1fYqNGqc/zOuFIF4JBgAJ
FROM docker.io/selenium/node-firefox:3.9.1
LABEL maintainer "Your Name <[email protected]>"

# Python is a dependency of getting the dynamicly mapped port on AWS
RUN sudo apt-get update && \
    sudo apt-get install -y python python-pip socat && \
    sudo apt-get clean && \
    sudo -H pip install --upgrade pip && \
    sudo -H pip install boto boto3 requests datetime

COPY ecs-get-port-mapping.py \
     override_entrypoint.sh \
     /opt/bin/

CMD ["bash", "-x", "/opt/bin/override_entrypoint.sh"]

The override_entrypoint.sh script:

#!/bin/sh
set -o errexit
set -o xtrace

export EC2_HOST=$(wget -O - http://169.254.169.254/latest/meta-data/local-ipv4 2> /dev/null)
result="$(python /opt/bin/ecs-get-port-mapping.py)"
eval "$result"

# Selenium specifics
export NODE_PORT=$PORT_TCP_5555
export SE_OPTS="-host $EC2_HOST -port $PORT_TCP_5555"
( sudo socat TCP4-LISTEN:5555,fork TCP4:localhost:$PORT_TCP_5555 ) &

exec /opt/bin/entry_point.sh

The ecs-get-port-mapping.py:

import os

import boto.utils
import boto3
import requests
import datetime
import time


def get_contents(filename):
    with open(filename) as f:
        return f.read()


def get_ecs_introspection_url(resource):
    # 172.17.0.1 is the docker network bridge ip
    return 'http://172.17.0.1:51678/v1/' + resource


def contains_key(d, key):
    return key in d and d[key] is not None


def get_local_container_info():
    # get the docker container id
    # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-introspection.html
    docker_id = os.path.basename(get_contents("/proc/1/cpuset")).strip()

    if docker_id is None:
        raise Exception("Unable to find docker id")

    ecs_local_task = requests.get(get_ecs_introspection_url('tasks') + '?dockerid=' + docker_id).json()

    task_arn = ecs_local_task['Arn']

    if task_arn is None:
        raise Exception("Unable to find task arn for container %s in ecs introspection api" % docker_id)

    ecs_local_container = None

    if contains_key(ecs_local_task, 'Containers'):
        for c in ecs_local_task['Containers']:
            if c['DockerId'] == docker_id:
                ecs_local_container = c

    if ecs_local_container is None:
        raise Exception("Unable to find container %s in ecs introspection api" % docker_id)

    return ecs_local_container['Name'], task_arn


def main():

    metadata = boto.utils.get_instance_metadata()
    region = metadata['placement']['availability-zone'][:-1]  # last char is the zone, which we don't care about

    ecs_metadata = requests.get(get_ecs_introspection_url('metadata')).json()
    cluster = ecs_metadata['Cluster']

    container_name, task_arn = get_local_container_info()

    # Get the container info from ECS. This will give us the port mappings
    ecs = boto3.client('ecs', region_name=region)
    response = ecs.describe_tasks(
        cluster=cluster,
        tasks=[
            task_arn,
        ]
    )

    task = None
    if contains_key(response, 'tasks'):
        for t in response['tasks']:
            if t['taskArn'] == task_arn:
                task = t

    if task is None:
        raise Exception("Unable to locate task %s" % task_arn)

    container = None
    if contains_key(task, 'containers'):
        for c in task['containers']:
            if c['name'] == container_name:
                container = c

    if container is None:
        raise Exception("Unable to find ecs container %s" % container_name)

    if contains_key(container, 'networkBindings'):
        for b in container['networkBindings']:
            print("export PORT_%s_%d=%d;" % (b['protocol'].upper(), b['containerPort'], b['hostPort']))


if __name__ == '__main__':
    main()

[aairey]

Had to change 'DockerId' to 'DockerID' yesterday to make it work again.
Guess AWS changed something in their introspection API, even their own docs were not reflecting this yesterday ...