Jupyterhub: Question: IPs and ports in bridge mode

[Spritekin]

Hi,
I have my jupyterhub running in docker using host networking. I need to move the hub to use AWS ECS and I don't have the hosted mode available.
So, I need to setup all to use bridge mode and I have some problems I hope someone may give me an idea.
I use Dockerspawner to run containers in a Swarm cluster. All goes ok in host network mode because I can specify my IPs to be the same as in the host:

  --JupyterHub.proxy_api_ip='172.31.45.231' \
  --JupyterHub.ip='172.31.45.231' \
  --JupyterHub.hub_ip='172.31.45.231' \
  --DockerSpawner.hub_ip_connect='172.31.45.231' \

Also, ports are all open so it can just take any port. All fine.

But the moment I set the bridge mode, that IP is no longer valid form the container point of view. I tried to change the first three IPs to use local IPs like:
docker run \
-e DOCKER_HOST=http://swarm.marathon.cdt-prod:2376 \
-p 8000:8000 \
-p 8001:8001 \
-p 8080:8080 \
-p 8081:8081 \
jupyterhub/jupyterhub \
--JupyterHub.proxy_api_ip='0.0.0.0' <<< or 127.0.0.1
--JupyterHub.ip='my.hub.server.com' <<<< or 172.31.45.231 or 0.0.0.0 or 127.0.0.0
--JupyterHub.hub_ip='0.0.0.0' <<< or 127.0.0.1
--DockerSpawner.hub_ip_connect='172.31.45.231' \

In other words, I have tried to let it start its networking as it was a localhost, then opening the ports I think may be used inside the container.

And it seems to start ok:

[I 2017-05-01 06:11:09.327 JupyterHub app:1453] Hub API listening on http://127.0.0.1:8081/hub/
[W 2017-05-01 06:11:09.329 JupyterHub app:1174] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2017-05-01 06:11:09.329 JupyterHub app:1176] Starting proxy @ http://*:8000/
[D 2017-05-01 06:11:09.329 JupyterHub app:1177] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--default-target', 'http://127.0.0.1:8081', '--error-target', 'http://127.0.0.1:8081/hub/error']
06:11:09.504 - info: [ConfigProxy] Proxying http://*:8000 to http://127.0.0.1:8081

Then I connect to the hub, I login successfully, then the container is created, but then I get:

Failed to connect to Hub API at 'http://172.31.45.231:8081/hub/api'.  Is the Hub accessible at this URL (from host: 2b8601a5005b)?

I think it is the remote docker container trying to connect to the hub but it can't. Its like the hub is listening but the port is not there.

So my question, Is there some specific IP/port configuration that I should be using?

Thanks!

[willingc]

Have you set this in your jupyterhub_config.py:

c.DockerSpawner.container_ip = "0.0.0.0"

Reference

Please share your networking and spawner configuration from jupyterhub_config.py and we can be more helpful.

[Spritekin]

Hi @willingc
Yes, I have this option and I think it affects the notebooks only. As I said the hub works fine in docker networking host mode and the notebooks can connect back to the hub. No problem there.

The problem is when I run the hub in docker bridge networking then the notebooks can't connect back to the hub (with the error mentioned above). Is like the 8081 port is closed or just not listening, even if I open all the ports using -P.

Thanks.

[Spritekin]

Ok I managed to make it work, I do not need to specify the --JupyterHub.ip, essentially just:
--JupyterHub.proxy_api_ip='0.0.0.0' \
--JupyterHub.hub_ip='0.0.0.0' \