Nomad: Nomad changes IPv6 to IPv4 when specifying port

[tsujp]

Nomad doesn't allow port_map or static ports using IPv6 currently, as using it with the docker driver for advertising IPv6 addresses forces IPv4 instead.

Is there a workaround for this available _now_?

[tgross]

Hi @tsujp! It might help us understand the problem you're seeing a bit better if you could provide a jobspec and the expected vs actual behavior.

The service section of the jobspec docs has an example of IPv6 container usage that you might find helpful as well.

[hoh]

Hi, I am having what I believe to be the same issue.

My goal is to run a service that is reachable using both IPv4 and IPv6 on a public interface, using the Docker driver.

Actual behaviour

As an example, I will use theIPv6 service section of the jobspec docs with Nomad v0.10.2 (with minor fix #6809 ).

When I run it, Docker binds only on the IPv4 address. docker ps gives:
a46df48e46ce redis:3.2 "docker-entrypoint.s…" 24 minutes ago Up 24 minutes {{public-ipv4-address}}:31086->6379/tcp, {{public-ipv4-address}}:31086->6379/udp redis-d6567fea-4376-36b0-fcbc-9ae562ee974f

The address for the allocation in Nomad's web interface is the IPv4 address {{public-ipv4-address}}:31086.

Expected behaviour

Let's consider the situation when running the service directly from the Docker command line:

docker run --rm -p [::]:31086:5678 redis:3.2

In this case, the service is available from both IPv4 and IPv6, and docker ps gives:
36cde442ff4a redis:3.2 "docker-entrypoint.s…" 11 minutes ago Up 11 minutes :::31086->5678/tcp, 6379/tcp agitated_meninsky

Additional notes

The second example using advertise_ipv6_address with address_mode=driver works to run services in containers and advertise their address in Consul.

[tgross]

Using the sample jobspec from https://www.nomadproject.io/docs/job-specification/service.html#ipv6-docker-containers I can confirm the behavior.

Created via the Docker CLI:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                          NAMES
9ed7439dba11        redis:3.2           "docker-entrypoint.s…"   22 seconds ago      Up 21 seconds       6379/tcp, :::31086->5678/tcp   tender_turing

Created via Nomad:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
9147fb844811        redis:3.2           "docker-entrypoint.s…"   9 seconds ago       Up 7 seconds        10.0.2.15:30251->6379/tcp, 10.0.2.15:30251->6379/udp   redis-4f375cbe-c628-3e5d-06af-5611cdef4690

Relevant section of client logs (trace-level), around the point where we call out to the Docker driver:

...
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=redis:3.2 image_id=sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916 references=1
2019-12-20T19:52:21.269Z [TRACE] client.driver_mgr.docker: binding volumes: driver=docker task_name=redis volumes=[/tmp/NomadClient537557249/4f375cbe-c628-3e5d-06af-5611cdef4690/alloc:/alloc, /tmp/NomadClient537557249/4f375cbe-c628-3e5d-06af-5611cdef4690/redis/local:/local, /tmp/NomadClient537557249/4f375cbe-c628-3e5d-06af-5611cdef4690/redis/secrets:/secrets]
2019-12-20T19:52:21.269Z [TRACE] client.driver_mgr.docker: no docker log driver provided, defaulting to json-file: driver=docker task_name=redis
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=redis memory=268435456 cpu_shares=500 cpu_quota=0 cpu_period=0
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=redis binds="[]string{"/tmp/NomadClient537557249/4f375cbe-c628-3e5d-06af-5611cdef4690/alloc:/alloc", "/tmp/NomadClient537557249/4f375cbe-c628-3e5d-06af-5611cdef4690/redis/local:/local", "/tmp/NomadClient537557249/4f375cbe-c628-3e5d-06af-5611cdef4690/redis/secrets:/secrets"}"
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=redis
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: allocated mapped port: driver=docker task_name=redis ip=10.0.2.15 port=30251
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: exposed port: driver=docker task_name=redis port=6379
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=redis labels=map[com.hashicorp.nomad.alloc_id:4f375cbe-c628-3e5d-06af-5611cdef4690]
2019-12-20T19:52:21.269Z [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=redis container_name=redis-4f375cbe-c628-3e5d-06af-5611cdef4690
2019-12-20T19:52:21.319Z [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=9147fb84481158166fca1da2c635eb36f059b862c21a7223533d20a909c88fc5
2019-12-20T19:52:21.425Z [DEBUG] client: updated allocations: index=13 total=1 pulled=0 filtered=1
2019-12-20T19:52:21.425Z [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
2019-12-20T19:52:21.425Z [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
2019-12-20T19:52:21.803Z [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=9147fb84481158166fca1da2c635eb36f059b862c21a7223533d20a909c88fc5
2019-12-20T19:52:21.803Z [DEBUG] client.driver_mgr.docker.docker_logger: starting plugin: driver=docker path=/opt/gopath/bin/nomad args=[/opt/gopath/bin/nomad, docker_logger]
2019-12-20T19:52:21.803Z [DEBUG] client.driver_mgr.docker.docker_logger: plugin started: driver=docker path=/opt/gopath/bin/nomad pid=6104
2019-12-20T19:52:21.804Z [DEBUG] client.driver_mgr.docker.docker_logger: waiting for RPC address: driver=docker path=/opt/gopath/bin/nomad
2019-12-20T19:52:21.809Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: plugin address: driver=docker address=/tmp/plugin105688170 network=unix @module=docker_logger timestamp=2019-12-20T19:52:21.809Z
2019-12-20T19:52:21.809Z [DEBUG] client.driver_mgr.docker.docker_logger: using plugin: driver=docker version=2
2019-12-20T19:52:21.810Z [DEBUG] client.driver_mgr.docker.docker_logger.nomad: using client connection initialized from environment: driver=docker @module=docker_logger timestamp=2019-12-20T19:52:21.810Z
...