Nomad: Add interpolation for driver IP address of another task within the group

[mildred]

Nomad version

Nomad v0.8.6 (ab54ebcfcde062e9482558b7c052702d4cb8aa1b+CHANGES)

Feature Request

There is currently no way to interpolate another task driver IP address within the same group. By driver IP address, I mean the IP address associated to the docker container within the docker network. The same address as advertised by address_mode = driver within a service stanza.

What would be nice is to be able to write ${NOMAD_DRIVER_IP_taskname} and have it evaluate to the IP address of the container for the task taskname.

The primary use case for this would be to use consul-connect within nomad and have a fixed port number for the proxied service. Having a fixed port can be sometimes necessary when the service being run does not accept communicating on another port number than the standard one. For example, many DNS clients refuse to connect to DNS servers on other ports than port 53.

Example use-case nomad job:

job "web" {
  type = "service"

  datacenters = ["dc1"]

  group "main" {
    task "main" {
      driver = "docker"

      config {
        # The web service is hardcoded to connect to its database on database:5432
        image = "web"

        extra_hosts = [
          "database:${NOMAD_DRIVER_IP_proxy}",
        ]

        port_map {
          http = 8080
        }
      }

      resources {
        cpu    = 100
        memory = 128
        network {
          mbits = 10
          port "http" {}
        }
      }
    }

    task "proxy" {
      driver = "docker"

      config {
        # here we have a connect proxy that listens on 8080 within the docker
        # container for incoming requests for the web service, and it is also
        # listening on port 5432 to proxy db requests coming from the web
        # service.
        image   = "consul"
        command = "consul"
        args    = [
            "connect", "proxy",
            "-service", "web",
            "-upstream", "db:5432",
            "-service-addr", "${NOMAD_ADDR_main_http}",
            "-listen", ":8080",
            "-register",
        ]
        port_map {
          web = 8080
          db  = 5432
        }
      }

      resources {
        network {
          port "web" {}
          port "db" {}
        }
      }
    }
  }
}

[Dirrk]

Not that this changes your feature request but wouldn't this be possible by using template with env true?

[preetapan]

@mildred We have network namespace support and Consul connect integration in the roadmap. With network namespace support, this should be possible. We will need to flesh out the details of how that would work but at a high level port_map could move to become a task group property so that tasks could see each other's port map.

[mildred]

Yep, some kind of consul-connect integration would solve this use case :) The suggested solution here was just an easy way to get connect easily in Nomad

[djenriquez]

Just wanted to add support for this request: adding sidecars can be problematic without this solution using the Docker driver. For example:

• Service A listens on port 80 and needs to route traffic through a proxy sidecar
• The proxy sidecar is sent requests through by a hostname, so it needs to listen to port 80 as well (cant allocate ports through hostname resolution)

I can:

1. Use container:name networking and send requests to localhost, but now I need to reserve port 80 for the proxy sidecar, eliminating the ability to run any service that listens on port 80 with that proxy sidecar (cant bind the same port on the same container network)
2. Route to the proxy sidecar through the host's IP:DynamicPort, but now i've introduced port configurability requirements for the service to communicate with upstreams, rather than simply using a hostname (i.e. http://foo:${NOMAD_PORT_upstream} vs http://foo)
3. Use the container:name and reserve some other port for the sidecar, but same problem as number 2, would need port interpolation on the service to upstreams
4. Use user-defined networks, which allow me to use network_aliases, but since Nomad cannot create user-defined networks, this becomes a non-starter

Number 2/3 isn't bad, but the goal is zero-touch compatibility for the code as its written today, these would break that goal. These also lose _some_ levels of security since now the same listener port for upstream egress on the sidecar needs to be exposed on the host level.

Number 4 is great, but I can see how user-defined network management is outside the scope of Nomad (or is it?).

However, IF we were able to use the driver IP, as @mildred suggested, we could add interpolation at extra_hosts such that foo resolves to the driver ip on the default docker bridge. This means we can use port 80 on both the service and the sidecar proxy and everyone wins.

Would love to have this interpolation variable 👍.

Context: we're using envoy as a sidecar proxy for a service-mesh powered by an in-house (hope to open source soon) xds-api that is scheduled by Nomad and utilizes Consul, its KV and intentions.

[djenriquez]

@preetapan you mentioned that port_map could be moved to become a task_group property, is this functionality available today?

[far-blue]

I'd like to add my vote to this issue.

We run using a flat IP space across our nomad nodes (using vxlan). Each node has a class D IP range and the class C range is incremented per node. So x.x.1.27 is a container running on node 1 while x.x.6.43 is running on node 6. Being a flat address space, all containers can talk to each other directly by IP.

We have various situations such as sidecars or apache/php task combinations where we are resorting to host port mapping or registering services in Consul and using DNS where it would be great to just be able to provide an ENV var or command parameter with the driver IP of the related task.

I'd prefer to not have to use port maps for this functionality because, having a flat, 'address_mode = driver' environment we don't have any need to map ports to the host.

Something like NOMAD_DRIVER_IP_<task> or even NOMAD_DRIVER_IP_<group>_<task> (which could provide a string of comma-separated IPs) would be great.

[djromberg]

I'd also like to add my vote. We currently have the following problem that is easily solvable by docker-compose:

1. A database (let's say MongoDB) that should be private to some API service and that listens on a fixed port
2. A public API service that uses the private database

How do you do this with Nomad? I don't want to map database ports to the host. I'd like to have the database task as sibling to my API task which in turn has a port_map for HTTP traffic and a service section defined to register to consul.
Currently we solve this by creating a bunch of user defined networks on each of our cluster nodes with a puppet script for these situations but I would really appreciate being able to interpolate the driver address of a task or to let nomad automatically create the docker network. I understand that the scope of Nomad should be kept small, however it already downloads artifacts, repulls latest images, etc. Auto-creating a Docker network via the corresponding driver would be an option that would be very valuable and would make life so much easier.

[sarneeh]

Me too! Nomad is awesome for a microservice infrastructure, but I went into trouble trying to do simple things like connecting nginx + php as docker containers. Being able to get IP's of containers within the task group would make it a lot easier.