Nomad: Docker `dns_servers` is not reflected in containers created

[lawliet89]

Nomad version

Nomad v0.7.0

Operating system and Environment details

Ubuntu 16.04 with Docker 17.11.0-ce

Issue

Consider the Job file below. I would expect the container created to have the DNS server included.

nomad inspect hashi-ui shows that the DNS servers are listed under the task config.

If I go to the Nomad worker node with the container deployed, and do a docker inspect, I get the following:

[
    {
        "Id": "bdccf6043303055b4f9d016e3ae07e2a73a89ce14b48f44e1ccc46157a5a01a2",
//...
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
 // snip
]

Notice that it says "Dns": null. I would have expected "Dns": ["127.0.0.1:8600"].

Doing any DNS lookup does not use the DNS server I specified, but whatever the host has as its DNS server.

Job file (if appropriate)

job "hashi-ui" {
  datacenters = ["ap-southeast-1a", "ap-southeast-1b"]
  region      = "ap-southeast-1"
  type        = "service"

  group "server" {
    count = 1

    task "hashi-ui" {
      driver = "docker"

      config {
        image = "jippi/hashi-ui:v0.22.0"
        // network_mode = "host"
        port_map {
          http = 3000
        }

        dns_servers = ["127.0.0.1:8600"]
      }

      service {
        port = "http"

        check {
          type     = "http"
          path     = "/"
          interval = "10s"
          timeout  = "2s"
        }
      }

      env {
        NOMAD_ENABLE = 1
        // NOMAD_ADDR   = "http://http.nomad.service.consul:4646"
        NOMAD_ADDR = "http://10.3.26.137:4646"

        // CONSUL_ENABLE = 1
        // CONSUL_ADDR   = "consul.service.consul:8500"

      }

      resources {
        cpu    = 500
        memory = 512

        network {
          mbits = 5

          port  "http"{}

          // port "http" {
          //   static = 3000
          // }
        }
      }
    }
  }
}

[schmichael]

Unfortunately dns_servers only supports IP addresses due to a limitation of resolv.conf on Linux not support arbitrary ports. You should see this error in your client logs:

2017/12/11 11:28:02.816594 [ERR] driver.docker: invalid ip address for container dns server: 127.0.0.1:8600

The recommended way to use Consul for DNS is to use dnsmasq: https://www.consul.io/docs/guides/forwarding.html

Sorry for the hassle!

[aschofie]

@schmichael I have my nomad client hosts setup with dnsmasq pointing at the consul agent, and

dns_servers = ["${attr.unique.network.ip-address}"]

is working-- I can go into a container and see /etc/resolv.conf is pointing at the nomad client's IP. Are there any problems with this, or is there a better/preferred way of setting up containers with consul?