Nomad: Host networking no longer working with non-deprecated syntax for network stanza

[apollo13]

Nomad version

Nomad v0.12.3 (2db8abd9620dd41cb7bfe399551ba0f7824b3f61)

Issue

This is a follow up to #8746 from a technical perspective. Setting mode=host on a network stanza in the group stanza does not seem to setup host networking.

Reproduction steps

$ ./nomad run example.nomad
$ docker ps|grep redis
$ docker inspect id_of_the_container
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "2e238cb6b229ca6f4f9b44fcfe9190f7d2ddd69f2178592825148a56f2348417",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "6379/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/2e238cb6b229",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "23314473eb8321885638cf1dee3d5f8cc5a2eb4b2dfd71d371041c41908e8634",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "207284846d01434486e362847159d663c1bbbb820ceec09c222b56338acba46d",
                    "EndpointID": "23314473eb8321885638cf1dee3d5f8cc5a2eb4b2dfd71d371041c41908e8634",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }

Job file (if appropriate)

job "example" {
  datacenters = ["dc1"]

  group "cache" {
    network {
      mode = "host"
      mbits = 10
      port "db" {}
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"
      }

      resources {
        cpu    = 500
        memory = 256

      }
    }
  }
}

[tgross]

Hi @apollo13! We'll look into this.

[nickethier]

Hey @apollo13

For docker you'll still need to set the docker driver config for network_mode=host.

I can see the confusion here and I'll update the documentation to try and illustrate the difference.

Essentially the Nomad network mode=host tells Nomad not create a network namespace. For other drivers such as exec this uses the host network namespace. Since docker has other networking feature that predate Nomad network modes, we didn't want to create a scenario where the default functionality changed.

Does this make sense?

[apollo13]

Kinda, but it is confusing. Ie can you imagine a usecase where this split between the network modes would make sense? (group -> network has host but the docker network_mode would be bridge?). Or what would that entail if group -> network has bridge and docker network_mode is also the default bridge -- double network?

[apollo13]

I just tested a few things on 0.12.4 and it seems to work as I'd expect it (I guess). The main reason for the confusion is probably that non-sensical configurations like:

job "example" {
  datacenters = ["dc1"]
  group "cache" {
    count = 1

    network {
      port "db" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        network_mode = "host"
        image = "redis:3.2"
      }
    }
  }
}

validate & start fine, the UI display a mapped port ala:

but since everything is in host mode there is no port mapping done. Since this does not create an error anywhere it is kinda easy to miss it when shuffling network sections around -- does that make any sense @nickethier ?

[nickethier]

Hey @apollo13 thanks for the detailed write up. I think I see what you're getting at. The bane of Nomad being flexible in supporting multiple task drivers means UX issues like this are sometimes hard to reason about. This isn't so non-nonsensical to Nomad, allocate a host port and declare that its going to get forwarded to 6379 then run a docker task. A driver's config block is opague to Nomad at job submission because its driver dependent, so the server has no idea that network_mode="host" for the task much less what that means.

I'm open to ideas and suggestions on how to make this UX a bit clearer.

Closing the issue but feel free to keep the discussion going here.