Nomad: Static ports on different host networks should not conflict

[phreakocious]

Nomad version

Nomad v0.12.7 (6147cb578794cb2d0c35d68fe1791728a09bb081)

Operating system and Environment details

Debian 10

Issue

Static ports in jobs with the same number, but with different host_network settings are treated as conflicting. They should be able to run on the same host as they would bind to distinct IP addresses. It appears that this persists beyond the "in the same job" scenario, as launching different jobs on the same host with the same network configuration will fail due to resource exhaustion. Should I submit a separate issue for that?

Reproduction steps

ensure the nomad.hcl has some host_networks defined

client {
  enabled       = true
  network_speed = 1000
  host_network "net1" { 
    cidr = "10.9.9.1/32"
  }
  host_network "net2" { 
    cidr = "10.9.9.2/32"
  }
}

bring up a couple new dummy interfaces with our IPs

# ip link add name nomad1 type dummy
# ip link add name nomad2 type dummy

# ip addr add 10.9.9.1/32 dev nomad1
# ip addr add 10.9.9.2/32 dev nomad2

# ip link set up dev nomad1
# ip link set up dev nomad2

restart nomad, try to run our job

# systemctl restart nomad
# nomad job plan static-port-test.hcl
Error during plan: Unexpected response code: 500 (1 error occurred:
        * Task group group1 validation failed: 1 error occurred:
        * Task group network validation failed: 1 error occurred:
        * Static port 32767 already reserved by taskgroup network:net1_port_32767

Job file (if appropriate)

job "static-port-test" {
  region = "global"
  datacenters = ["dc1"]
  type = "batch"

  group "group1" {

    network {
      port "net1_port_32767" {
        static = 32767
        to = 32767
        host_network = "net1"
      }
      port "net2_port_32767" {
        static = 32767
        to = 32767
        host_network = "net2"
      }
    }

    task "net1_task" {
      driver = "docker"

      config {
        image = "alpine/socat:latest"
        args = [ "tcp-listen:32767", "file:/etc/hostname" ]
        ports = [ "net1_port_32767" ]
      }

      resources {
        cpu    = 20 # MHz
        memory = 16 # MB
      }
    }

    task "net2_task" {
      driver = "docker"

      config {
        image = "alpine/socat:latest"
        args = [ "tcp-listen:32767", "file:/etc/hostname" ]
        ports = [ "net2_port_32767" ]
      }

      resources {
        cpu    = 20 # MHz
        memory = 16 # MB
      }
    }
  }
}

[shoenig]

Thanks for pointing this out, @phreakocious. Indeed our static port reuse detection predates multi-networks, and we'll need to do some additional plumbing in the scheduler + jobspec validation to enable this. I think we can cover both of those with this issue.