Nomad: [feature] Add support (and documentation) for consul ingress gateways

Created on 26 Jun 2020 · 3Comments · Source: hashicorp/nomad

It would be great (given the usage of consul & connect in nomad) to have first class support for ingress gateways (https://www.consul.io/docs/connect/ingress-gateway) in Nomad. For an outsider it kinda feels weird that all the examples (https://www.hashicorp.com/blog/ingress-gateways-in-hashicorp-consul-1-8/) are either bare-bones or kubernetes specific. You've got a great scheduler here, make use of it :)

themconsul themconsuconnect typenhancement

Source

apollo13

👍9 🎉1

Most helpful comment

Here's a full group definition for something that works:

  group "web" {
    network {
      mode = "bridge"
      port "http" { to = 80 }
    }

    service {
      name = "web"
      tags = ["http"]
      port = "http"

      connect {
        sidecar_service {
          proxy {
            config {
              protocol = "http"
            }
            local_service_port = 80
          }
        }
      }

      check {
        type = "http"
        path = "/"
        interval = "10s"
        timeout = "10s"
      }
    }

    task "web" {
      driver = "docker"

      meta {
        version = 49
      }

      config = {
        image = "fill in the blanks"
      }
    }
  }

manveru on 17 Jul 2020

👍2

All 3 comments

I made a proof of concept where I add the connect->sidecar_service to my nomad job's group->service definition.

// File: vault.nomad
// job->group->service
  service {
      name = "vault"
      port = "http"
      tags = ["http"]
      connect {
        sidecar_service {}
      }
    }

Then following the tutorial here (https://learn.hashicorp.com/consul/developer-mesh/ingress-gateways) to create an ingress-gateway service and run consul connect envoy -kind ingress-gateway .... manually.

A fair approach would be if there was a simple way to run the envoy ingress-gateway process as a job in Nomad (maybe there already is?). Then as long as nomad job group services have the sidecar_service enabled we could deploy the ingress-gateway using a nomad job separately, defining the port and listener etc.

Another issue I encountered is that in our use case we want one ingress-gateway routing to multiple services which can only be achieved with HTTP, and the services created by Nomad jobs use TCP by default in Consul. I then used a service-defaults for each service to make it HTTP afterwards, which is a bit cumbersome but not a showstopper.

// File: vault-service-defaults.hcl
Kind      = "service-defaults"
Name      = "vault"
Protocol  = "http"

Definition of an ingress-gateway could then look like:

// File: ingress-gateway.hcl
Kind = "ingress-gateway"
Name = "ingress-service"

Listeners = [
 {
   Port = 80
   Protocol = "http"
   Services = [
     {
       Name = "vault"
     },
     {
       Name = "other-service"
     }
   ]
 }
]

and run consul config write ingress-gateway.hcl.

jlarfors on 26 Jun 2020

👍2

Here's a full group definition for something that works:

  group "web" {
    network {
      mode = "bridge"
      port "http" { to = 80 }
    }

    service {
      name = "web"
      tags = ["http"]
      port = "http"

      connect {
        sidecar_service {
          proxy {
            config {
              protocol = "http"
            }
            local_service_port = 80
          }
        }
      }

      check {
        type = "http"
        path = "/"
        interval = "10s"
        timeout = "10s"
      }
    }

    task "web" {
      driver = "docker"

      meta {
        version = 49
      }

      config = {
        image = "fill in the blanks"
      }
    }
  }

manveru on 17 Jul 2020

👍2

Support for ingress gateways was added in https://github.com/hashicorp/nomad/pull/8709
https://www.nomadproject.io/docs/job-specification/gateway#gateway-stanza