Consul: Consul Connect: Proxy to route by name?

[aclowkey]

Let's say for example, I have 2 services: API, ElasticSearch. The way to connect them is to: install client on each node, and a proxy. Then, when API wants to talk to ElasticSearch, it will connect to localhost:9200, which will reach the proxy, then routed to ElasticSearch.

Now let's say I don't want to know port numbers, I just want to access elasticsearch.service.consul, but through the proxy, is this possible?

What is the best approach to this? If my API has 10,000 upstream microservices, does it need to remember 10,000 different ports?

Also, do I have to define upstream for each service? This becomes a problem if I have many many ,and they're constantly changing. Doesn't it beat the service discovery purpose?

[jonathanbeber]

That's a good issue. I had the same feeling when starting with connect.

On the roadmap preview blog post, they said that:

the data plane exposes upstream services to your application via local ports. A new feature of Consul introduces the ability for advanced layer 7 routing; traffic can be routed to different upstream services based on Path or HTTP header.

and more:

While a 1-1 relationship between a local port and upstream service is fine for many purposes, the capability for HTTP routing makes it easy to build aggregated services such as API gateways or to replace centralized load balancers with a smarter decentralized approach.

I really believe that's on the roadmap.

Maybe this issue is good to discuss the fact Connect has a lot of expectations on it and I didn't find an easy way to have access to its design document. Include on the documentation, a design document for the product would help to address this kind of questions.

[wbertelsen]

+1 to this. In our self-configured envoy mesh, we use the cluster_header routing rule supported by envoy to route dynamically based on a header. It would be nice if connect also supported this

virtual_hosts:
  - name: cloud_service
    domains: ["*"]
    routes:
    - match: { prefix: "/" }
      route: { cluster_header: service-name } # This routes to remote clusters named by the svc-name header

[chrisboulton]

You should be able to do this today, assuming I understand -- it's similar to a use case that we have for Connect.

When you configure your proxy sidecar, configure it with a single upstream, say http-out:

"connect": {
  "sidecar_service": {
    "proxy": {
      "upstreams": [
        {
          "destination_name": "http-out",
          "local_bind_port": 4140
        }
      ]
    }
  }
}

Create a service-defaults for http-out:

kind = "service-defaults"
name = "http-out"
protocol = "http"

Create a service-resolver that maps to concrete services: (this example uses :authority/Host headers):

kind = "service-router"
name = "http-out"
routes = [
  {
    match {
      http {
        header = [
          {
            name = ":authority"
            exact = "backend-service-1"
          }
        ]
      }
    }

    destination {
      service = "backend-service-1"
    }
  },
  {
    match {
      http {
        header = [
          {
            name = ":authority"
            exact = "backend-service-2"
          }
        ]
      }
    }

    destination {
      service = "backend-service-2"
    }
  },
]

Update the service-router when you add new services. (it would be nice to be able to capture groups or values and use them, but that doesn't seem to be supported today)

[uc-dang-tiki]

I encounter this problem when I try to use consul connect. Hope Consul team could add it soon.