Consul: Integrate consul with envoy proxy

[bcba25]

Hello. I'm currently trying to link consul with my envoy proxy in order to obtain cluster hosts dynamically.

This is my envoy sample config

admin:
    access_log_path: /tmp/admin_access.log
    address:
      socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
    listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8080 }
      filter_chains:
      - filters:
        - name: envoy.http_connection_manager
          config:
            codec_type: auto
            stat_prefix: ingress_http
            route_config:
              name: local_route
              virtual_hosts:
              - name: local_service
                domains: ["*"]
                routes:
                - match: { prefix: "/App.Grpc.kGRPCService" }
                  route:
                    cluster: main_service
                    max_grpc_timeout: 0s
                - match: { prefix: "/App.Grpc.NotificationsService" }
                  route:
                    cluster: notifications_service
                    max_grpc_timeout: 0s
                cors:
                  allow_origin:
                  - "*"
                  allow_methods: GET, PUT, DELETE, POST, OPTIONS
                  allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                  max_age: "1728000"
                  expose_headers: custom-header-1,grpc-status,grpc-message
            http_filters:
            - name: envoy.grpc_web
            - name: envoy.cors
            - name: envoy.router
    clusters:
    - name: main_service
      connect_timeout: 0.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      hosts: [{ socket_address: { address: host.docker.internal, port_value: 50053 }}]
    - name: notifications_service
      connect_timeout: 0.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      hosts: [{ socket_address: { address: host.docker.internal, port_value: 50054 }}] 

So my question is how can I replace for example hosts: [{ socket_address: { address: host.docker.internal, port_value: 50053 }}] this part with information from consul ?
I'm sorry I haven't found enough information on how to integrate both.
Currently when my 2 services starting, I successfully registering them in consul and monitor their health.

[crhino]

@bcba25 Is the goal here to provide an edge proxy that routes to various Consul services? Are you using Connect or no?

We currently are working on implementing a basic Envoy Ingress functionality for Connect-based services, see https://github.com/hashicorp/consul/issues/5695#issuecomment-539185301.

Alternatively, you might be able to use consul-template to retrieve the desired configuration from Consul and template it into an Envoy config.

[bcba25]

@crhino thanks for trying to assist on the issue. I haven't found any information or example of how to connect them together. I'm not using consul connect or templates as well. I was told that I need to somehow specify url for consul server inside envoy configuration and envoy will automatically catch the hosts and load balance between them, thats my goal basically.

[bcba25]

Seriously, is there any default example on how to integrate with envoy-proxy ? At least a basic one? You guys have included in your documentation section about integration but never provided any examples 😞

[crhino]

@bcba25 Sorry for the long wait time. You're right, we don't have much documentation around using Envoy in this way, pretty much all of our documentation around Envoy is related to using it as a Connect proxy.

That said, I can see a couple of options here:

1. Replace address: host.docker.internal with address: <service-name>.service.consul, putting in the correct Consul DNS name for the service lookup. This relies on using Consul DNS for service discovery, and does hardcode the DNS address into the Envoy config. However, it is pretty simple to setup.
2. Use consul-template to dynamically change the Envoy configuration file and then reload it in Envoy. This essentially replicates the flow consul-template uses with HAProxy and Nginx. Here you would change the cluster type to STATIC and have the hosts field look something like this:

hosts:{{range service "notifications"}}
- { socket_address: { address: {{.Address}}, port_value: {{.Port}} }}{{end}}

That should template out each individual IP address of a healthy notifications service.

Unfortunately, non-Connect services are not discoverable via EDS (endpoint discovery service) at the moment, and you would not be able to utilize that method. It is certainly on our radar as something to support in the future though.

You guys have included in your documentation section about integration but never provided any examples

If you could point me to what exact section your are mentioning here, I can take a look and update it to clarify exactly what we mean by integration and see about adding examples, thanks!

[bcba25]

@crhino thank you for your reply! What do you mean by

Unfortunately, non-Connect services are not discoverable via EDS (endpoint discovery service) at the moment, and you would not be able to utilize that method. It is certainly on our radar as something to support in the future though.

Broken Link:
https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/rds.html#config-http-conn-man-rds

Location: https://www.envoyproxy.io/learn/service-discovery bottom of the page

Regarding the examples I mean your whole documentation actually. I've noticed that you mentioning for example that xds configuration is possible with envoy, k8s etc, but apparently I have not found any articles or examples. Also tried to google on envoy consul integration tutorial or basic example, also didn't managed to find it. The only examples I found are located here at Github which is very sad. Comparing to for example nginx, tons of information including courses etc. I hope you guys will improve on that so that more people will start using your products, which are really decent ones!

[crhino]

@bcba25 What I mean by that is, Consul can expose a GRPC server that can interact with Envoy's xDS API, however the XDS configuration that Consul sends to Envoy is only applicable when Envoy is being used as a sidecar proxy for services. The Envoy process itself needs to be registered as a proxy service in Consul.

Probably our best guide of the current Envoy integration is our Learn guide, but I don't believe that is the use case you are trying to address.

I'll take the documentation feedback and see what we can do to improve it. The Envoy integration has is comparatively very new versus Nginx/HAProxy, but we are actively working on making it as simple and easy to setup as possible so you should see improvements soon!

For your specific case though, have you considered the two options I presented above?

[ellisgeek]

Seriously, is there any default example on how to integrate with envoy-proxy ? At least a basic one? You guys have included in your documentation section about integration but never provided any examples 😞

This has been one of my issues while rolling out consul in our environment (we aren't doing anything with containers besides some docker containers that are just stuck on single hosts). I finally settled on using flightpath as an XDS go between for my services in consul and envoy because it can be run stand-alone without relying on Kubernetes.

It's not very flexible but it works well, at some point when I have time I'd like to contribute some features.

[bcba25]

@ellisgeek thanks for sharing your experience, was it difficult to setup ?
Does your method works with dynamic gRPC hosts ?
And do you have any example config or something to share, would be veeery much appreciated

[blake]

@bcba25 @ellisgeek We just announced the beta release of Consul 1.8 which includes native support for Ingress gateways using Envoy. See the announcement blog for more details (https://www.hashicorp.com/blog/announcing-hashicorp-consul-1-8/), or head over to https://www.consul.io/docs/connect/ingress_gateway for info configuring a gateway.

Please share your feedback whether this feature will work for your use case.

[bcba25]

@blake thanks for letting me know, ill definitely try and let you know soon !

[ellisgeek]

@blake Thanks, I've been updating servers to Consul 1.8 and plan on exploring the ingress gateways feature in the coming weeks.

[bcba25]

@ellisgeek honestly speaking, documentation is still quite difficult to read. Would be nice if you guys could make some basic tutorials, at least BASIC ones on how to integrate your products together.

[ellisgeek]

@bcba25 yeah after trying to get an ingress proxy working for a couple hours i'm pretty frustrated with the documentation. In general it feels like it was written by someone who expects the reader to already be familiar with how the whole thing is supposed to work. the fact that parts of the documentation boil down to "here is an example config with every possible option" but there is no explanation of the options is incredibly annoying.

[blake]

@bcba25 @ellisgeek I appreciate you sharing your feedback, and I understand the frustration here. We're actively working on improving these docs & developing guides to help folks better understand how to configure new features like Ingress and Terminating Gateways.

A step-by-step guide for configuring Terminating Gateways was published a few hours ago (https://learn.hashicorp.com/consul/developer-mesh/terminating-gateways). A similar guide for Ingress should be available in the coming days, ahead of the GA release of 1.8.

[blake]

@bcba25 @ellisgeek As of yesterday Consul 1.8 is now GA. We've published a couple of blogs and guides on configuring Ingress gateways.

• https://www.hashicorp.com/blog/ingress-gateways-in-hashicorp-consul-1-8
• https://learn.hashicorp.com/consul/developer-mesh/ingress-gateways

I believe ingress gateways should satisfy this feature request and so I am closing this issue. Please feel free to reach out on discuss.hashicorp.com if you have further questions about configuring and using ingress/terminating gateways.

[suman148]

@crhino Quoting one of the options that you mentioned above in the thread:

Replace address: host.docker.internal with address: .service.consul, putting in the correct Consul DNS name for the service lookup. This relies on using Consul DNS for service discovery, and does hardcode the DNS address into the Envoy config. However, it is pretty simple to setup.

1. Is there any example of integrating Envoy with the Consul service registry (via DNS) without running Consul connect or consul agent on the application host?

2. address: .service.consul seems to be a partial configuration that only takes care of the service name. So, how to configure the Consul server address and port in Envoy's static resource section?