Consul: Support exposing multiple ports in a sidecar proxy.

[vinay-g]

Feature Description

Add feature so that consul proxy can serve multiple services via single proxy. Currently via envoy or consul only a single service can be served via a proxy.

Use Case(s)

Consider a pod consists of multiple containers, each container listens to a specific port. Currently via connect only one service per pod can be served. Instead of creating one proxy per service, if a single proxy can handle multiple services it would reduce resource consumption and simplify the design.

[suchisubhra]

Hi Guys,
we really like to have this feature and like to know how to escalate and if possible how to contribute.
Is there any way you guys let us know what is the procedure? Really appreciate your help in advance.

[mkeeler]

@suchisubhra In general adding a 👍 to the top level issue is a good start. Then if you have other information to add about your particular use case you would leave that in the comment.

[msuarezd]

We are using nomad with the java driver and fat jars. Currently we need to start an envoy sidecar for each service deployed on a node. Being able to register all services (sidecars) in the local agent to an unique envoy instance would simplify our setup drastically. It would also help with other topics like collecting statistics over Prometheus, etc.

[banks]

@msuarezd one issue with the model you propose is that any workload that nomad might schedule on any host becomes "trusted". In other words none of the security benefits of Connect Intentions would really hold between any workloads that are allowed to be scheduled behind the same "sidecar".

In a typical nomad cluster where all jobs can be scheduled on any machine that basically means that Connect Intentions can't really enforce anything in your proposed setup unless you are explicitly limiting jobs that are logically the same "trusted unit" only scheduled together.

So while we don't rule out supporting that, it would come with a lot of caveats as it basically relies on the scheduler configuration to ensure any of the security properties of Connect.

One of the main reasons for this issue is that currently Consul only supports a single port per service so to model a pod that exposes different protocol/APIs on different ports, you have to register each as a separate service to be individually discovered but then need a way to configure a single sidecar for all of the services. The problem now is that as far as Consul Connect is concerned, the three services are different "identities" when they might all be the same process.

It's ambiguous whether they should be different "identities" - in some cases they might just be alternate protocols giving access to same stuff so should be the same "identity" (and have same Intention rules applied" in other cases they might be legitimately different - e.g. a public port and an internal port that should have different access control.

So the ultimate solution here probably involves making Consul Services more flexible so that you can expose multiple ports for the same logical service. We've not made a firm decision on that but it seems the cleanest way to model @vinay-g's use case. The question then is whether you then need to be able to change Intentions based on which port as well as just service name. That's much less obvious as the right thing to do.

I think we'll certainly have to come back and consider this more before too long but any additional info people can share on their use-cases would be really helpful. Specifically:

• Do your multiple ports need different intentions defined?
• Is it OK that all the services on a shared Envoy can access all of each other's upstreams? e.g. if web is allowed to access db but billing is not, running both web and billing behind the same Envoy would allow billing (if compromised by malicious user/code exploit etc) to access db defeating intention graph.

[msuarezd]

Hi @banks,
thank you for the feedback and sorry for the late Answer.

As you say, every service would be able to access the upstreams of other services running on the same host. But this is the case anyway if we have one sidecar pro job.
If I have service A and B on the same nomad node, nothing keeps service B to access the upstreams from server A which are listening to localhost on the same node (even if they are handled by different envoy instances).
We would have some benefits from connect however: TLS encryption in all inter-node communication, upstream load balancing, easier configuration standard, observability, etc.

To your specific questions:

Do your multiple ports need different intentions defined?

No. What I mean is a way to expose all services registered to a consul agent to be exposed over the same envoy. A way to point envoy to the consul agent at localhost and receive the configuration for all sidecar proxies registered on that agent. Envoy would act like an edge router on every node for each service running there.

Is it OK that all the services on a shared Envoy can access all of each other's upstreams? e.g. if web is allowed to access db but billing is not, running both web and billing behind the same Envoy would allow billing (if compromised by malicious user/code exploit etc) to access db defeating intention graph.

Yes. It would be lovely to restrict that, but to my understanding one would have to isolate those services on different network namespaces with an own sidecar proxy on it (like a kubernetes pod). And that is not possible with the java driver at the moment. But as there is no way to enforce intentions for services running on the same node for the java or exe driver anyway (correct me if I am wrong!), we could still profit from some other features of connect.

I hope I could help

[banks]

Thanks. That's useful - we certainly will need a solution to this and as you said the "multiple ports and services in one pod" we're considering the best option there.

[bencyoung]

This would also benefit us, as we're not so interested in the security side of connect but using it in an environment where a legacy app can't be changed to talk to consul directly and the environment is locked down do that we can't use DNS either! We are looking to use Connect proxies to enable these services to do service discovery by it means a lot of proxies having to be spun up manually!

[jsosulska]

Hi Folks,

Please track #6357 as the top-level ticket for this question. This current ticket, #5388, is marked as duplicate and will be closed.

Thank you!

[jsosulska]

Hi Folks - Going to reopen this and provide some context for what this issue is going to track.

The Problem:
Currently, there is a one to one mapping of service and exposed port back to the sidecar proxy. This causes issues for pods that expose multiple service ports, like for different functionality. An example of this is when a service has a public HTTP port, and a more privileged admin port. Currently, the proxy is unable to determine which port the traffic should end up being routed to.

The Clarification:
Neither Ingress nor Terminating gateways solve the problem described here.

Both Connect mesh gateways and Ingress gateways run into this problem where a single proxy does not accurately reflect multiple exposed listeners for the service, and can't route traffic in those cases. However, today, the functionality exists to define each exposed listener as a unique service, and route traffic using those unique service names.

We definitely want to solve this in the near future, but we recognize that any solution will likely have larger consequences on ACLs, intentions, and the catalog & addressing models.

Renaming this to: Feature Request - Support exposing multiple ports in a sidecar proxy.

For additional Context:
For inbound, this can be solved by SNI headers, where a different cert is used based on the SNI(server name indication) header provided by the client. This is what is being done today by Ingress and Terminating gateways.

For outbound, this becomes more difficult, as the proxy doesn't have a way to differentiate & define a certificate of the originating service's exposed subsystem.

Ingress gateways solve the use case of non-Connect traffic to Connect traffic to external (non-connect) traffic.
Terminating gateway solve the reverse use case, where Connect enabled services are communicating out of the service mesh to services.

[tunh]

I have the same issue. Currently I have a service which serve on multiple ports but sidecar proxy only support binding one-to-one port.
I have to register multiple services for multiple ports and end up with one service - multiple ports - multiple proxies.