Envoy: Discussion: supporting alternative L7 protocols based on UDP & TCP

[beriberikix]

With the recent alpha support of UDP listener filters, I'm hoping to kickoff a discussion on how we should holistically approach adding support for new L7 protocols for Envoy users, with the goal of something like a best practices doc and/or a few proposed new APIs (if needed.)

Envoy has first-class support for HTTP1/2/3ish, which includes connection management, proxying, sniffing & routing. HTTP is also extended to support different HTTP-based protocols like gRPC. But since Envoy is an L3/L4 proxy at the core, it has "complete" support for TCP-based protocols like MongoDB and Redis. Complete is in quotes because there are also many protocols that are implemented simply at the data layer as Filters, since many are HTTP-based.

As the reader may already know, there are many IP-based protocols in the world beyond HTTP and popular databases. Here's just a few of the ones I've investigated for use with Envoy, as well as their category and protocol:

• IoT - CoAP:UDP, MQTT:TCP
• Gaming - GameNetworkingSockets: UDP, netcode.io:UDP
• Telephony: WebRTC:UDP
• IP Suite: DNS:UDP, NTP:UDP, TFTP:UDP

Admittedly the list above is a biased sampling of L7 protocols from personal use cases and many of the protocols actual support both TCP and UDP. But bias aside, I believe there are others who would also like to leverage Envoy with these and other protocols. So the main question I hope to answer is:

How do you implement an L7 protocol as a first-class protocol in Envoy?

First-class here means the same level of functionality as HTTP, or roughly connection management, proxying, sniffing & routing, as well as any protocol-specific features. Sub-questions that fall from the main questions are:

• Is the existing functionality in Envoy sufficient to implement new L7 protocols? IE are the current Listeners sufficient for extending to many/most other protocols?
• Are there the right APIs available for developers to implement new protocols?
• How should new protocols be added? It doesn't make sense to bloat the codebase, so is Wasm the right way forward?

At this point I'd love to start the discussion and hear what other people think!

/cc folks who have given me early feedback: @markmandel (from agones) @PiotrSikora @jplevyak @duderino

[mattklein123]

This is a pretty high level question so I'm not quite sure exactly how to answer it. In general, we would expect Envoy to be extendable to support almost any L7 protocol. If the right APIs don't exist, we should add them. With that said, there are a few different related issues and concepts here for reference:

• To your point, we are quickly approaching the the number of extensions that is not easily maintainable from a quality/CI/security/etc. standpoint. I'm going to be doing a proposal soon which will allow for some type of sandbox repository for hosted extensions that are not in the main build. See https://github.com/envoyproxy/envoy/issues/7582.
• Related to the previous, I do think the WASM and the stable ABI is the way forward here. If we can eventually move all extensions to WASM and host them on a WASM hub, I think we remove a lot of the pain points in the previous item.
• There does exist a lot of overlapping functionality between protocols, especially as it relates to routing. We can likely do better here to share more code and APIs. See https://github.com/envoyproxy/envoy/issues/9656

[beriberikix]

Thanks for those references! I'm digging into routing proposal, as well as the UDPA work in general. Both references definitely feel like they're going in the right direction!

One way to make the question more concrete is to take a protocol and walk through the process and document the highs and lows. The first one that comes to mind is DNS (ala CoreDNS) because of its familiarity and general usefulness. NTP/SNTP and TFTP would also be good candidates due to their simplicity. Anyone interested in collaborating, or perhaps has already starting that work?

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[beriberikix]

I'm going to close this - discussion has moved to a doc.