Contour: IngressRoute design: represent envoy directly

[phylake]

Describe the solution you'd like
I'd like envoy to be represented 1:1 in IngressRoute.

Anything else you would like to add:
Main advantages

1. You could link to envoy's excellent documentation directly, instead of re-documenting parts of envoy as they're represented in Contour.
2. Contour could leverage protobuf-generated Go (notice the json tags already present on each field) and validation instead of going through unwieldy translation layers. The following links illustrate this with websockets: IngressRoute's EnableWebsockets bool is assigned to dag.Route's Websocket bool which is finally assigned to the (protobuf-generated) RouteAction's UseWebsocket that's passed to envoy. That's 3 names for the same boolean. If the RouteAction struct was deserialized directly there would be no translation. Admittedly, I don't know how the DAG would be impacted but I'd be happy to work through it. The cognitive, lines of code, and testing overhead of this translation can't be overstated.
3. If you could do 2. then adding the many features requested (I believe that would minimally cover #805, #608, #437, #361, #70, #69, and #515 which were all open as of this writing) would be trivial, or even automatic as new fields are added to the protobufs
4. The CRD could be generated automatically
5. The CRD could be versioned to match envoy versions. There would be no version matrix or backward compatibility to deal with.

My biggest question: is a goal of Contour to abstract envoy?

• If the answer is no, then I'm not sure why this wasn't the approach taken in the first place once a CRD was on the table.
• If the answer is yes, then I think you need to change the project's description: "Contour is a Kubernetes ingress controller for Lyft's Envoy proxy."

Are there any challenges besides representing the DAG that I'm missing? I'd much rather tackle this problem than repeat #856 for every bit of config in envoy that we need.

Environment:
N/A

[davecheney]

@phylake thanks for raising this issue. In answer to your question

My biggest question: is a goal of Contour to abstract envoy?

The answer is _yes_. It's probably never been explicitly stated, but the goal of Contour is to provide an opinionated deployment of Envoy as a HTTP reverse load balancer. There are many things that Envoy can do, the goal of Contour is _not_ to expose all of those to end users.

With that said I want to give your proposal close thought. It does seem redundant to reimplement the envoy data structures as a CRD, but I offer the following mitigating concerns.

1. envoy's data structures change _a lot_. To some extend Contour hides this churn which is a good thing.
2. the envoy grpc api is very complex and provides no opinions. In some way Contour is an opinionated shim over the envoy api. Interacting with envoy via gRPC types gives you some guard rails via compiler errors, navigating the complex API via k8s yaml would be a frustrating experience.

wrt to the project description, do you have any alternative suggestions?

[davecheney]

github ate my edit; TL;DR a shim that lets us put the envoy data types into CRD's and use the k8s API server as an xDS grpc server is a fascinating idea, but not on the short term road map.

[stevesloka]

Another goal of ingress route was to not tie it to envoy, others could implement the CRD and not use Envoy.

[phylake]

Thanks for the clarification @davecheney and @stevesloka. I think it would be good to call out the project's goals somewhere.

envoy's data structures change _a lot_. To some extend Contour hides this churn which is a good thing.

Do you agree there's a tradeoff here? That hiding Envoy's churn slows down contour development?

navigating the complex API via k8s yaml would be a frustrating experience

It couldn't be more frustrating than configuring Envoy directly which isn't frustrating and is really what I'm asking for. While the many open feature requests may not be asking for it directly (i.e. the requestors may not care "how" the feature's exposed) I don't know that they'd be frustrated by having all of envoy's features available to them. Have you asked them? That said I recognize it goes against the goal of not tying the CRD to envoy.

wrt to the project description, do you have any alternative suggestions?

I think it's worth adding "opinionated" to the description and elaborate on the opinions somewhere. That'll be important to people evaluating ingress controllers. I'd like to know what the opinions are too.

The answer is _yes_
...
Another goal of ingress route was to not tie it to envoy

I think your abstraction is leaking. You've already exposed the exact Load Balancing Strategies in envoy. This is indirection, not abstraction. HAProxy and Nginx (as examples of other reverse proxies that might implement the CRD) don't have MagLev afaik.

The CRD structurally matches Envoy's Route, RouteAction, and WeightedCluster

A third example will be #856 if you merge it.

I'm being critical, not malicious, btw :)

My final thought on the goal of not tying IngressRoute to Envoy: If others are like me, they know which proxy they want to use and look for an ingress controller for it, not the other way around. I don't _want_ or _need_ the abstraction because I don't _want_ to have to figure out how the CRD maps to Envoy config, and I don't _need_ to swap out the underlying proxy

[unicell]

@phylake I'm also facing the similar issue. The route I'm currently taking is to extend IngressRoute CRD with some new protobuf defined fields. And those protos could either be defined within the project, or use the one from Envoy dataplan-api. For the latter one, it becomes a one to one mapping to what Envoy can be configured through xDS.

I found this approach provides the benefits from both sides, manageable in-house changes and yet not too intrusive for Contour code base. Wondering if I can contribute it back somehow.

[bgagnon]

Relevant info from the Envoy project:

• envoyproxy/envoy#6271
• Breaking Change Policy
• xDS Protocol Versioning

From the documents above, I gather that:

• the Envoy community is very much aware of API breakage issues
• there are concrete plans to address this
• there are concerns this will slow down velocity on Envoy

Since it's not a completely solved problem on the Envoy side, I sympathize with Contour's desire to shield the end user from this churn. As things mature and stabilize, however, I have to agree that the double translation between Envoy and Contour leads to undesirable things illustrated by @phylake :

• leaky abstractions
• redundant structures and aliasing
• cognitive overhead in the translation of concepts and configurations

Moreover, I see a pattern of re-assessing the value of each Envoy feature from the Contour side (ex: cors, auth, sticky sessions) and re-having the debate on how things should be named and configured and if they are out of scope or not for this project.

The one thing I have a hard time wrapping my head around is the idea that the IngressRoute CRD should be abstracted enough to work with other proxies, and perhaps one day graduate from not being a CRD anymore and moving to K8S core API itself.

In my experience, the native Ingress resource clearly demonstrated this was really hard and ultimately resulted in highly specialized annotations that basically render the average non-trivial Ingress object in the wild entirely proxy-specific.

This led to the proliferation of CRDs in all modern takes on ingress controllers, and the resulting objects and documentation have been a lot more sane than the decorated native objects. I for one don't see a reason for someone starting out to use Ingress with Contour; I would go directly for IngressRoute. I have the same preference for CRDs with Istio, Gloo, Ambassador and Voyager.

Perhaps the only redeeming quality of the native Ingress is that it can be dropped in a Helm chart with _some_ confidence that it _might_ just work for the receiving user without modification. And maybe that is what Contour is hoping for with IngressRoute.

In any case, I will be watching this "fork in the road" issue closely.

[davecheney]

Hello,

Thank you for the discussion on this issue. This issue has been open for a long time without comment. I apologise for that. My workload is immense.

I appreciate that it is frustrating to have to wait for Contour to add its own translations of Envoy concepts, and I hope that this will improve in the future as the team hopefully grows, but this isn't a good justification to just drop the Envoy gRPC api on k8s users wanting a HTTP ingress solution. By exposing the full verbosity of Envoy's gRPC we provide no guidance, no defaults, no opinion to help our users get started, no option to improve their experience other than pushing them towards a templating solution.

We will not expose Envoy's data structures directly to the k8s API. This is a design decision based on Contour's desire to focus on web applications, not be a generic service proxy. There are other projects that want to be generic service proxies, we have no interest in competing with them.