Contour: Tighter feedback loop with knative-sandbox/net-contour

[mattmoor]

_This came up while investigating https://github.com/knative-sandbox/net-contour/issues/226_

Today Knative supports Contour via the integration at https://github.com/knative-sandbox/net-contour. This repository currently pins a version of Contour, which is "supported" and used to validate the integration. As of now this has lagged to 1.4, and updating to 1.7 uncovered an issue that was introduced in 1.5, which only manifested in one of Knative's scale tests.

The overarching goal of this issue is to identify ways to enable net-contour to track Contour as closely as possible, so that we identify upstream issues as quickly as possible, and potentially avoid a bad release of Contour.

---

There are currently two levels of testing that Knative runs against Contour:

1. "kingress conformance" - This is a suite of testing that we run against every PR to net-contour to validate the semantics of Knative's "kingress" abstraction. testgrid

1. knative/serving e2e - The knative/serving repo contains "drops" from the net-contour repo, which it runs the full suite of knative/serving e2e tests against. testgrid

As a further datapoint, knative/serving currently runs 2. in 4(!) ways for Istio: {mesh, no-mesh} x {stable, latest}, where stable-no-mesh is what we test against all PRs presubmit.

---

I think a good goal state would be to:

1. Add a leg of "kingress conformance" against a ~nightly drop of Contour.
2. Expand what net-contour "drops" to knative/serving to include the above, and add a full e2e leg for Contour nightly.

Not sure if Contour has any sort of "nightly" construct that we can take advantage of here, but I'd love to kick off that discussion! 🥾

[stevesloka]

We currently only do builds against each PR + merge to the main branch, but would be easy to add more actions which are time based.

I'd like to work together with you on this to enable nightly builds and then possibly have a comment structure (e.g. /test knative or something) that would let us run them adhoc on PR's that might be interesting to test.

[jpeach]

This probably requires some work to make the install YAML more sustainable right?

[mattmoor]

We can discuss it, but our tweaks have been able to carry us forward for a while now without much headache on the yaml front.

[evankanderson]

I missed a link to the issue that was discovered, but Knative does a bunch of ... interesting things with HTTP routing in combination:

1. Plays dynamic games with Services to swap Pods in and out of rotation
2. Is sensitive to latency in propagating routes vs reporting ready
3. Adds a lot of load all at once when testing.

One other option would be that Knative / net-contour could produce a nightly drop after load-testing with its known-good version of Contour which could be feed back into the upstream Contour repo as a stress test. (So knative-sandbox/net-contour imports a "blessed" contour, and projectcontour/contour imports a "blessed" knative-serving-loadtest. Each then uses the results to decide whether it should bless the next release.)

[mattmoor]

So we used to be able to pull Contour's configs via git, and rebuild Contour itself from vendor with ko, which WOULD have allowed us to hack ./hack/update-deps.sh --upgrade track the main branch and given us this. We'd freeze to the latest Contour release branch prior to Knative releases, but there are two things standing in the way of that today.

1. Client version skew (in particular breaking signature changes in client-go 1.18)
2. yaml configs inaccessible via vendor from main.

We're due to switch to the 1.18 clients in Knative soon, so it's plausible that we could consider this as a sort of "nightly" or "bi-nightly" approach for validating Contour at head... 🤔

[mattmoor]

The "Client version skew" is being addressed imminently, and I have stacked a PR (https://github.com/knative-sandbox/net-contour/pull/254) that resumes building Contour directly from source.

If we had a way to pull either nightly rendered configs, or the configs from main then we could get a relatively quick feedback loop here 😍

---

A couple other notable pieces:

1. I have the kingress conformance suite running in a github action now.
2. @n3wscott has been playing with actions to pre-flight changes in downstream dependents.

With these combined we could potentially even know before a contour change lands whether it will break the kingress conformance suite.

[mattmoor]

You can see 2. in action here: https://github.com/knative/pkg/actions/runs/250592356

[n3wscott]

Here is an example of what we are doing in pkg for downstream projects in knative:

https://github.com/knative/pkg/pull/1653

[youngnick]

@mattmoor what do you need to be able to pull the configs from main? If there's changes Contour needs to make to help, we can sort something out.

[mattmoor]

What we were doing before (dummy.go) should work. We switched to consuming your release yamls to get past 1.4.

[xaleeks]

Is this still something we need to do @youngnick ?

[youngnick]

Hmm, I'm not sure. I think that knative may have solved this specific issue in other ways. Not sure if we currently have another issue to cover better integration with the net-contour test suite.

@mattmoor, anything more you want on this issue?

[mattmoor]

I'd like to find a way to test against Contour at HEAD so that if things break downstream we know before a release.

I had been avoiding this myself as I felt it might be a good ramp-up task, so lemme see if I can find a taker 😉

[stevesloka]

If you can't find a taker I'm happy to help out! I'd really like this feedback loop as well. =)

[mattmoor]

Maybe we can sort out an owner when we sync with Sameer next Tuesday 🤩