Eventing: Clarify the scope of each data-plane component

[lionelvillard]

Problem
There is a wild disparity wrt scoping:

• cluster-scope: these are components deployed in knative-eventing and they handle resources in all namespaces. Example: in-mem dispatcher, kafka dispatcher, etc..)
• namespace-scope: these are components deployed in a namespace. They handle resources in the namepace. Example: broker filter
• resource-scope: same as namespace-scope except they handle only one resource. Example: APIServerSource receive adapter, PingSource receive adapter, etc...

Few component supports both, cluster-scope and namespace-scope (eg. in-mem and Kafka dispatchers). None supports all three.

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.

Clear documentation about the currently supported scope(s) for each component and future plan.

Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1

Additional context (optional)
Add any other context about the feature request here.

[aslom]

Not only documentation but also have some metadata in CRDs to discover want scopes given component supports? So users could query and different setups could be tested?

[mattmoor]

In addition to documenting the scope of particular dataplane component as foo-scope, it is important to quantify the per-foo overhead incurred by that implementation. I would also argue that if we want different flavors of these, that what we really want are different "classes" of resource instead of different modes(?). I also think that we cannot be ignorant to the per-foo cost when choosing foos for a particular class.

To echo the sentiment I expressed in #eventing as this pertains to the Broker, I feel like presenting users choice with respect to the tenancy model is a good thing, but that is best done through different classes vs. moving parts (e.g. trying to support modes, or optionally depending on Serving).

What I believe should be our goals for our out-of-the-box default (read: not everything, just our default) in eventing has the following constraints:

1. Production-capable (durability, scale): Folks shouldn't have to make major changes to reach day 2.
2. Doesn't depend on serving: serving is fairly heavy, and comes with its own baggage.
3. Little/No footprint when unused: It's a little silly to do all this work to scale to zero if you need a Pod to stick around for each of the things that might trigger it.

It strikes me as though namespace-scope and resource-scope options will really struggle to achieve 3. without 2., and a flag to swap Serving for Deployments just puts which of 2. or 3. to violate in the users hands and I don't think that's right when we can give them all three.

[lionelvillard]

@mattmoor +1 for classes.

How confident are we about achieving 1. without 2, specially wrt. scalability?

The out-of-the-box default should be cluster-scope. The dependency on serving is an orthogonal issue.

[mattmoor]

At that point we're in "activator" territory, so we have the same "problem" in serving (so 2. isn't a silver bullet!). We also have the same problem for the gateway pods of every KIngress implementation.

I think there's two elements to my answer:

1. I think we can get a long way towards 1. by having HPA over the shared resource.
2. 3. for shared system components a baseline of 1-2 pods is probably an acceptable default (you didn't mention it, so I assume you agree)

One of the things we need for Serving is a "provisioning guide" to try and help folks make the right choices for the "spikiness" of their traffic for day 2. For instance, if your cluster is cold, a single node, and the activator and Envoys have scaled to 1-2, you shouldn't expect to go from 0 to 100,000 QPS in seconds without 5xx 🤣

[lionelvillard]

Yup I agree for 3.

[grantr]

/label proposal/0.14

[grantr]

Lionel are you able to start a draft of this document for 0.14?

[lionelvillard]

/assign

[liu-cong]

Maybe you can clarify the resource-scope but I don't get the value to make it a separate scope. It's just a special case of namespace-scoped, and a resource-scoped component may evolve to namespace-scoped. What's the motivation to make resource-scope stand out?

[lionelvillard]

They are many components today that are resource-scoped so at least we should mention it.

I'm not so sure that all components will evolve to be namespace-scoped. Like the APIServerSource requires special RBAC rules per resource.

[liu-cong]

Yeah I agree there is such a case. I am just trying to argue if we should make it an explicit user facing concept that forces the users to understand it.

Under a multi-tenancy context people mostly talk about cluster vs namesapce scoped. Adding a new scope adds complexity for user to understand. We can mention it maybe in docs but I will try not to make it an explicit scope, unless there are use cases where users do care about this distinction.

[lionelvillard]

yes agree. It all boils down to whether there is a compelling use case.

One idea is channel prioritization where the traffic is first send to a resource-scoped channel and then to the ns/cluster scoped channels...

[lionelvillard]

/reopen