Eventing: Advanced Filtering on Triggers

[nachocano]

Problem
We need more advanced filtering mechanisms than just exact match on Cloud Event types and sources.

Persona:
Event Consumer

Exit Criteria
Being able to configure advanced filters on triggers, either with CEL or some other language. E2E tests are necessary.

Time Estimate (optional):
~2 weeks, depending on how long the design discussions take.

[matzew]

@evankanderson while I like the idea, not sure if - for 0.6 we should introduce some "language" for _advanced_ filters

[bbrowning]

I feel like something a bit more advanced is warranted. Even if it's not a full-blown expression language, it would be great to expand exact matching to fields in the Cloud Event data or allow some kind of basic wildcard matching.

To give some concrete samples of more advanced filtering I'd like:

• I want to receive all events from the K8s Event Source for namespace foo with Reason: RevisionReady
• I want to receive all events from the K8s Event Source for all Pods in namespace foo

[evankanderson]

Discussing with @lindydonna, it may make sense to actually go all the way in and then figure out what the commonalities are. (i.e. start with a full CEL expression or some equivalent, and then figure out that most customers end up writing "source like X, type is Y, and extracted field Z is present.")

The rationale being that once we have the advanced model, it's easy to implement the common items as syntactic sugar that gets expanded server-side, whereas if we started with smaller models, we may end up rewriting the filtering side multiple times. (And this way, the feature is "_X_ is even easier" rather than "complicated use case _Y_ is now more-possible without writing code".)

[grantr]

:+1: Also consider that CEL may be a similar amount of work to prototype as homegrown simpler filtering. See https://github.com/grantr/cel-playground.

[nachocano]

Grant is taking care of this one!

/assign @grantr

[grantr]

/milestone v0.7.0

[lionelvillard]

Have you guys thought about providing the same capability to Subscription? @nachocano @grantr @evankanderson

[grantr]

@lionelvillard yep, that's come up before. At the time it was argued that adding filtering to the Subscription interface is a burden for channel authors because the dispatcher is more complex to implement. From a user perspective, a world in which each channel implements filters slightly differently makes channels less interchangeable than they are today.

[wlynch]

Hi! Just want to chime in here after talking to @vaikas-google at Kubecon.

I suspect that we might want to extend advanced filtering past CEL expressions, since there will often be scenarios where triggers might want to fetch additional data to make decisions on triggering.

The motivating example here is that for source triggering you might need to make calls back to GitHub to fetch additional metadata that was not included in the webhook event, such as files that were modified in the change, or whether the author of the change was a collaborator on the repository.

Something that would be nice is to allow the user to provide an image in their trigger configuration that could be invoked with an event to make arbitrary filtering decisions, returning a simple allow/deny response.

Thoughts?

[evankanderson]

Looking up additional information outside the event itself seems like it could be the first step in a Pipeline, and the filter could then choose to either return no event (drop event) or the same event (identity transformation).

This can also be accomplished today by delivering from a Trigger to a Channel -> Subscription (Filter) -> Channel configuration, which is what Pipeline would be wrapping.

I'm concerned that by introducing general-purpose compute into Trigger, we'll be losing the ability to efficiently solve common-case problems like "all events of type X" or "all events about objects > 4MB" (assuming that object size is included in the objectChange event).

[vaikas]

Yeah, so my thinking was that we should be just building a "filtering function", that looks like just a normal function that would then handle these more advanced filtering capabilities. At least until we more experience with them, it seems like general purpose compute should be handled by a function and then it can either emit events that the "real function" can take action on (if using triggers), or as @evankanderson mentioned, it could be the first step in the pipeline and either emit a decorated event for downstream processing, or not at all, which effectively filters the event from further processing.

[grantr]

/milestone clear

[n3wscott]

@slinkydeveloper is also trying to fix this.

[slinkydeveloper]

This issue is duplicate/more specific i think #3359

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[lionelvillard]

/remove-lifecycle stale

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.