Sensu-go: Deleting checks should cause associated events to be deleted

[roganartu]

Events for deleted checks persist in the datastore and create annoyances, like TTL timeout events, but in kinda weird ways. I can't think of a good reason you would want an event created by a particular check to persist after the check has been deleted (otherwise, why are you deleting the check?).

Expected Behavior

1) Create a check in Sensu with publish set to True and a non-zero TTL
2) Wait for an event from that check to be created
3) Delete the check
4) The created event should be automatically deleted

Current Behavior

• Events for the deleted check persist
• Events for the deleted check continue to trigger TTL expiry events. Resolving the event by sending an OK event does not prevent these TTL expiries from continuing, however it does cause them to stop changing the status (they simply update all the timestamps, it's kind of weird and perhaps a bug of its own)

Context

1966 seems related, but that was about checks continuing to be scheduled after deletion which is different to this. A comment in there from @grepory acknowledges the issue of events persisting after check deletion:

We know that events can persist after a check is renamed/deleted.

2846 implements this behaviour for entity deletion. It seems confusing to users that events would be deleted on event deletion but not check deletion.

[rgarcia89]

What is the status with this?

[echlebek]

@roganartu how would you feel about an explicit method to delete multiple events by check or entity, vs implicitly via deleting the check?

[majormoses]

From my experience when people delete a check in sensu from a client (entity) they expect that it should remove all history of it. It can confuse people when they see the dashboard and it may or may not be in a failure state. I can't tell you how many times I had to perform redis cleanup for these failed events that would freak people out, I know its not redis anymore but it feels like the same problem all over again. Maybe we can find middle ground and delete any non OK status events implicitly and then have an explicit call to delete all events?

[rgarcia89]

I can totally follow @majormoses opinion.

[gojkoz]

We would like to vote for this to be fixed too. We have some steady turnover of check definitions and when people click on an event that does not have a valid check, the UI crashes (and offers to reload state). It's confusing and also tedious work to remove the stale events. Remember the check is gone so if it alerted before, there's nothing to clear the alert now. If it did not alert, it's not going to start alerting again.

I'd be happy if this would be a configurable check setting - whether to auto clean events upon deletion of the check. That way, default behavior does not have to change if you think that will not work well for most deployments.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mcbsd]

I also see this as very helpful functionality to have.

[rgarcia89]

Same here as this would clean-up events that do not have a related check anymore
FYI @calebhailey

[arkpoah]

if it can help, here a dirty script to delete old events : https://raw.githubusercontent.com/arkpoah/sensu-scripts/main/clean_events.sh