Sensu-go: Backend not applying filters during shutdown

Created on 3 Dec 2020 · 3Comments · Source: sensu/sensu-go

Expected Behavior

When receiving a SIGTERM, sensu backend keeps applying filters on events processed until the whole backend is properly stopped.

Current Behavior

While shutting down, sensu backend executes handlers for events that should be filtered by custom filters, but are not.

Possible Solution

Maybe the components are not stopped in the right order (?)

Steps to Reproduce (for bugs)

Sensu backend is running on a Kubernetes cluster.
Deleting the pod running sensu backend container is enough to see some events being sent to handler.

Context

We are running sensu on an EKS cluster based on spot instances.
When AWS reclaims an instance, the pod is deleted in order to be rescheduled elsewhere.
We are running about ~400 agents with an avg ~15 checks on each and each time a node is reclaimed we get a load of 10 to 30 unwanted "OK" events on our alerts dedicated slack channel.
This adds some noise on our monitoring system.

The signal received
cmd_signal_received

Filters errors
pipelined_errors

Components shutdown order
backend_shutdown_order

Your Environment

Sensu version used: sensu-backend v6.1.2 OSS
Installation method: docker
Operating System and version: kubernetes 1.17 (EKS)

bug

Source

emmanueljamet

👍5

Most helpful comment

Hi @echlebek.
The 'fatigue_check' and 'period_check' are in fact the names of the filters not applied.
This is not expected to fail :)

emmanueljamet on 9 Dec 2020

👍3

All 3 comments

Thanks for the bug report. We haven't verified this behaviour but it seems fairly conclusive based on your report. I'll tag it as a bug and we'll prioritize it accordingly.

Are you expecting that your javascript event filters should fail like they are in your screenshot? It seems like you have some undefined symbols.