Sensu-go: Keepalive check / handler ignores filter

[brieken]

I'm running the Beta 4 release. What I did is the following

systemctl stop sensu-backend
rm -r /var/lib/sensu/etcd

then sensuctl configure en created users and changed passwords. And then created two handlers

sensuctl handler create slack --command 'dummyslacktest' --type pipe --timeout 30 --filters is_incident,not_silenced
sensuctl handler create keepalive --handlers slack --type set

and created some checks

sensuctl check create DiskCheckWindows --command 'c:\\programdata\\sensu\\checks\\check_disk.exe -u GB -w 10% -c 5%' --handlers slack --interval 1800 --subscriptions Windows --ttl 1860 --timeout 60
sensuctl check create ServerHardwareCheckWindows --command 'C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -file C:\\programdata\\sensu\\checks\\check_server_hardware.ps1 ;exit $LastExitCode' --handlers slack --interval 3600 --subscriptions Server,Windows --ttl 3660 --timeout 60

Everything is working, but i keep receiving resolved slack messages on the keepalive check. Some log lines:

Aug 22 12:45:51 monitor.*****.net sensu-backend[28739]: {"check":"keepalive","component":"pipelined","entity":"**************************","environment":"default","handler":"slack","level":"info","msg":"sending event to handler","organization":"default","time":"2018-08-22T12:45:51+02:00"}
Aug 22 12:45:51 monitor.*****.net sensu-backend[28739]: {"component":"pipelined","environment":"default","handler":"slack","level":"info","msg":"event pipe handler executed","organization":"default","output":"dummyslacktest","status":0,"time":"2018-08-22T12:45:51+02:00"}

the other checks are filtered like expected:

Aug 22 12:07:09 monitor.*****.net sensu-backend[28739]: {"check":"ServerHardwareCheckWindows","component":"pipelined","entity":"**************************","environment":"default","handler":"slack","level":"info","msg":"event filtered","organization":"default","time":"2018-08-22T12:07:09+02:00"}

[palourde]

Hi @brieken

Unfortunately I wasn't able to reproduce this issue with the same handlers configured on my local instance.

Could it be possible that the keepalive events for the redacted entity are non-zero status events? If you look at this particular event (sensuctl event info ENTITY_ID keepalive), what's the current status?

[brieken]

Check:     keepalive
Output:
Status:    0
History:   0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1
Silenced:  false
Timestamp: 2018-08-22 20:37:57 +0200 CEST

[brieken]

To further investigate I did the following. I decided to run sensu-backend inside sensu/sensu docker container instead of ubuntu package. So I can easily start from scratch. I have a sensuctl script so I can configure the backend quickly.

In my environment i have Windows sensu agents trying to connect to the backend. When I spin up a Beta 3 docker container from scratch keepalive checks are filtered. And not send to a handler. Like it should be. If i spin up a Beta 4 container. The keepalive checks are not filtered. And send to a handler.

While i'm doing this docker create and rm of containers the windows agents are always trying to connect. So when spinning up a Beta 3 container everything ok. And when spinning up a Beta 4 container the keepalive checks are nog filtered.

[palourde]

Hi @brieken

Thank you for investigating this issue further. Using the same Docker environment, could you try to use the latest image (sensu/sensu:nightly) and see if your keepalives checks are being filtered or not?

Thank you

[brieken]

Tested with sensu/sensu:nightly but still the same issue.

[ccressent]

@brieken
I've managed to reproduce the issue. Thank you very much for reporting it and
putting effort in testing with different versions!

We are going to work on finding the root cause and implementing a fix in the next
few days.