Sensu-go: ttl / dead man's switch functionality does not work as described in documentation

[spykermj]

Following the instructions here, with version 5.4 of sensu-go installed:

https://docs.sensu.io/sensu-go/5.4/reference/agent/#creating-a-dead-man-s-switch

(modify the ttl to a smaller value like 60 so you don't have to wait 7 hours to see that it doesn't work)

Expected Behavior

If the ttl expires, I would expect an incident to be created until a new status 0 event is sent to clear it.

Current Behavior

The ttl time expires and there is no incident created, the last status 0 event still shows as the current state.

Possible Solution

Steps to Reproduce (for bugs)

On a host with sensu agent running succesfully:

1. yum install -y nc
2. echo '{"name": "backup_mysql", "ttl": 60, "output": "backed up mysql successfully | size_mb=568", "status": 0}' | nc localhost 3030
3. wait more than 60 seconds, and observe the status for "backup_mysql" is still OK

Context

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent):
  sensu-go-cli-5.4.0-1327.x86_64
  sensu-go-agent-5.4.0-1327.x86_64
  sensu-go-backend-5.4.0-1327.x86_64

• Installation method (packages, binaries, docker etc.):

Used https://github.com/sensu/sensu-puppet and vagrant to set this up:

vagrant up sensu-backend

• Operating System and version (e.g. Ubuntu 14.04):

CentOS Linux release 7.6.1810

[apaskulin]

The examples shown in https://docs.sensu.io/sensu-go/5.4/reference/agent/#creating-a-dead-man-s-switch for creating a dead man's switch using the agent socket do not work because the agent socket ignores all event attribute except the ones listed here: https://github.com/sensu/sensu-go/blob/master/types/v1/check.go.

In the short term, I'll remove that section from the docs to reduce confusion. However, in the long term, I believe this feature would provide substantial value to users. In the corresponding docs issue, @spykermj noted that "this is a big impediment to migrating to sensu go as it is quite an important feature." To support this functionality, we'll need to add the ttl attribute to the agent socket's supported attributes, similar to adding the handlers attribute in https://github.com/sensu/sensu-go/pull/2702.

My apologies for the confusion, @spykermj. Thanks for uncovering this!

[asachs01]

Related is #2679

[asachs01]

@annaplotkin @echlebek having done some testing in favor of using the agent API instead of a tcp socket, it looks like TTL's aren't actually going into crit after they're exceeded.

[asachs01]

@annaplotkin @echlebek further to that, it looks like in Core we had the option to provide the ttl_status attribute to change the desired status of ttl events. To that end, implementing this in Go would be quite desirable.