Sensu-go: Checks fail to run after having been unsubscribed/resubscribed.

[nixwiz]

Removing a subscription from an entity/agent then re-adding that subscription to the agent and the checks for that subscription do not run. My problem exists with proxied checks/entities, I've not tested with non-proxied.

Expected Behavior

When re-adding a previously removed subscription, the checks should start running again and creating events.

Current Behavior

Checks are not running on their intervals. Checks can be manually initiated from the dashboard or via sensuctl.

Possible Solution

Steps to Reproduce (for bugs)

1. Create a proxied check and assign to a new subscription
2. On a client, add that subscription to the agent.yml and restart
3. Verify the check is running and the new proxied entity exists
4. On the client, remove (or comment out) that subscription and restart the agent
5. Verify that the proxied entity/check still exist but are no longer generating events
6. On the client, re-add (or uncomment) that subscription and restart the agent
7. Check events for the proxied check, none will occur on the scheduled interval

# cat true.json
{
  "type": "CheckConfig",
  "spec": {
    "command": "/bin/true",
    "handlers": [ "mail" ],
    "interval": 60,
    "timeout": 0,
    "subscriptions": [
      "truth-proxied"
    ],
    "proxy_entity_name": "source-of-truth",
    "publish": true,
    "metadata": {
      "name": "check-for-truth",
      "namespace": "AWS"
    }
  }
}

# sensuctl create -f true.json

# sensuctl check info check-for-truth --format wrapped-json
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check-for-truth",
    "namespace": "AWS"
  },
  "spec": {
    "check_hooks": null,
    "command": "/bin/true",
    "env_vars": null,
    "handlers": [
      "mail"
    ],
    "high_flap_threshold": 0,
    "interval": 60,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "proxy_entity_name": "source-of-truth",
    "publish": true,
    "round_robin": false,
    "runtime_assets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "truth-proxied"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

# sensuctl entity info example1.aws.example.com
=== example1.aws.example.com
Name:                   example1.aws.example.com
Entity Class:           agent
Subscriptions:          linux, truth-proxied, entity:example1.aws.example.com
Last Seen:              2019-01-11 09:56:16 -0800 PST
Hostname:               example1.aws.example.com
OS:                     linux
Platform:               ubuntu
Platform Family:        debian
Platform Version:       16.04
Auto-Deregistration:    false
Deregistration Handler:

# sensuctl entity info source-of-truth
=== source-of-truth
Name:                   source-of-truth
Entity Class:           proxy
Subscriptions:          entity:source-of-truth
Last Seen:              N/A
Hostname:
OS:
Platform:
Platform Family:
Platform Version:
Auto-Deregistration:    false
Deregistration Handler:

# sensuctl event info source-of-truth check-for-truth
=== source-of-truth - check-for-truth
Entity:    source-of-truth
Check:     check-for-truth
Output:
Status:    0
History:   0,0,0,0,0,0
Silenced:  false
Timestamp: 2019-01-11 10:03:16 -0800 PST

=================================================
Remove subscription from client/agent and restart
=================================================

# sensuctl entity info example1.aws.example.com
=== example1.aws.example.com
Name:                   example1.aws.example.com
Entity Class:           agent
Subscriptions:          linux, entity:example1.aws.example.com
Last Seen:              2019-01-11 10:04:19 -0800 PST
Hostname:               example1.aws.example.com
OS:                     linux
Platform:               ubuntu
Platform Family:        debian
Platform Version:       16.04
Auto-Deregistration:    false
Deregistration Handler:

# sensuctl event info source-of-truth check-for-truth
=== source-of-truth - check-for-truth
Entity:    source-of-truth
Check:     check-for-truth
Output:
Status:    0
History:   0,0,0,0,0,0,0
Silenced:  false
Timestamp: 2019-01-11 10:04:16 -0800 PST

=================================================
Wait several minutes, timestamp has not changed
=================================================

# sensuctl event info source-of-truth check-for-truth
=== source-of-truth - check-for-truth
Entity:    source-of-truth
Check:     check-for-truth
Output:
Status:    0
History:   0,0,0,0,0,0,0
Silenced:  false
Timestamp: 2019-01-11 10:04:16 -0800 PST

Only once I deleted the proxied entities, events, and deleted and recreated the checks and then restarted the agent on the client, it started working again.

Context

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent): sensu-backend version 5.1.0#b2ea9fc, build b2ea9fcdb21e236e6e9a7de12225a6d90c786c57, built '2018-12-18T21:31:11+0000'
  sensu-agent version 5.1.0#b2ea9fc, build b2ea9fcdb21e236e6e9a7de12225a6d90c786c57, built '2018-12-18T21:31:11+0000'
  sensuctl version 5.1.0#b2ea9fc, build b2ea9fcdb21e236e6e9a7de12225a6d90c786c57, built '2018-12-18T21:31:11+0000'
• Installation method (packages, binaries, docker etc.): packages
• Operating System and version (e.g. Ubuntu 14.04): Ubuntu 16.04

[annaplotkin]

@nikkiki To review and see if any recent changes changed this behavior.

[nikkictl]

I'll reevaluate this issue after https://github.com/sensu/sensu-go/pull/2703 lands.

[nikkictl]

Hey @nixwiz, I've tried to reproduce this issue in a few different environments.

1. Per your example: Using the same 5.1.0 packages on Ubuntu 16.04, in a clustered environment, agent has subscriptions in addition to truth-proxied
2. Clean state: Using go binaries at 5.2.1 locally on MacOS 10.14.2, with a single backend, agent only has the truth-proxied subscription

Neither scenario has allowed me to reproduce the behavior you are describing. I am able to toggle the truth-proxied subscription in an agent config, and the check is run and event is published when the subscription is in the config, and it does not run/get published when the subscription is not in the config.

Have you been able to reproduce this issue since the original incident? Have you been able to reproduce this issue on 5.2.1? If you are able to reproduce, could you attach any backend/agent debug logs?

Any additional information would be helpful so I can track down this bug for you 😄

[nixwiz]

@nikkiki I will give it a shot with 5.2 over the next couple of days.

[nixwiz]

@nikkiki I just tested this with the brand spanking new 5.2.1 this morning and it appears to be fixed. Woot!

However, during my testing it brought up an interesting question for me. How to deregister proxy entities, especially those created by subscribing to a proxy check (e.g. not created with 'sensuctl create').

After I completed the above testing, I edited the proxy entity and set deregister to true. I then removed the subscription from the entity that subscribed to the proxy check. However the entity still remains. Is that expected? Am I missing something for how to deregister proxy entities?

[nikkictl]

@nixwiz So glad you're not experiencing the issue any more! That's a great question, but in order to answer that, we need to understand how proxy entities differ from agent entities. The main difference in this case being Keepalives. An ephemeral agent entity (entity-class=agent, deregister=true) will be removed when a failing keepalive is observed. Since proxy entities do not have a keepalive mechanism, they cannot be ephemeral. You can manually delete the proxy entity (or agent entity) if it is no longer relevant to any checks/subscriptions. From my understanding, this is at parity with 1.x behavior.

I'm going to close out this issue, but feel free to open a feature request, or chat in the community slack!