Sensu-go: Backend unable to fully reestablish connection with agents after temporary network outage causing stale checks

[gtufte]

Expected Behavior

When the sensu backend temporary loses network connection for some reason, it should be able to reestablish connection with the sensu agents and continue scheduling checks when the network connection comes back.

Current Behavior

When the backend comes back from a temporary lost network connection, it might not be able to fully reestablish connection with agents ending up in a state where the keepalive check is OK, but other checks for the agent doesn't get scheduled/executed and end up stale. I can't see any patterns in which checks ending up stale, it seems to random. E.g. I have one host performing 400+ proxy checks where only some of them (random) end up stale. The same is true for non-proxy entities.

Possible Solution

Only a workaround: Set a TTL on every check to get notified on all the stale checks

Steps to Reproduce (for bugs)

1. Run a backend with several agents and checks
2. Plug out network (either cable or disable network card in hyper visor)
3. Plug in network
4. Observe stale checks

Context

We had a temporary network outage in our data center, backend lost network connection with agents. When network came back up, everything was OK in the sensu dashboard. What we didn't knew was that many of our checks were stale and we had a degraded monitoring system. This was before we set a TTL on every check.

Your Environment

sensu-backend version 5.11.1#39d2bf2, build 39d2bf22c416f9a78f22c4817dc74ab62d7f4d63, built 2019-07-18T13:30:28Z

sensuctl version 5.11.1#39d2bf2, build 39d2bf22c416f9a78f22c4817dc74ab62d7f4d63, built 2019-07-18T13:31:59Z

sensu-agent version 5.11.1#39d2bf2, build 39d2bf22c416f9a78f22c4817dc74ab62d7f4d63, built 2019-07-18T13:31:26Z

One server running Sensu backend, installed and configured with the Puppet-module mod 'sensu-sensu', '3.4.0', CentOS 7.

Agents also installed with the same Puppet-module, running mostly on CentOS7 and Windows Server 2012, 2016 and 2019

Backend have:

• ~ 500 checks
• ~ 150 proxy entites
• ~ 150 non-proxy entities

[adammcdonagh]

Sounds very much like the issue I raised #3156. Not so much with proxy entities, but I have seen similar behaviour with pretty much any check just not getting scheduled anymore

[palourde]

Hi @gtufte,

As suggested by @adammcdonagh, it appears to be similar to #3156. Would you be able to look back at the sensu-backend's logs and try to look to see if you see something similar to these log entries, around the time you had the network outage?

{"component":"schedulerd","error":"error while starting ring watcher: etcdserver: request timed out","level":"error","msg":"error scheduling check","name":"proxy_check","namespace":"default","scheduler_type":"round-robin interval","time":"2019-08-12T18:32:57Z"}
{"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"proxy_check","namespace":"default","scheduler_type":"round-robin interval","time":"2019-08-12T18:32:57Z"}

You could also filter the logs with schedulerd to make it easier to find those entries.

Thanks!

[echlebek]

We discovered that when sensu-backend encounters a network error when trying to write a message to an agent, it will stop all future writes, expecting the connection to terminate. This would cause the agent to re-establish a connection.

However, what actually happens is that the connection can survive this error, leaving only the connection reader active. As a result, it appears that check scheduling has halted. Keepalives will still continue to be sent from the agent to the backend, but the backend will not send any further messages to the agent.

What we should be doing, instead, is to close the connection at the first sign of trouble, which will cause the agent to reconnect.

[palourde]

Hi everyone! I just wanted to let you know that Sensu Go 5.14.0 is now available and hopefully it will solve this issue for good. If you believe you are still facing this problem, please reach us via Discourse, Slack or open a new issue with some details and logs so we can start investigating. Thanks