Sensu-go: Entity with missing label breaks proxy checks

Created on 23 May 2020 · 7Comments · Source: sensu/sensu-go

I have a number of proxy entities, which i've added via sensuctl. These entities have various annotations, including 'wan_ip' and 'lan_ip'.

Most entities have both lan_ip and wan_ip. I do have a few which either don't have a wan_ip or don't have a lan_ip.

I then have a check with:

  proxy_requests:
    entity_attributes:
    - entity.entity_class == 'proxy'
    - entity.labels.role == 'console-server'
    - entity.annotations.wan_ip != ''

Expected Behavior

I can understand it if no wan_ip annotation results in an error on: entity.annotations.wan_ip != '' (though the ideal is that it checks for presence before evaluating the expression and works fine!).

However, even then, i'd expect it to be only the entities with missing information which fail.

Current Behavior

The first few entities work fine, then it (alphabetically) gets to one without the wan_ip annotation and I get this in the logs:

May 22 21:07:50 sensu sensu-backend[4630]: {"check":"console-servers-ping-wan","component":"schedulerd","entity":"console1.mydomain.com","error":"command: unmatched token: template: command:1:53: executing \"command\" at \u003c.annotations.wan_ip\u003e: map has no entry for key \"wan_ip\"","level":"error","msg":"unable to substitute tokens","time":"2020-05-22T21:07:50Z"}

it then doesn't attempt to execute the check against any of the other entities.

Possible Solution

I think there is 2 things here.

An error of a check against one entity shouldn't affect all of the others.

Then ideally, entity.annotations.wan_ip != '' would work, even if wan_ip doesn't exist.

Steps to Reproduce (for bugs)

I have a number of entities like this:

type: Entity
api_version: core/v2
metadata:
  annotations:
    eth0_ip: 10.0.0.1
    eth1_ip: 192.168.0.1
  labels:
    role: console-server
    sensu.io/managed_by: sensuctl
  name: console1.mydomain.com
  namespace: default
spec:
  entity_class: proxy
  subscriptions: null

I've then got a check which includes:

spec:
  command: check_ping -H {{ .annotations.wan_ip }} -w 100,20% -c 300.0,50%
  proxy_requests:
    entity_attributes:
    - entity.entity_class == 'proxy'
    - entity.labels.role == 'console-server'
    - entity.annotations.wan_ip != ''
  round_robin: true

Remove the wan_ip annotation from one of the entities.

Context

I'm trying to reduce the number of (almost identical) checks by having a single check which applies to many entities.

Your Environment

5.20.1-12427 - official packages on Debian.

bug

Source

ichilton

All 7 comments

A hidden impact of this bug is that you can easily end up with checks which are supposed to be running but aren't (i.e things which you think are monitored but aren't).

For example, I had a check_http script which monitored some public websites based on the value of a https annotation. That was working fine.

However, 3 days ago I added some new proxy entries. They don't have the https annotation - because they are not web servers.

Today while debugging another issue in the sensu web interface, I happened to spot that those https checks haven't ran for 3 days. So adding some unrelated entities had broken those checks :(

ichilton on 26 May 2020

Thanks, we were able to reproduce this and are looking into it.

echlebek on 26 May 2020

As discussed, there's two concerns here.

The entity_attributes expressions that use != '' should probably first determine if the labels or annotations field exist or not, by doing some like this first: !!entity.annotations.lan_ip. This is a limitation of Javascript and not a lot we can do about that.
When a token cannot be substituted or the execution fails, we should only log the error and continue with the next entity, instead of failing for all proxy entities: https://github.com/sensu/sensu-go/blob/34db22c8067a7763a40e4bed9ec4dfc10d5ae3c1/backend/schedulerd/executor.go#L336-L342