Sensu-go: Bug: proxy entity_attributes are ignored since upgrade to 5.10.0

[Surian]

Expected Behavior

On Sensu 5.10.0 (upgraded from 5.7.0), proxy checks should be filtered with entity_attributes filters and executed on matching proxy entities.

Current Behavior

entity_attributes seems ignored as proxy checks are executed on all proxy entities.

Steps to Reproduce (for bugs)

Entity:

---
type: Entity
api_version: core/v2
metadata:
  labels:
    proxy_type: "temp-sensor"
    proxy_location: "srt"
    fqdn: "10.10.1.3"
    sensornumber: "S1"
    tempcritical: "280"
    tempwarning: "250"
  name: server-room-temp-probe1
  namespace: "facilities-prod"
spec:
  entity_class: proxy

Proxy check 1:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: proxy-ping-wfr
  namespace: "facilities-prod"
spec:
  check_hooks: null
  command: ping -H {{ .labels.fqdn | default "localhost" }} -w {{ .labels.pingwarningrta | default 25 }},{{ .labels.pingwarningpl | default 20 }} -c {{ .labels.pingcriticalrta | default 45 }},{{ .labels.pingcriticalpl | default 40}}
  env_vars:
    - LANG=C
  handlers:
    - sensu-events-history
    - iris
    - email
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: []
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
    - system-linux
    - system32-linux
  subscriptions:
    - proxy-check-wfr
  proxy_requests:
    entity_attributes:
      - "entity.entity_class == 'proxy'"
      - "(entity.labels.proxy_type == 'ping' || entity.labels.proxy_type == 'netapp-system')"
      - "entity.labels.proxy_location == 'wfr'"
    splay: true
    splay_coverage: 50
  stdin: false
  subdue: null
  timeout: 15
  ttl: 120

Proxy check 2:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: proxy-temp-srt
  namespace: "facilities-prod"
spec:
  check_hooks: null
  command: "check_temp {{ .labels.sondenumber | default \"S1\" }} {{ .labels.tempcritical | default 280 }} {{ .labels.tempwarning | default 250 }} {{ .labels.fqdn }}"
  env_vars:
    - "LANG=C"
  handlers:
    - sensu-events-history
    - iris
    - email
  high_flap_threshold: 0
  interval: 60
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: []
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
    - environmental-sensor-linux
  stdin: false
  subdue: null
  subscriptions:
    - proxy-check-srt
  proxy_requests:
    entity_attributes:
      - "entity.entity_class == 'proxy'"
      - "entity.labels.proxy_type == 'temp-sensor'"
      - "entity.labels.proxy_location == 'srt'"
    splay: true
    splay_coverage: 50
  timeout: 10
  ttl: 120

As you see, the first proxy check have attributes to filter only entities that are proxies entities, with labels proxy_location = "wfr" and proxy_type = 'ping' or proxy_type = 'netapp-system'.

The second proxy check have attributes to filter only entities that are proxies entities, with labels proxy_location = "srt" and proxy_type = "temp-sensor".

Labels on the entity are: proxy_location = "srt" and proxy_type = "temp-sensor" so I expect that only the second proxy check to be executed on this proxy entity but the two checks are currently executed on the proxy entity.

Everything was working fine when the backend was in version 5.7.0. Everything is broken since we upgraded to 5.10.0. (Upgrade process from documentation has been followed).

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent): 5.10.0
• Installation method (packages, binaries, docker etc.): Ubuntu 18.04 packages
• Operating System and version (e.g. Ubuntu 14.04): Ubuntu 18.04

[Surian]

We just downgraded backend to 5.9.0 and we don't face this issue. So the bug has been introduced by the 5.10.0 release.

[becon-cmichel]

I can confirm that.

[palourde]

I was able to reproduce this issue and confirm that it was introduced in the 5.10.0 release. It might have been caused by https://github.com/sensu/sensu-go/pull/3016 or https://github.com/sensu/sensu-go/pull/3064.

Here are the resources I've used, which use the default namespace and only contain relevant attributes:

---
type: Entity
api_version: core/v2
metadata:
  labels:
    proxy_type: "temp-sensor"
    proxy_location: "srt"
    fqdn: "10.10.1.3"
    sensornumber: "S1"
    tempcritical: "280"
    tempwarning: "250"
  name: server-room-temp-probe1
  namespace: "default"
spec:
  entity_class: proxy
---
type: CheckConfig
api_version: core/v2
metadata:
  name: proxy-check-1
  namespace: default
spec:
  command: echo '{{ .labels.fqdn | default "localhost" }} -w {{ .labels.pingwarningrta | default 25 }},{{ .labels.pingwarningpl | default 20 }} -c {{ .labels.pingcriticalrta | default 45 }},{{ .labels.pingcriticalpl | default 40}}'
  interval: 60
  publish: true
  subscriptions:
    - entity:whisky
  proxy_requests:
    entity_attributes:
      - "entity.entity_class == 'proxy'"
      - "(entity.labels.proxy_type == 'ping' || entity.labels.proxy_type == 'netapp-system')"
      - "entity.labels.proxy_location == 'wfr'"
    splay: true
    splay_coverage: 50
  timeout: 15
  ttl: 120
---
type: CheckConfig
api_version: core/v2
metadata:
  name: proxy-check-2
  namespace: "default"
spec:
  command: echo '{{ .labels.sondenumber | default "S1" }} {{ .labels.tempcritical | default 280 }} {{ .labels.tempwarning | default 250 }} {{ .labels.fqdn }}'
  interval: 60
  publish: true
  subscriptions:
    - entity:whisky
  proxy_requests:
    entity_attributes:
      - "entity.entity_class == 'proxy'"
      - "entity.labels.proxy_type == 'temp-sensor'"
      - "entity.labels.proxy_location == 'srt'"
    splay: true
    splay_coverage: 50
  timeout: 10
  ttl: 120

[palourde]

For reference, I've identified the exact problem, a patch is on its way!

[schampseix]

Good news @palourde

[palourde]

Hello @Surian, @becon-cmichel & @schampseix!

We just released 5.10.1, which includes this bugfix: https://docs.sensu.io/sensu-go/5.10/release-notes/#5-10-1-release-notes.

Please let us know if you still face any issue!

Thanks