Kibana: Watcher UI: A watch can be marked with an error status despite having executed fine

Created on 21 Aug 2018 · 6Comments · Source: elastic/kibana

Kibana version: 6.3.2
Elasticsearch version: 6.3.2

Describe the bug:

A watch action that has once failed in the past, can be forever marked as a failure/error in the UI, unless that error goes away. The reason for this is the fields that are being checked to mark sth as a failure.

Steps to reproduce:

Run these in the dev tools console

DELETE .watcher-history-*

DELETE _xpack/watcher/watch/my_watch

POST _xpack/watcher/watch/my_watch
{
  "trigger": {
    "schedule": {
      "interval": "10h"
    }
  },
  "input": {
    "simple": {}
  },
  "actions": {
    "send_email_to_admin": {
      "condition" : {
        "script" : "return ctx.payload.foo == true"
      },
      "email": {
        "to": "[email protected]",
        "from": "[email protected]",
        "subject": "subject",
        "body": {
          "text": "body"
        }
      }
    }
  }
}

GET _xpack/watcher/watch/my_watch

# action condition will execute
POST _xpack/watcher/watch/my_watch/_execute
{
  "record_execution": true,
  "alternative_input": {
    "foo": true
  }
}

# action condition will not execute
POST _xpack/watcher/watch/my_watch/_execute
{
  "record_execution": true,
  "alternative_input": {
    "foo": false
  },
  "action_modes": {
    "_all" : "force_execute"
  }
}

GET _xpack/watcher/watch/my_watch

To understand the issue, check the execute watch API calls. The first one will actually try to send an email, but fails because no email server is listening on the remote port. Thus the action is marked as successful: false.

The second watch execution is just not trying to send an email because the condition was not met. The last execution thus is now older than the last run of the watch, but still has the information that this particular execution has failed - which seems to mark the watch as error in the UI.

Expected behavior:

The watch should not be marked as an error in the UI, because the last run was just as expected. And the entry that seems to be checked seems to be older.

Screenshots (if relevant):

When checking the watcher UI