Kibana: [Task Manager] Task timeout does nothing if task run does not have a cancel method

[tsullivan]

Kibana version: 6.7-8.0/master

Describe the bug:

• A task instance can stay in state: running forever
• Task manager continuously logs a warning about a task that can not be cancelled. Example:
  log [16:33:44.786] [warning][task_manager] The task maps_telemetry "Maps-maps_telemetry" is not cancellable.

Steps to reproduce:

1. Kibana plugin that registers a task type and schedules a task. See the demo repo
2. Schedule a task that returns a Promise that never resolves. Test run function:
   async () => { return new Promise(() => { console.log('waiting for nothing to happen'); }); }
3. Eventually Kibana will repeatedly log:
   waiting for nothing to happen server log [20:11:13.361] [warning][task_manager] The task demo_form_scheduler "9--uzmgBXf7pSraMfOwb" is not cancellable. waiting for nothing to happen server log [20:16:13.984] [warning][task_manager] The task demo_form_scheduler "9--uzmgBXf7pSraMfOwb" is not cancellable. waiting for nothing to happen server log [20:21:14.973] [warning][task_manager] The task demo_form_scheduler "9--uzmgBXf7pSraMfOwb" is not cancellable. waiting for nothing to happen server log [20:26:15.838] [warning][task_manager] The task demo_form_scheduler "9--uzmgBXf7pSraMfOwb" is not cancellable. waiting for nothing to happen

Expected behavior:
Task Manager should mark the task as failed, remove the task instance, but ultimately stop re-trying

[elasticmachine]

Pinging @elastic/kibana-platform

[thomasneirynck]

I ran into this too at some point. not sure how to reproduce.

[thomasneirynck]

@tsullivan @aaronjcaldwell to remove the warning clutter from the console coming from the maps-app, would it be just sufficient to implement a cancel method on the maps-telemery task?

[tsullivan]

@thomasneirynck it's definitely a Task Manager / Platform bug. If the task runner doesn't return a cancellable promise, you should see a warning log 3 times (given retry logic is happening). But then the task should be marked as failed and not retried.

[mmack000]

How do you solve the problem, if the task never completes?

[tsullivan]

It would be nice if the Task Manager exposed a cancelable promise utility that plugins can use as a helper. I haven't worked on cancelable promises before, but there are/have been examples in other parts of the Kibana code that use Rxjs.

@chrisdavies had an idea to expose the helper in Task Manager phase 1, but it got scrapped.

[tsullivan]

How do you solve the problem, if the task never completes?

If the task never completes, the timeout check in Task Manager should mark it as a failed run. It will re-try after a backoff period. If the task fails X number of times, Task Manager will unschedule the task. Through this, there should be useful logging - a task that isn't completing is a bug that should be fixed.

[GNullev]

Hi everyone!
I installed ELK from repo v7 on the Fedora30 server.
And i have similar problem.
Can you help me?

[kimxogus]

I'm having the same problem with elasticsearch and kibana 7.2.0 in k8s.

[elasticmachine]

Pinging @elastic/kibana-stack-services

[afrokick]

ubuntu 16.04 with kibana 7.2.0 - also have an issue

However after downgraded to 7.1.1 it is working properly.

[Madko]

same problem on, CentOS kibana-7.3.0-1.x86_64

[bluikko]

The message started appearing after updating from 7.2.0 to 7.3.0.

It repeats every 1 minute.

Edit: the error message was displayed when Kibana + Elasticsearch was updated but only Kibana was automatically restarted. Restarting Elasticsearch made the warning (and the obvious "Kibana requires Elasticsearch 7.3.0"...) go away.

[elasticmachine]

Pinging @elastic/kibana-gis

[bmcconaghy]

Can someone on the maps team look at implementing a cancel function for the maps telemetry task?

[aaronjcaldwell]

My understanding was that this was an issue upstream in telemetry that happens to manifest in the maps app. I'll take a fresh look though and see if I can determine the exact issue

[bmcconaghy]

Task manager expects that tasks have cancel functions, and the maps telemetry one does not. There may also be an upstream issue, but the lack of a cancel function is definitely itself an issue.