Kibana: [warning][plugins][taskManager][taskManager] The task lens_telemetry "Lens-lens_telemetry" is not cancellable in Kibana logs

[bhavyarm]

Kibana version: 7.7.0 BC6

Elasticsearch version: 7.7.0 BC6

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: I was working on lens and area chart in the visualize when I noticed these log lines in Kibana logs

log   [18:12:25.231] [warning][plugins][taskManager][taskManager] The task lens_telemetry "Lens-lens_telemetry" is not cancellable.
  log   [18:12:25.231] [warning][plugins][taskManager][taskManager] The task vis_telemetry "oss_telemetry-vis_telemetry" is not cancellable.
  log   [18:16:25.437] [warning][plugins][taskManager][taskManager] The task lens_telemetry "Lens-lens_telemetry" is not cancellable.
  log   [18:23:25.762] [warning][plugins][taskManager][taskManager] The task vis_telemetry "oss_telemetry-vis_telemetry" is not cancellable.
  log   [18:31:26.214] [warning][plugins][taskManager][taskManager] The task lens_telemetry "Lens-lens_telemetry" is not cancellable.

[elasticmachine]

Pinging @elastic/pulse (Team:Pulse)

[bhavyarm]

cc @TinaHeiligers Thanks!

[bhavyarm]

cc @mikecote @gmmorris

[gmmorris]

I’m not sure why this is appearing now, but basically it looks like those tasks don’t have a cancel function specified, so when TM tries to cancel an overdue task (probably a telemetry task left over between Kibana restarts) it notifies us that it doesn’t know how to do that.

This behaviour isn’t new (predate me joining Elastic in fact :)) so I’m not sure why it hasn’t come up before.

The fix would be for those tasks to add a cancel function (even an empty one) along side their run function. But I’ll leave it to you to decide if that needs to be done in 7.7 or not. That might be up to whoever owns the tasks 🤷‍♂️

We can revisit whether this behaviour is correct, but probably not rushed into 7.7

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[afharo]

Thanks for finding it out @bhavyarm

I'm removing the Pulse mentions and labels since we don't own those components.

[mikecote]

This may also be an indicator that the task timed out after a few minutes. The logs could be caused by this and it’s trying to cancel the task before trying again.

[bhavyarm]

More log lines if its useful:

  log   [20:30:19.619] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection
 error  [22:33:55.142]  Error: Internal Server Error
    at HapiResponseAdapter.toError (/Users/bhavyarajumandya/Desktop/BC6_7.7.0/kibana-7.7.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:130:19)
    at HapiResponseAdapter.toHapiResponse (/Users/bhavyarajumandya/Desktop/BC6_7.7.0/kibana-7.7.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:84:19)
    at HapiResponseAdapter.handle (/Users/bhavyarajumandya/Desktop/BC6_7.7.0/kibana-7.7.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:79:17)
    at Router.handle (/Users/bhavyarajumandya/Desktop/BC6_7.7.0/kibana-7.7.0-darwin-x86_64/src/core/server/http/router/router.js:162:34)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  log   [13:02:22.590] [warning][plugins][taskManager][taskManager] The task apm-telemetry-task "apm-telemetry-task" is not cancellable.
  log   [13:02:23.095] [warning][plugins][taskManager][taskManager] Task apm-telemetry-task "apm-telemetry-task" failed in attempt to run: [task:apm-telemetry-task]: version conflict, required seqNo [9], primary term [2]. current document has seqNo [14] and primary term [2]: [version_conflict_engine_exception] [task:apm-telemetry-task]: version conflict, required seqNo [9], primary term [2]. current document has seqNo [14] and primary term [2], with { index_uuid="5qcobLhFRZOp7Kj6jPs-Gw" & shard="0" & index=".kibana_task_manager_2" }

[wylieconlon]

@gmmorris @mikecote what is supposed to happen in a cancel function? It seems like this warning was added, but I can't find any tasks that are actually set up correctly.

[gmmorris]

@wylieconlon that's really up to the task.
The best I can tell is that the assumption was (predates me, so inferring from context here) that a task will provide a function that will be called when Task Manager want to tell the task that it should stop whatever it's doing and abort. The assumption that the lack of a cancel function should mean the task can't be cancelled, rather than it meaning that it can simply be cancelled without requiring any custom behaviour _does_ feel like something we can revisit, but at the moment, that's not something we've invested time investigating.

if you don't feel there's any custom behaviour you want running when the task is cancelled, I'd suggest defining a cancel handler that simply logs out that the telemetry task didn't complete in time and was canceled by TM. If you don't care about logging it, then, that can simply be an empty function. 🤷‍♂️

[wylieconlon]

@gmmorris Okay, so given what you've said I think all of these tasks are cancellable. I am going to merge my PR that sets the cancellation function to empty for all tasks. https://github.com/elastic/kibana/pull/63756