Kibana: Task vis_telemetry "oss_telemetry-vis_telemetry" failed in attempt to run in default distro logs

[bhavyarm]

Kibana version: 7.3.0 BC5

Server OS version: darwin_x86_64

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: On default distro of BC5 without security on basic - these logs are seen in Kibana logs:

 log   [15:44:51.406] [warning][task_manager] Task vis_telemetry "oss_telemetry-vis_telemetry" failed in attempt to run: [version_conflict_engine_exception] [oss_telemetry-vis_telemetry]: version conflict, required seqNo [7], primary term [2]. current document has seqNo [8] and primary term [2], with { index_uuid="8x3gi6qSRZa59XLva_IPxQ" & shard="0" & index=".kibana_task_manager" }
  log   [15:44:51.426] [info][listening] Server running at http://localhost:5601

[elasticmachine]

Pinging @elastic/kibana-stack-services

[Bamieh]

@bhavyarm Is this still happening? I've tried to reproduce but i'm unable to find this error. I have a fresh ES and kibana install on version 7.x

[Bamieh]

Ah yes restarting kibana does reproduce this warning, not on the first run.

server    log   [13:39:30.169] [warning][telemetry] Error scheduling task, received [task:oss_telemetry-vis_telemetry]: version conflict, document already exists (current version [3]): [version_conflict_engine_exception] [task:oss_telemetry-vis_telemetry]: version conflict, document already exists (current version [3]), with { index_uuid="1YG4OLf7TSC65uhR5be-fA" & shard="0" & index=".kibana_task_manager_1" }
server    log   [13:39:30.170] [warning][maps] Error scheduling telemetry task, received [task:Maps-maps_telemetry]: version conflict, document already exists (current version [3]): [version_conflict_engine_exception] [task:Maps-maps_telemetry]: version conflict, document already exists (current version [3]), with { index_uuid="1YG4OLf7TSC65uhR5be-fA" & shard="0" & index=".kibana_task_manager_1" }

[Bamieh]

@mikecote Bill suggested to have a flag when creating a new task to query if the task already exists before trying to create a new one. This would solve this issue and possibly future similar ones. Alternatively we can try/catch and detect this error and swallow it per each task created for telemetry.

[bhavyarm]

@Bamieh I saw the same warning for maps too. Is that a general warning? Thanks!

[Bamieh]

@bhavyarm This warning occurs when the we attempt to create a task under the task manager that already exists. The above 2 warnings happen when we restart kibana as they are attempted to be created every time kibana is started.

[mikecote]

@Bamieh, @gmmorris will look into adding something like scheduleIfNotExists.

[freakingid]

@Bamieh @gmmorris is it safe to manually delete the 2 problematic tasks from kibana_task_manager when we run in to this issue? Thanks.

[gmmorris]

I never got a chance to look into this sadly.
@mikecote @Bamieh do you have enough background knowledge to answer @freakingid 's question?

[freakingid]

FYI, we deleted those tasks, then started Kibana successfully. However, due to network problems, we never had a successful browser session to Kibana. That is a separate issue, but I note that perhaps those 2 tasks do not get processed if there is never a browser session initiated? I say that because simply stopping the kibana service leaves the tasks orphaned.

[freakingid]

Oh, I guess that makes sense, from above:

The above 2 warnings happen when we restart kibana as they are attempted to be created every time kibana is started.

[nellicus]

I've just hit this on latest 7.4.1 using deb installation (ES+Kibana). seems quite serious as it totally prevents Kibana from starting up.

{"type":"log","@timestamp":"2019-10-27T22:12:41Z","tags":["warning","telemetry"],"pid":2003,"message":"Error scheduling task, received [task:oss_telemetry-vis_telemetry]: version conflict, document already exists (current version [3]): [version_conflict_engine_exception] [task:oss_telemetry-vis_telemetry]: version conflict, document already exists (current version [3]), with { index_uuid=\"568NokmeQiqchY-jX7XyJg\" & shard=\"0\" & index=\".kibana_task_manager_1\" }"}
{"type":"log","@timestamp":"2019-10-27T22:12:41Z","tags":["warning","maps"],"pid":2003,"message":"Error scheduling telemetry task, received [task:Maps-maps_telemetry]: version conflict, document already exists (current version [3]): [version_conflict_engine_exception] [task:Maps-maps_telemetry]: version conflict, document already exists (current version [3]), with { index_uuid=\"568NokmeQiqchY-jX7XyJg\" & shard=\"0\" & index=\".kibana_task_manager_1\" }"}

best steps to reproduce I can think of:

1) install ES and Kibana using deb packages
2) reboot the server
3) attempt to restart kibana service

[Kaltaoka]

I've just hit this on latest 7.4.1 using deb installation (ES+Kibana). seems quite serious as it totally prevents Kibana from starting up.

{"type":"log","@timestamp":"2019-10-27T22:12:41Z","tags":["warning","telemetry"],"pid":2003,"message":"Error scheduling task, received [task:oss_telemetry-vis_telemetry]: version conflict, document already exists (current version [3]): [version_conflict_engine_exception] [task:oss_telemetry-vis_telemetry]: version conflict, document already exists (current version [3]), with { index_uuid=\"568NokmeQiqchY-jX7XyJg\" & shard=\"0\" & index=\".kibana_task_manager_1\" }"}
{"type":"log","@timestamp":"2019-10-27T22:12:41Z","tags":["warning","maps"],"pid":2003,"message":"Error scheduling telemetry task, received [task:Maps-maps_telemetry]: version conflict, document already exists (current version [3]): [version_conflict_engine_exception] [task:Maps-maps_telemetry]: version conflict, document already exists (current version [3]), with { index_uuid=\"568NokmeQiqchY-jX7XyJg\" & shard=\"0\" & index=\".kibana_task_manager_1\" }"}

best steps to reproduce I can think of:

1. install ES and Kibana using deb packages

2. reboot the server

3. attempt to restart kibana service

Hi everyone,

I want confirm that with the new Windows installation of the Elastic Search and Kibana i also have the same problem.

I am doing this for testing purpose only

Scenario (if it helps...)
Running Windows version
Elasticsearch version 7.4.2
Kibana version 7.4.2

VM prepared: Windows Server 2019 Standard with the Elasticsearch installed on the machine (running OK), Kibana also on the same machine

When i run the first time e did run OK, didn't get this warnnings, but after i test to stop the kibana.bat and then restarting the Elasticsearch for the first time it started apearing this warning even after i did the restart to the server

[fredbcode]

Same issue here, kibana is stopped right after this message

Sometimes works, sometimes not ...

Kibana and elastic 7.4.2 linux

[andy5997090]

FYI, we deleted those tasks, then started Kibana successfully. However, due to network problems, we never had a successful browser session to Kibana. That is a separate issue, but I note that perhaps those 2 tasks do not get processed if there is never a browser session initiated? I say that because simply stopping the kibana service leaves the tasks orphaned.

How to manually delete those task?