Kibana: [Ingest Manager] Saved Object client issues after ES restart

Created on 19 Aug 2020  ·  11Comments  ·  Source: elastic/kibana

More info coming. This is a WIP to get started

Kibana version:
8.x

Elasticsearch version:
8.0 snapshot

Steps to reproduce:

  1. start kibana: yarn start --no-base-path
  2. start ES: yarn es snapshot --license=trial -E xpack.security.authc.api_key.enabled=true
  3. no ingest-outputs or epm-packages saved objects at first
    ```

curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
"page": 1,
"per_page": 20,
"total": 0,
"saved_objects": []
}

> curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages" | jq
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}
```

  1. call setup
    curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme {"isInitialized":true}
  2. see expected saved objects
    curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages" // two objects, the default
  3. restart ES. Kibana disconnects, then eventually re-establishes with :9200
  4. _run same commands, see no saved objects_
    ```

curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages"
{"page":1,"per_page":20,"total":0,"saved_objects":[]}
curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
"page": 1,
"per_page": 20,
"total": 0,
"saved_objects": []
}
curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: ' --user elastic:changeme
{"isInitialized":true}
curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq
{
"page": 1,
"per_page": 20,
"total": 0,
"saved_objects": []
}
curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages"
{"page":1,"per_page":20,"total":0,"saved_objects":[]}
```

  1. restart Kibana, ES stays up
  2. previously created (but not shown) objects are now available
    curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=ingest-outputs" | jq { "page": 1, "per_page": 20, "total": 1, "saved_objects": [ { "type": "ingest-outputs", "id": "8794b999-cb1a-4e58-b551-831ac164a42d", "attributes": { "name": "default", "is_default": true, "type": "elasticsearch", "hosts": [ "http://localhost:9200" ] }, "references": [], "updated_at": "2020-08-19T20:45:29.309Z", "version": "WzE5LDFd", "score": 0 } ] } curl --user elastic:changeme "localhost:5601/api/saved_objects/_find?type=epm-packages" {"page":1,"per_page":20,"total":2,"saved_objects":[ ... ]}

Describe the feature:

After ES restarts, our SO client isn't able to read previously created saved objects. After Kibana restarts, all the expected objects are available.

Video showing flow like described above
master-fleet-setup-forceRecreate-with-ES-restart-errors-no-default-output-2048-half.m4v.zip

It affects at least two /api/ingest_manager routes, but didn't seem to affect the SO API (curl --user elastic:changeme -X PUT "localhost:9200/_ingest/pipeline/my-pipeline-id").

I tried adding a refresh: 'wait_for' and refresh: true to the soClient.create calls but it didn't have any effect

Ingest Management
Source
picture jfsiii

All 11 comments

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine picture elasticmachine on 20 Aug 2020

Thanks for filing this issue! I know that we had seen this behavior before but didn't log it. One thing I'm curious about, is if this behavior only occurs for our SO types, or does it apply to other Kibana SOs as well?

jen-huang picture jen-huang on 20 Aug 2020

@elastic/kibana-platform any thoughts about how I can debug / fix?

Here's the pattern we use to get the Saved Object client from a route handler and pass it to a service
https://github.com/elastic/kibana/blob/cefb62c2c30a2907d5f3aabede4d0a6cba2fae64/x-pack/plugins/ingest_manager/server/routes/setup/handlers.ts#L79-L80

Are we "holding it wrong"? It feels like a flush issue which is what had me trying the refresh values, but if it was about the way we used the ES client, I'd expect it to have the same "write but not read" issue the first time it's run. This behavior only seems to appear after an ES restart.

Let me know if I can do anything more to help diagnose.

jfsiii picture jfsiii on 25 Aug 2020

@nchaulet @skh @neptunian for awareness.

ph picture ph on 25 Aug 2020

Is wiping all ES data, including .kibana indices, without restarting KIbana a supported scenario? I could imagine that this affects all saved objects (as @jen-huang already mentions), and maybe even more areas in the platform.

skh picture skh on 26 Aug 2020

@skh the failures / issues we're seeing aren't because .kibana or other data was removed. They're because we cannot persist changes after the restart. And that seems related to whatever is going on with the client.

/setup would happily re-create a default output, install default packages, etc after a restart/wipe. The issue is that when it tries, ES says it successfully created an output, even returning the new object, but then it's not available for another part of the system that expects it. Some of the system is resilient enough to create something if it's missing but, again, those updates are "succeeding" but not updating ES (or not available to be read via the ES client).

Updates not failing, but not ... updating is a challenge.

jfsiii picture jfsiii on 26 Aug 2020

The queries you run above are based on the SO API. I wonder if you would run the query directly on the .kibana index and search for the documents if they are there or not.

ruflin picture ruflin on 26 Aug 2020

the failures / issues we're seeing aren't because .kibana or other data was removed.

My guess is that Kibana does some housekeeping on startup, including setting up its own indices and the saved object client, and makes the assumption that its indices stay where they are even if the connection to ES is intermittently lost.

Restarting yarn es snapshot is a clean wipe, so (I'm still guessing) the still-running Kibana makes assumptions about indices that are now gone. When Kibana is restarted, it runs through its own startup code again, and finds (or creates?) the new indices.

If I'm correct, I could well imagine that this is a scenario that is not supported, either intentionally or by oversight, hence my question.

@elastic/kibana-platform is of course better qualified to answer that than I am ;-)

skh picture skh on 26 Aug 2020
👍1

@skh TL;DR: "different indexes" didn't occur to me & is looking more likely

server    log   [10:26:49.931] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [10:26:49.932] [info][savedobjects-service] Starting saved objects migrations
server    log   [10:26:49.954] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server    log   [10:26:49.959] [info][savedobjects-service] Creating index .kibana_1.
server    log   [10:26:50.523] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server    log   [10:26:50.573] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.

You reminded me that I thought I saw logs from ES during the failing operations.

Here's what's logged in ES during an initial or successful call to curl --user elastic:changeme -X POST 'http://localhost:5601/api/ingest_manager/setup' --header 'kbn-xsrf: <string>'

ES logs from POST /setup success (before restart)

   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata_current-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata_current] for index patterns [metrics-endpoint.metadata_current-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.entropy] for index patterns [metrics-system.entropy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network_summary] for index patterns [metrics-system.network_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.raid] for index patterns [metrics-system.raid-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket] for index patterns [metrics-system.socket-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.service] for index patterns [metrics-system.service-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.users] for index patterns [metrics-system.users-*]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]

ES logs from POST /setup after restart

   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata_current-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata_current] for index patterns [metrics-endpoint.metadata_current-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.entropy] for index patterns [metrics-system.entropy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network_summary] for index patterns [metrics-system.network_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.raid] for index patterns [metrics-system.raid-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.service] for index patterns [metrics-system.service-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket] for index patterns [metrics-system.socket-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.users] for index patterns [metrics-system.users-*]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]

I noticed there are several lines like this with .kibana_1 in the initial logs

   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/Ltn-IgekTOyTNk8JH9UoEw] update_mapping [_doc]

but .kibana after restart

   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana/OYeNf0mKRjSXttNUpoQyNw] update_mapping [_doc]

The "different indexes" hypothesis gets stronger 😃

I captured the logs from Kibana & ES during inital boot & restart
es_initial.log.txt
es_restart.log.txt
kibana_while_es_restarts.log.txt
kibana_initial.log.txt

Looking more closely at the Kibana boot logs for .kibana, I saw

server    log   [10:26:49.931] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
server    log   [10:26:49.932] [info][savedobjects-service] Starting saved objects migrations
server    log   [10:26:49.954] [info][savedobjects-service] Creating index .kibana_task_manager_1.
server    log   [10:26:49.959] [info][savedobjects-service] Creating index .kibana_1.
server    log   [10:26:50.523] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1.
server    log   [10:26:50.573] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.

That seems relevant 😃

jfsiii picture jfsiii on 26 Aug 2020

Things appear to work as expected, if I start & restart ES with the local binary (like .es/8.0.0/bin/elasticsearch created by yarn es snapshot)

I think I misunderstood _why_ there was no data after restarting with yarn es snapshot

I think I'll close this but, either way, thanks for helping me work through this @skh

jfsiii picture jfsiii on 26 Aug 2020
❤1

yarn es snapshot, in case it is relevant, also takes an optional parameter to specify a data directory, which will survive restarts if it's outside the snapshot folder:

yarn es snapshot -E path.data=../data

(from https://www.elastic.co/guide/en/kibana/master/running-elasticsearch.html )

skh picture skh on 26 Aug 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tbragin picture tbragin  ·  81Comments
JulienPalard picture JulienPalard  ·  95Comments
seti123 picture seti123  ·  100Comments
pkubat picture pkubat  ·  75Comments
bquartier picture bquartier  ·  79Comments