Kibana: [Ingest Management]: Error 'Unable to initialize Ingest Manager: Bad Request' is displayed on clicking 'Ingest Manager' link on 8.0.0-SNAPSHOT Kibana

Created on 10 Sep 2020  ·  25Comments  ·  Source: elastic/kibana

Kibana version:
Kibana: 8.0.0-SNAPSHOT

Elasticsearch version:
Elasticsearch: 8.0.0-SNAPSHOT

Host OS and Browser version:
Windows 10 , All

Original install method (e.g. download page, yum, from source, etc.):
From 8.0.0-SNAPSHOT on staging cloud environment

Description
Preconditions

  1. 8.0.0-SNAPSHOT Kibana cloud environment should be available.

Steps to Reproduce

  1. Open the 8.0.0-SNAPSHOT Kibana cloud environment in browser.
  2. Open the left hand menu and click 'Ingest Manager' link under Management.
  3. Observe that error 'Unable to initialize Ingest Manager: Bad Request' is displayed.

Test data
N/A

Impacted Test case id
N/A

Actual Result
Expected Result
What's working

  • Issue is not occurring on 7.9.2-SNAPSHOT Kibana cloud environment.
  • Issue is not occurring on refreshing the page.

What's not working

  • N/A

Screenshot
blocker_8 0

Logs
Error: 'Failed to load resource: the server responded with a status of 500 ()' is displayed in browser console logs.

blocker_console_logs

Ingest Management bug high
Source
picture rahulgupta-qasource

Most helpful comment

An additional option to get testing going is to run the following curl command after your 8.0 snapshot build completes and you get your Kibana URL. Do the below BEFORE going to the Ingest page.

  • Get the username <user>, it's usually elastic
  • Get your password <pass>, the long string that's randomly generated
  • Get your Kibana URL <kibana-url>, you can either "Copy Endpoint" on the cloud screen or just copy the URL including the port from your browser's address bar.

Open a terminal, fill in the above, run this command. After that Ingest should load.

curl -X POST -H 'kbn-xsrf: xxx' --user <user>:<pass> \
 -H "Content-Type: application/json" -d '{"force" : true}' \
 <kibana-url>/api/ingest_manager/epm/packages/endpoint-0.15.0

The above is only necessary until we get another 8.0 SNAPSHOT build. The root issue should be fixed there.

picture kevinlog on 17 Sep 2020
❤2 👍2

All 25 comments

@karanbirsingh-qasource Please review the bug.

rahulgupta-qasource picture rahulgupta-qasource on 10 Sep 2020

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine picture elasticmachine on 10 Sep 2020

Reviewed and assigned to @EricDavisX

karanbirsingh-qasource picture karanbirsingh-qasource on 10 Sep 2020

looks to me that this will happen every time the user opens Kibana again, the first time they access Ingest Manager the setup api gives a 500, then it returns ok the subsequent times:
POST to /api/ingest_manager/setup

I agree with High impact therefore. Assigning to PH to see who can take a look

EricDavisX picture EricDavisX on 10 Sep 2020

@jfsiii Can you take the lead to reproduce that issue?

ph picture ph on 10 Sep 2020

Bringing Slack convo here, I can reproduce in Cloud and am looking into this now. I'm taking snapshots & HAR files and will try to reproduce locally.

jfsiii picture jfsiii on 10 Sep 2020

_Update_: TL;DR: can repro on Cloud but I see nothing in logs that gives a clue what's happening. Next snapshot should have better detail about the source.

Is there a way to determine which commits are in a snapshot/build? I want to see if certain PRs are included.

staging instance

kibana: https://1aef124d8bc9424ea7f6e2ca049ca7e0.us-central1.gcp.foundit.no:9243/app/home
deployment: ab7982fed4464548905609779c2be080
ece.cluster: 1aef124d8bc9424ea7f6e2ca049ca7e0

On plugin load in browser
POST /ingest_manager/setup fails w/500 & POST /ingest_manager/setup fails with 502
Screen Shot 2020-09-10 at 9 40 05 AM
Screen Shot 2020-09-10 at 9 40 12 AM
POST /ingest_manager/setup response
Screen Shot 2020-09-10 at 9 40 52 AM

Kibana logs OK
Screen Shot 2020-09-10 at 11 34 18 AM

And looking at ES all the appropriate "adding *" messages appear to be in
Screen Shot 2020-09-10 at 12 57 55 PM

The "Stopping all plugins" in the Kibana logs 30 seconds before the "Internal Server Error" from our request seems noteworthy. However, I later triggered the 500 a few times and there were no other "Stopping ..." messages.

Screen Shot 2020-09-10 at 11 34 30 AM

There are some "Request error, retrying" messages which seem to point to ES connectivity issues, but they aren't present near _all_ the errors, so ...

I can't find any interesting activity on ES around the time of the failures. See 500's at 16:36 & 16:38 here
Screen Shot 2020-09-10 at 12 42 35 PM

But nothing on ES at that time
Screen Shot 2020-09-10 at 12 42 54 PM

jfsiii picture jfsiii on 10 Sep 2020

I twice tried to create a deploy from an 8.x snapshot this morning but both failed. The ES instance wouldn't start (due to an issue with the plan?). It's also a company-wide holiday today so I'll check back on Monday and see how things look.

jfsiii picture jfsiii on 11 Sep 2020

Is there a way to determine which commits are in a snapshot/build? I want to see if certain PRs are included.

On the Kibana side, the top level /status endpoint should give you the git commit hash the snapshot is built from. (As in: http(s)://KIBANA_URL:KIBANA_PORT/status , don't include api)

skh picture skh on 14 Sep 2020

@skh thanks, I should have resolved that question. @EricDavisX gave me the tip in Slack and it worked great.

Latest update: I cannot build an 8.x snapshot on staging. ES is failing with a bad plan. I've pinged the Cloud Slack channel with the details and I will update again later with any changes.

jfsiii picture jfsiii on 14 Sep 2020

We found a bug. The Cloud team is aware and tracking in a ticket. I asked about expediting a fix. Will update more later.

jfsiii picture jfsiii on 14 Sep 2020

Cloud is rolling back the source of the issue. The expectation is that we'll be able to deploy snapshots again today and I can resume investigating.

jfsiii picture jfsiii on 15 Sep 2020

The rollback is complete, but not yet available in all zones. AWS eu-central definitely works, though

Screen Shot 2020-09-15 at 10 45 48 AM

I deployed an 8.x snapshot (f5a9fecfc854456387d94cdedcfb8a94) and got a more detailed error

Screen Shot 2020-09-15 at 11 08 45 AM

This looks to be a forwarded 400 response from Elasticsearch. Pinging @elastic/endpoint-management for visibility and any suggestions. I'll post in Slack as well.

jfsiii picture jfsiii on 15 Sep 2020

I tried this:

Reproduce locally

  1. Try reproducing locally (local ES, local Kibana)
  2. Everything works as expected, transform runs, Endpoint UI works ✅

Reproduce locally by connecting local Kibana to Cloud ES

  1. Created a cloud instance in staging, 8.0
  2. Connected my local Kibana to the remote ES in cloud
  3. Opened Ingest
  4. Got the same error as reported in the bug ❌

Comment out transform code locally, connect local Kibana to fresh Cloud ES

  1. Created a cloud instance in staging, 8.0
  2. Commented out the code locally which installs the transform
  3. Connected to my local Kibanan to the remote (and fresh) ES in cloud
  4. Opened Ingest
  5. UI loaded, but code this 500 error from a setup() call ❌ 
GET http://localhost:5601/api/ingest_manager/fleet/setup
error: "Internal Server Error"
message: "Unable to decrypt attribute "fleet_enroll_username""
statusCode: 500

The Fleet page remains spinning in the UI
image

Some logs from the console

server    log   [18:26:09.901] [error][ingestManager][plugins] { Error: Unable to decrypt attribute "fleet_enroll_username"
    at EncryptedSavedObjectsService.attributesToDecryptIterator (/Users/klogan/kibana-kev/kibana/x-pack/plugins/encrypted_saved_objects/server/crypto/encrypted_saved_objects_service.ts:407:15)
    at attributesToDecryptIterator.throw (<anonymous>)
    at EncryptedSavedObjectsService.decryptAttributes (/Users/klogan/kibana-kev/kibana/x-pack/plugins/encrypted_saved_objects/server/crypto/encrypted_saved_objects_service.ts:336:9)
  attributeName: 'fleet_enroll_username',
  operation: 1,
  cause:
   Error: Unsupported state or unable to authenticate data
       at Decipheriv.final (internal/crypto/cipher.js:164:28)
       at decrypt (/Users/klogan/kibana-kev/kibana/x-pack/node_modules/@elastic/node-crypto/src/crypto.ts:157:73)
       at Object.<anonymous> (/Users/klogan/kibana-kev/kibana/x-pack/node_modules/@elastic/node-crypto/src/crypto.ts:188:14)
       at step (/Users/klogan/kibana-kev/kibana/x-pack/node_modules/@elastic/node-crypto/lib/crypto.js:45:23)
       at Object.next (/Users/klogan/kibana-kev/kibana/x-pack/node_modules/@elastic/node-crypto/lib/crypto.js:26:53)
       at fulfilled (/Users/klogan/kibana-kev/kibana/x-pack/node_modules/@elastic/node-crypto/lib/crypto.js:17:58) }

FYI @jfsiii @nnamdifrankie @neptunian

kevinlog picture kevinlog on 16 Sep 2020
👍1

Sorry for the short message now, but I don't have much time until tomorrow.

I added a bunch of logging in the install path (mostly server/services/epm/packages/install.ts) and did the "local kibana + cloud es" setup. Here's the log I got from setup

install failure

server    log   [20:45:28.866] [info][server][Kibana][http] http server running at http://0.0.0.0:5601
setupIngestManager called
awaitIfPending called
awaitIfPending try
createSetupSideEffects called
createSetupSideEffects call ensureDefault*, etc in parallel. wait for result
ensureInstalledPackage system { installed_kibana:
   [ { id: 'system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'dashboard' },
     { id: 'system-277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs',
       type: 'dashboard' },
     { id: 'system-5517a150-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'dashboard' },
     { id: 'system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs',
       type: 'dashboard' },
     { id: 'system-CPU-slash-Memory-per-container-ecs',
       type: 'dashboard' },
     { id: 'system-Filebeat-syslog-dashboard-ecs',
       type: 'dashboard' },
     { id: 'Metrics-system-overview-ecs', type: 'dashboard' },
     { id: 'system-c431f410-f9ac-11e9-90e8-1fb18e796788',
       type: 'dashboard' },
     { id: '62439dc0-f9c9-11e6-a747-6121780e0414-ecs',
       type: 'search' },
     { id: '8030c1b0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'search' },
     { id: 'Syslog-system-logs-ecs', type: 'search' },
     { id: 'b6f321e0-fa25-11e6-bbd3-29c986c96e5a-ecs',
       type: 'search' },
     { id: 'eb0039f0-fa7f-11e6-a1df-a78bd7504d38-ecs',
       type: 'search' },
     { id: '089b85d0-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '12667040-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs',
       type: 'visualization' },
     { id: '1aae9140-1b93-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '2e224660-1b19-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '327417e0-8462-11e7-bab8-bd2f0fb42c54-ecs',
       type: 'visualization' },
     { id: '341ffe70-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '346bb290-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '3cec3eb0-f9d3-11e6-8a3e-2b904044ea1d-ecs',
       type: 'visualization' },
     { id: '3d65d450-a9c3-11e7-af20-67db8aecb295-ecs',
       type: 'visualization' },
     { id: '4b254630-f998-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '4d546850-1b15-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '51164310-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs',
       type: 'visualization' },
     { id: '5c7af030-fa2a-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '5dd15c00-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: '6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs',
       type: 'visualization' },
     { id: '78b74f30-f9cd-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: '825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs',
       type: 'visualization' },
     { id: '83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '855899e0-1b1c-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '8c071e20-f999-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '96976150-4d5d-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: '99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: '9c69cad0-f9b0-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'Container-Block-IO-ecs', type: 'visualization' },
     { id: 'Container-CPU-usage-ecs', type: 'visualization' },
     { id: 'Container-Memory-stats-ecs', type: 'visualization' },
     { id: 'Syslog-events-by-hostname-ecs', type: 'visualization' },
     { id: 'Syslog-hostnames-and-processes-ecs',
       type: 'visualization' },
     { id: 'System-Navigation-ecs', type: 'visualization' },
     { id: 'a30871f0-f98f-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'bb3a8720-f991-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: 'c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: 'd16bb400-f9cc-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: 'd2e80340-4d5c-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: 'd3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: 'd3f51850-f9b6-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'd56ee420-fa79-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'dc589770-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: 'e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'e121b140-fa78-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'e6e639e0-f992-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'f398d2f0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: 'fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' } ],
  installed_es:
   [ { id: 'logs-system.auth-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.auth-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.auth', type: 'index_template' },
     { id: 'metrics-system.core', type: 'index_template' },
     { id: 'metrics-system.cpu', type: 'index_template' },
     { id: 'metrics-system.diskio', type: 'index_template' },
     { id: 'metrics-system.entropy', type: 'index_template' },
     { id: 'metrics-system.filesystem', type: 'index_template' },
     { id: 'metrics-system.fsstat', type: 'index_template' },
     { id: 'metrics-system.load', type: 'index_template' },
     { id: 'metrics-system.memory', type: 'index_template' },
     { id: 'metrics-system.network', type: 'index_template' },
     { id: 'metrics-system.network_summary', type: 'index_template' },
     { id: 'metrics-system.process', type: 'index_template' },
     { id: 'metrics-system.process_summary', type: 'index_template' },
     { id: 'metrics-system.raid', type: 'index_template' },
     { id: 'metrics-system.service', type: 'index_template' },
     { id: 'metrics-system.socket', type: 'index_template' },
     { id: 'metrics-system.socket_summary', type: 'index_template' },
     { id: 'logs-system.syslog', type: 'index_template' },
     { id: 'metrics-system.uptime', type: 'index_template' },
     { id: 'metrics-system.users', type: 'index_template' } ],
  es_index_patterns:
   { auth: 'logs-system.auth-*',
     core: 'metrics-system.core-*',
     cpu: 'metrics-system.cpu-*',
     diskio: 'metrics-system.diskio-*',
     entropy: 'metrics-system.entropy-*',
     filesystem: 'metrics-system.filesystem-*',
     fsstat: 'metrics-system.fsstat-*',
     load: 'metrics-system.load-*',
     memory: 'metrics-system.memory-*',
     network: 'metrics-system.network-*',
     network_summary: 'metrics-system.network_summary-*',
     process: 'metrics-system.process-*',
     process_summary: 'metrics-system.process_summary-*',
     raid: 'metrics-system.raid-*',
     service: 'metrics-system.service-*',
     socket: 'metrics-system.socket-*',
     socket_summary: 'metrics-system.socket_summary-*',
     syslog: 'logs-system.syslog-*',
     uptime: 'metrics-system.uptime-*',
     users: 'metrics-system.users-*' },
  name: 'system',
  version: '0.5.3',
  internal: false,
  removable: false,
  install_version: '0.5.3',
  install_status: 'installed',
  install_started_at: '2020-09-16T00:42:50.907Z' }
installPackage endpoint-0.16.0-dev.0
ensureInstalledPackage endpoint { installed_kibana: [],
  installed_es:
   [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.file-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.library-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.network-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.policy-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.process-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.security-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.alerts', type: 'index_template' },
     { id: 'logs-endpoint.events.file', type: 'index_template' },
     { id: 'logs-endpoint.events.library', type: 'index_template' },
     { id: 'metrics-endpoint.metadata', type: 'index_template' },
     { id: 'metrics-endpoint.metadata_current',
       type: 'index_template' },
     { id: 'metrics-endpoint.metrics', type: 'index_template' },
     { id: 'logs-endpoint.events.network', type: 'index_template' },
     { id: 'metrics-endpoint.policy', type: 'index_template' },
     { id: 'logs-endpoint.events.process', type: 'index_template' },
     { id: 'logs-endpoint.events.registry', type: 'index_template' },
     { id: 'logs-endpoint.events.security', type: 'index_template' },
     { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
       type: 'transform' } ],
  es_index_patterns:
   { alerts: 'logs-endpoint.alerts-*',
     file: 'logs-endpoint.events.file-*',
     library: 'logs-endpoint.events.library-*',
     metadata: 'metrics-endpoint.metadata-*',
     metadata_current: 'metrics-endpoint.metadata_current-*',
     metrics: 'metrics-endpoint.metrics-*',
     network: 'logs-endpoint.events.network-*',
     policy: 'metrics-endpoint.policy-*',
     process: 'logs-endpoint.events.process-*',
     registry: 'logs-endpoint.events.registry-*',
     security: 'logs-endpoint.events.security-*' },
  name: 'endpoint',
  version: '0.16.0-dev.0',
  internal: false,
  removable: false,
  install_version: '0.16.0-dev.0',
  install_status: 'installing',
  install_started_at: '2020-09-16T00:42:50.768Z' }
installPackage installedPkg { id: 'endpoint',
  type: 'epm-packages',
  namespaces: [],
  updated_at: '2020-09-16T00:42:58.910Z',
  version: 'WzExMywxXQ==',
  attributes:
   { installed_kibana: [],
     installed_es:
      [ [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object] ],
     es_index_patterns:
      { alerts: 'logs-endpoint.alerts-*',
        file: 'logs-endpoint.events.file-*',
        library: 'logs-endpoint.events.library-*',
        metadata: 'metrics-endpoint.metadata-*',
        metadata_current: 'metrics-endpoint.metadata_current-*',
        metrics: 'metrics-endpoint.metrics-*',
        network: 'logs-endpoint.events.network-*',
        policy: 'metrics-endpoint.policy-*',
        process: 'logs-endpoint.events.process-*',
        registry: 'logs-endpoint.events.registry-*',
        security: 'logs-endpoint.events.security-*' },
     name: 'endpoint',
     version: '0.16.0-dev.0',
     internal: false,
     removable: false,
     install_version: '0.16.0-dev.0',
     install_status: 'installing',
     install_started_at: '2020-09-16T00:42:50.768Z' },
  references: [],
  migrationVersion: undefined }
installPackage save new kibana refs before installing the assets
installPackage saveKibanaAssetsRefs
installPackage installKibanaAssets
deleteTransforms [ 'metrics-endpoint.metadata-current-default-0.16.0-dev.0' ]
stopTransforms [ 'metrics-endpoint.metadata-current-default-0.16.0-dev.0' ]
POST _stop metrics-endpoint.metadata-current-default-0.16.0-dev.0
awaitIfPending ok. result: undefined
awaitIfPending onResolve(result)
awaitIfPending isPending=false false
awaitIfPending end. return a promise which will resolve/reject Promise {
  <rejected> { Error: Bad Request
      at respond (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
      at checkRespForFailure (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
      at HttpConnector.<anonymous> (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
      at IncomingMessage.wrapper (/Users/jfsiii/work/kibana/node_modules/lodash/lodash.js:4949:19)
      at IncomingMessage.emit (events.js:203:15)
      at endReadableNT (_stream_readable.js:1145:12)
      at process._tickCallback (internal/process/next_tick.js:63:19)
    status: 400,
    displayName: 'BadRequest',
    message: 'Bad Request',
    path:
     '_transform/metrics-endpoint.metadata-current-default-0.16.0-dev.0/_stop',
    query: 'force=true',
    body: '400 Bad Request',
    statusCode: 400,
    response: '400 Bad Request',
    toString: [Function],
    toJSON: [Function] } }
server    log   [20:45:54.321] [error][ingestManager][plugins] { Error: Bad Request
    at respond (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/jfsiii/work/kibana/node_modules/lodash/lodash.js:4949:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  displayName: 'BadRequest',
  message: 'Bad Request',
  path:
   '_transform/metrics-endpoint.metadata-current-default-0.16.0-dev.0/_stop',
  query: 'force=true',
  body: '400 Bad Request',
  statusCode: 400,
  response: '400 Bad Request',
  toString: [Function],
  toJSON: [Function] }
server    log   [20:45:54.322] [error][ingestManager][plugins] Bad Request response from _transform/metrics-endpoint.metadata-current-default-0.16.0-dev.0/_stop: 400 Bad Request

jfsiii picture jfsiii on 16 Sep 2020

@jfsiii
From your logs the transform for version 0.16.0.dev.0 already exist before the transform install is called. I think one of the prior installers is saving the transform ref.

nnamdifrankie picture nnamdifrankie on 16 Sep 2020

I added some more console.logs and pushed those changes to a branch if anyone wants to try.

Here's logs from two scenarios I ran

local Kibana / Cloud ES (❌ like OP)

local Kibana logs [1]

setupIngestManager called
awaitIfPending called
awaitIfPending try
createSetupSideEffects called
createSetupSideEffects call ensureDefault*, etc in parallel. wait for result
ensureInstalledPackage system { installed_kibana:
   [ { id: 'system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'dashboard' },
     { id: 'system-277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs',
       type: 'dashboard' },
     { id: 'system-5517a150-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'dashboard' },
     { id: 'system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs',
       type: 'dashboard' },
     { id: 'system-CPU-slash-Memory-per-container-ecs',
       type: 'dashboard' },
     { id: 'system-Filebeat-syslog-dashboard-ecs',
       type: 'dashboard' },
     { id: 'Metrics-system-overview-ecs', type: 'dashboard' },
     { id: 'system-c431f410-f9ac-11e9-90e8-1fb18e796788',
       type: 'dashboard' },
     { id: '62439dc0-f9c9-11e6-a747-6121780e0414-ecs',
       type: 'search' },
     { id: '8030c1b0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'search' },
     { id: 'Syslog-system-logs-ecs', type: 'search' },
     { id: 'b6f321e0-fa25-11e6-bbd3-29c986c96e5a-ecs',
       type: 'search' },
     { id: 'eb0039f0-fa7f-11e6-a1df-a78bd7504d38-ecs',
       type: 'search' },
     { id: '089b85d0-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '12667040-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs',
       type: 'visualization' },
     { id: '1aae9140-1b93-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '2e224660-1b19-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '327417e0-8462-11e7-bab8-bd2f0fb42c54-ecs',
       type: 'visualization' },
     { id: '341ffe70-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '346bb290-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '3cec3eb0-f9d3-11e6-8a3e-2b904044ea1d-ecs',
       type: 'visualization' },
     { id: '3d65d450-a9c3-11e7-af20-67db8aecb295-ecs',
       type: 'visualization' },
     { id: '4b254630-f998-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '4d546850-1b15-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '51164310-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs',
       type: 'visualization' },
     { id: '5c7af030-fa2a-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '5dd15c00-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: '6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs',
       type: 'visualization' },
     { id: '78b74f30-f9cd-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: '825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs',
       type: 'visualization' },
     { id: '83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '855899e0-1b1c-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '8c071e20-f999-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '96976150-4d5d-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: '99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: '9c69cad0-f9b0-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'Container-Block-IO-ecs', type: 'visualization' },
     { id: 'Container-CPU-usage-ecs', type: 'visualization' },
     { id: 'Container-Memory-stats-ecs', type: 'visualization' },
     { id: 'Syslog-events-by-hostname-ecs', type: 'visualization' },
     { id: 'Syslog-hostnames-and-processes-ecs',
       type: 'visualization' },
     { id: 'System-Navigation-ecs', type: 'visualization' },
     { id: 'a30871f0-f98f-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'bb3a8720-f991-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: 'c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: 'd16bb400-f9cc-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: 'd2e80340-4d5c-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: 'd3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: 'd3f51850-f9b6-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'd56ee420-fa79-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'dc589770-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: 'e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'e121b140-fa78-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'e6e639e0-f992-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'f398d2f0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: 'fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' } ],
  installed_es:
   [ { id: 'logs-system.auth-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.auth-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.3', type: 'ingest_pipeline' },
     { id: 'logs-system.auth', type: 'index_template' },
     { id: 'metrics-system.core', type: 'index_template' },
     { id: 'metrics-system.cpu', type: 'index_template' },
     { id: 'metrics-system.diskio', type: 'index_template' },
     { id: 'metrics-system.entropy', type: 'index_template' },
     { id: 'metrics-system.filesystem', type: 'index_template' },
     { id: 'metrics-system.fsstat', type: 'index_template' },
     { id: 'metrics-system.load', type: 'index_template' },
     { id: 'metrics-system.memory', type: 'index_template' },
     { id: 'metrics-system.network', type: 'index_template' },
     { id: 'metrics-system.network_summary', type: 'index_template' },
     { id: 'metrics-system.process', type: 'index_template' },
     { id: 'metrics-system.process_summary', type: 'index_template' },
     { id: 'metrics-system.raid', type: 'index_template' },
     { id: 'metrics-system.service', type: 'index_template' },
     { id: 'metrics-system.socket', type: 'index_template' },
     { id: 'metrics-system.socket_summary', type: 'index_template' },
     { id: 'logs-system.syslog', type: 'index_template' },
     { id: 'metrics-system.uptime', type: 'index_template' },
     { id: 'metrics-system.users', type: 'index_template' } ],
  es_index_patterns:
   { auth: 'logs-system.auth-*',
     core: 'metrics-system.core-*',
     cpu: 'metrics-system.cpu-*',
     diskio: 'metrics-system.diskio-*',
     entropy: 'metrics-system.entropy-*',
     filesystem: 'metrics-system.filesystem-*',
     fsstat: 'metrics-system.fsstat-*',
     load: 'metrics-system.load-*',
     memory: 'metrics-system.memory-*',
     network: 'metrics-system.network-*',
     network_summary: 'metrics-system.network_summary-*',
     process: 'metrics-system.process-*',
     process_summary: 'metrics-system.process_summary-*',
     raid: 'metrics-system.raid-*',
     service: 'metrics-system.service-*',
     socket: 'metrics-system.socket-*',
     socket_summary: 'metrics-system.socket_summary-*',
     syslog: 'logs-system.syslog-*',
     uptime: 'metrics-system.uptime-*',
     users: 'metrics-system.users-*' },
  name: 'system',
  version: '0.5.3',
  internal: false,
  removable: false,
  install_version: '0.5.3',
  install_status: 'installed',
  install_started_at: '2020-09-16T00:42:50.907Z' }
installPackage endpoint-0.16.0-dev.0
ensureInstalledPackage endpoint { installed_kibana: [],
  installed_es:
   [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.file-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.library-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.network-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.policy-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.process-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.security-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
       type: 'transform' },
     { id: 'logs-endpoint.alerts-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.file-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.library-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.network-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.policy-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.process-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.security-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.alerts', type: 'index_template' },
     { id: 'logs-endpoint.events.file', type: 'index_template' },
     { id: 'logs-endpoint.events.library', type: 'index_template' },
     { id: 'metrics-endpoint.metadata', type: 'index_template' },
     { id: 'metrics-endpoint.metadata_current',
       type: 'index_template' },
     { id: 'metrics-endpoint.metrics', type: 'index_template' },
     { id: 'logs-endpoint.events.network', type: 'index_template' },
     { id: 'metrics-endpoint.policy', type: 'index_template' },
     { id: 'logs-endpoint.events.process', type: 'index_template' },
     { id: 'logs-endpoint.events.registry', type: 'index_template' },
     { id: 'logs-endpoint.events.security', type: 'index_template' } ],
  es_index_patterns:
   { alerts: 'logs-endpoint.alerts-*',
     file: 'logs-endpoint.events.file-*',
     library: 'logs-endpoint.events.library-*',
     metadata: 'metrics-endpoint.metadata-*',
     metadata_current: 'metrics-endpoint.metadata_current-*',
     metrics: 'metrics-endpoint.metrics-*',
     network: 'logs-endpoint.events.network-*',
     policy: 'metrics-endpoint.policy-*',
     process: 'logs-endpoint.events.process-*',
     registry: 'logs-endpoint.events.registry-*',
     security: 'logs-endpoint.events.security-*' },
  name: 'endpoint',
  version: '0.16.0-dev.0',
  internal: false,
  removable: false,
  install_version: '0.16.0-dev.0',
  install_status: 'installing',
  install_started_at: '2020-09-16T00:45:47.538Z' }
installPackage installedPkg { id: 'endpoint',
  type: 'epm-packages',
  namespaces: [],
  updated_at: '2020-09-16T00:45:52.578Z',
  version: 'WzEyNCwxXQ==',
  attributes:
   { installed_kibana: [],
     installed_es:
      [ [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object],
        [Object] ],
     es_index_patterns:
      { alerts: 'logs-endpoint.alerts-*',
        file: 'logs-endpoint.events.file-*',
        library: 'logs-endpoint.events.library-*',
        metadata: 'metrics-endpoint.metadata-*',
        metadata_current: 'metrics-endpoint.metadata_current-*',
        metrics: 'metrics-endpoint.metrics-*',
        network: 'logs-endpoint.events.network-*',
        policy: 'metrics-endpoint.policy-*',
        process: 'logs-endpoint.events.process-*',
        registry: 'logs-endpoint.events.registry-*',
        security: 'logs-endpoint.events.security-*' },
     name: 'endpoint',
     version: '0.16.0-dev.0',
     internal: false,
     removable: false,
     install_version: '0.16.0-dev.0',
     install_status: 'installing',
     install_started_at: '2020-09-16T00:45:47.538Z' },
  references: [],
  migrationVersion: undefined }
installPackage endpoint-0.16.0-dev.0 update endpoint saved object
installPackage endpoint save new kibana refs before installing the assets
installPackage endpoint saveKibanaAssetsRefs
installPackage endpoint installKibanaAssets
saveInstalledEsRefs endpoint given [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' } ]
saveInstalledEsRefs endpoint set installed_es to [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
    type: 'transform' },
  { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.alerts', type: 'index_template' },
  { id: 'logs-endpoint.events.file', type: 'index_template' },
  { id: 'logs-endpoint.events.library', type: 'index_template' },
  { id: 'metrics-endpoint.metadata', type: 'index_template' },
  { id: 'metrics-endpoint.metadata_current',
    type: 'index_template' },
  { id: 'metrics-endpoint.metrics', type: 'index_template' },
  { id: 'logs-endpoint.events.network', type: 'index_template' },
  { id: 'metrics-endpoint.policy', type: 'index_template' },
  { id: 'logs-endpoint.events.process', type: 'index_template' },
  { id: 'logs-endpoint.events.registry', type: 'index_template' },
  { id: 'logs-endpoint.events.security', type: 'index_template' },
  { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' } ]
saveInstalledEsRefs endpoint given [ { id: 'logs-endpoint.alerts', type: 'index_template' },
  { id: 'logs-endpoint.events.file', type: 'index_template' },
  { id: 'logs-endpoint.events.library', type: 'index_template' },
  { id: 'metrics-endpoint.metadata', type: 'index_template' },
  { id: 'metrics-endpoint.metadata_current',
    type: 'index_template' },
  { id: 'metrics-endpoint.metrics', type: 'index_template' },
  { id: 'logs-endpoint.events.network', type: 'index_template' },
  { id: 'metrics-endpoint.policy', type: 'index_template' },
  { id: 'logs-endpoint.events.process', type: 'index_template' },
  { id: 'logs-endpoint.events.registry', type: 'index_template' },
  { id: 'logs-endpoint.events.security', type: 'index_template' } ]
saveInstalledEsRefs endpoint set installed_es to [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
    type: 'transform' },
  { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.alerts', type: 'index_template' },
  { id: 'logs-endpoint.events.file', type: 'index_template' },
  { id: 'logs-endpoint.events.library', type: 'index_template' },
  { id: 'metrics-endpoint.metadata', type: 'index_template' },
  { id: 'metrics-endpoint.metadata_current',
    type: 'index_template' },
  { id: 'metrics-endpoint.metrics', type: 'index_template' },
  { id: 'logs-endpoint.events.network', type: 'index_template' },
  { id: 'metrics-endpoint.policy', type: 'index_template' },
  { id: 'logs-endpoint.events.process', type: 'index_template' },
  { id: 'logs-endpoint.events.registry', type: 'index_template' },
  { id: 'logs-endpoint.events.security', type: 'index_template' } ]
installTransformForDataset endpoint
installTransformForDataset endpoint await deleteTransforms [ 'metrics-endpoint.metadata-current-default-0.16.0-dev.0' ]
deleteTransforms [ 'metrics-endpoint.metadata-current-default-0.16.0-dev.0' ]
stopTransforms [ 'metrics-endpoint.metadata-current-default-0.16.0-dev.0' ]
POST _stop metrics-endpoint.metadata-current-default-0.16.0-dev.0
awaitIfPending ok. result: undefined
awaitIfPending onResolve(result)
awaitIfPending isPending=false false
awaitIfPending end. return a promise which will resolve/reject Promise {
  <rejected> { Error: Bad Request
      at respond (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
      at checkRespForFailure (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
      at HttpConnector.<anonymous> (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
      at IncomingMessage.wrapper (/Users/jfsiii/work/kibana/node_modules/lodash/lodash.js:4949:19)
      at IncomingMessage.emit (events.js:203:15)
      at endReadableNT (_stream_readable.js:1145:12)
      at process._tickCallback (internal/process/next_tick.js:63:19)
    status: 400,
    displayName: 'BadRequest',
    message: 'Bad Request',
    path:
     '_transform/metrics-endpoint.metadata-current-default-0.16.0-dev.0/_stop',
    query: 'force=true',
    body: '400 Bad Request',
    statusCode: 400,
    response: '400 Bad Request',
    toString: [Function],
    toJSON: [Function] } }
server    log   [09:44:39.611] [error][ingestManager][plugins] { Error: Bad Request
    at respond (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/Users/jfsiii/work/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/Users/jfsiii/work/kibana/node_modules/lodash/lodash.js:4949:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  displayName: 'BadRequest',
  message: 'Bad Request',
  path:
   '_transform/metrics-endpoint.metadata-current-default-0.16.0-dev.0/_stop',
  query: 'force=true',
  body: '400 Bad Request',
  statusCode: 400,
  response: '400 Bad Request',
  toString: [Function],
  toJSON: [Function] }
server    log   [09:44:39.612] [error][ingestManager][plugins] Bad Request response from _transform/metrics-endpoint.metadata-current-default-0.16.0-dev.0/_stop: 400 Bad Request

local Kibana / local ES ✅

From scratch (no ES data): Kibana [2]

setupIngestManager called
awaitIfPending called
awaitIfPending try
createSetupSideEffects called
createSetupSideEffects call ensureDefault*, etc in parallel. wait for result
ensureInstalledPackage system undefined
ensureInstalledPackage system installLatestPackage
ensureInstalledPackage endpoint undefined
ensureInstalledPackage endpoint installLatestPackage
installPackage endpoint-0.16.0-dev.0
installPackage system-0.5.4
installPackage installedPkg undefined
installPackage installedPkg undefined
installPackage endpoint-0.16.0-dev.0 createInstallation { pkgName: 'endpoint',
  pkgVersion: '0.16.0-dev.0',
  internal: false,
  removable: false,
  installed_kibana: [],
  installed_es: [],
  toSaveESIndexPatterns:
   { alerts: 'logs-endpoint.alerts-*',
     file: 'logs-endpoint.events.file-*',
     library: 'logs-endpoint.events.library-*',
     metadata: 'metrics-endpoint.metadata-*',
     metadata_current: 'metrics-endpoint.metadata_current-*',
     metrics: 'metrics-endpoint.metrics-*',
     network: 'logs-endpoint.events.network-*',
     policy: 'metrics-endpoint.policy-*',
     process: 'logs-endpoint.events.process-*',
     registry: 'logs-endpoint.events.registry-*',
     security: 'logs-endpoint.events.security-*' } }
installPackage system-0.5.4 createInstallation { pkgName: 'system',
  pkgVersion: '0.5.4',
  internal: false,
  removable: false,
  installed_kibana: [],
  installed_es: [],
  toSaveESIndexPatterns:
   { auth: 'logs-system.auth-*',
     core: 'metrics-system.core-*',
     cpu: 'metrics-system.cpu-*',
     diskio: 'metrics-system.diskio-*',
     entropy: 'metrics-system.entropy-*',
     filesystem: 'metrics-system.filesystem-*',
     fsstat: 'metrics-system.fsstat-*',
     load: 'metrics-system.load-*',
     memory: 'metrics-system.memory-*',
     network: 'metrics-system.network-*',
     network_summary: 'metrics-system.network_summary-*',
     process: 'metrics-system.process-*',
     process_summary: 'metrics-system.process_summary-*',
     raid: 'metrics-system.raid-*',
     service: 'metrics-system.service-*',
     socket: 'metrics-system.socket-*',
     socket_summary: 'metrics-system.socket_summary-*',
     syslog: 'logs-system.syslog-*',
     uptime: 'metrics-system.uptime-*',
     users: 'metrics-system.users-*' } }
installPackage endpoint save new kibana refs before installing the assets
installPackage endpoint saveKibanaAssetsRefs
installPackage system save new kibana refs before installing the assets
installPackage system saveKibanaAssetsRefs
installPackage endpoint installKibanaAssets
saveInstalledEsRefs endpoint given [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' } ]
installPackage system installKibanaAssets
saveInstalledEsRefs system given [ { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' } ]
saveInstalledEsRefs endpoint set installed_es to [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' } ]
saveInstalledEsRefs system set installed_es to [ { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' } ]
saveInstalledEsRefs endpoint given [ { id: 'logs-endpoint.alerts', type: 'index_template' },
  { id: 'logs-endpoint.events.file', type: 'index_template' },
  { id: 'logs-endpoint.events.library', type: 'index_template' },
  { id: 'metrics-endpoint.metadata', type: 'index_template' },
  { id: 'metrics-endpoint.metadata_current',
    type: 'index_template' },
  { id: 'metrics-endpoint.metrics', type: 'index_template' },
  { id: 'logs-endpoint.events.network', type: 'index_template' },
  { id: 'metrics-endpoint.policy', type: 'index_template' },
  { id: 'logs-endpoint.events.process', type: 'index_template' },
  { id: 'logs-endpoint.events.registry', type: 'index_template' },
  { id: 'logs-endpoint.events.security', type: 'index_template' } ]
saveInstalledEsRefs endpoint set installed_es to [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.alerts', type: 'index_template' },
  { id: 'logs-endpoint.events.file', type: 'index_template' },
  { id: 'logs-endpoint.events.library', type: 'index_template' },
  { id: 'metrics-endpoint.metadata', type: 'index_template' },
  { id: 'metrics-endpoint.metadata_current',
    type: 'index_template' },
  { id: 'metrics-endpoint.metrics', type: 'index_template' },
  { id: 'logs-endpoint.events.network', type: 'index_template' },
  { id: 'metrics-endpoint.policy', type: 'index_template' },
  { id: 'logs-endpoint.events.process', type: 'index_template' },
  { id: 'logs-endpoint.events.registry', type: 'index_template' },
  { id: 'logs-endpoint.events.security', type: 'index_template' } ]
saveInstalledEsRefs system given [ { id: 'logs-system.auth', type: 'index_template' },
  { id: 'metrics-system.core', type: 'index_template' },
  { id: 'metrics-system.cpu', type: 'index_template' },
  { id: 'metrics-system.diskio', type: 'index_template' },
  { id: 'metrics-system.entropy', type: 'index_template' },
  { id: 'metrics-system.filesystem', type: 'index_template' },
  { id: 'metrics-system.fsstat', type: 'index_template' },
  { id: 'metrics-system.load', type: 'index_template' },
  { id: 'metrics-system.memory', type: 'index_template' },
  { id: 'metrics-system.network', type: 'index_template' },
  { id: 'metrics-system.network_summary', type: 'index_template' },
  { id: 'metrics-system.process', type: 'index_template' },
  { id: 'metrics-system.process_summary', type: 'index_template' },
  { id: 'metrics-system.raid', type: 'index_template' },
  { id: 'metrics-system.service', type: 'index_template' },
  { id: 'metrics-system.socket', type: 'index_template' },
  { id: 'metrics-system.socket_summary', type: 'index_template' },
  { id: 'logs-system.syslog', type: 'index_template' },
  { id: 'metrics-system.uptime', type: 'index_template' },
  { id: 'metrics-system.users', type: 'index_template' } ]
saveInstalledEsRefs system set installed_es to [ { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
  { id: 'logs-system.auth', type: 'index_template' },
  { id: 'metrics-system.core', type: 'index_template' },
  { id: 'metrics-system.cpu', type: 'index_template' },
  { id: 'metrics-system.diskio', type: 'index_template' },
  { id: 'metrics-system.entropy', type: 'index_template' },
  { id: 'metrics-system.filesystem', type: 'index_template' },
  { id: 'metrics-system.fsstat', type: 'index_template' },
  { id: 'metrics-system.load', type: 'index_template' },
  { id: 'metrics-system.memory', type: 'index_template' },
  { id: 'metrics-system.network', type: 'index_template' },
  { id: 'metrics-system.network_summary', type: 'index_template' },
  { id: 'metrics-system.process', type: 'index_template' },
  { id: 'metrics-system.process_summary', type: 'index_template' },
  { id: 'metrics-system.raid', type: 'index_template' },
  { id: 'metrics-system.service', type: 'index_template' },
  { id: 'metrics-system.socket', type: 'index_template' },
  { id: 'metrics-system.socket_summary', type: 'index_template' },
  { id: 'logs-system.syslog', type: 'index_template' },
  { id: 'metrics-system.uptime', type: 'index_template' },
  { id: 'metrics-system.users', type: 'index_template' } ]
installTransformForDataset endpoint
installTransformForDataset endpoint await deleteTransforms []
deleteTransforms []
saveInstalledEsRefs endpoint given [ { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
    type: 'transform' } ]
saveInstalledEsRefs endpoint set installed_es to [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.file-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.library-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.network-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'metrics-endpoint.policy-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.process-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.events.security-0.16.0-dev.0',
    type: 'ingest_pipeline' },
  { id: 'logs-endpoint.alerts', type: 'index_template' },
  { id: 'logs-endpoint.events.file', type: 'index_template' },
  { id: 'logs-endpoint.events.library', type: 'index_template' },
  { id: 'metrics-endpoint.metadata', type: 'index_template' },
  { id: 'metrics-endpoint.metadata_current',
    type: 'index_template' },
  { id: 'metrics-endpoint.metrics', type: 'index_template' },
  { id: 'logs-endpoint.events.network', type: 'index_template' },
  { id: 'metrics-endpoint.policy', type: 'index_template' },
  { id: 'logs-endpoint.events.process', type: 'index_template' },
  { id: 'logs-endpoint.events.registry', type: 'index_template' },
  { id: 'logs-endpoint.events.security', type: 'index_template' },
  { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
    type: 'transform' } ]
installTransformForDataset system
installTransformForDataset system await deleteTransforms []
deleteTransforms []
leaving installPackage system installedTransforms []
leaving installPackage endpoint installedTransforms [ { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
    type: 'transform' } ]
createSetupSideEffects ensureDefault*, etc worked
system === system? true
createSetupSideEffects system should be installed? true
createSetupSideEffects system is installed? false
createSetupSideEffects system addPackageToAgentPolicy
ensureInstalledPackage system { installed_kibana:
   [ { id: 'system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'dashboard' },
     { id: 'system-277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs',
       type: 'dashboard' },
     { id: 'system-5517a150-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'dashboard' },
     { id: 'system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs',
       type: 'dashboard' },
     { id: 'system-CPU-slash-Memory-per-container-ecs',
       type: 'dashboard' },
     { id: 'system-Filebeat-syslog-dashboard-ecs',
       type: 'dashboard' },
     { id: 'system-Metrics-system-overview-ecs', type: 'dashboard' },
     { id: 'system-c431f410-f9ac-11e9-90e8-1fb18e796788',
       type: 'dashboard' },
     { id: '62439dc0-f9c9-11e6-a747-6121780e0414-ecs',
       type: 'search' },
     { id: '8030c1b0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'search' },
     { id: 'Syslog-system-logs-ecs', type: 'search' },
     { id: 'b6f321e0-fa25-11e6-bbd3-29c986c96e5a-ecs',
       type: 'search' },
     { id: 'eb0039f0-fa7f-11e6-a1df-a78bd7504d38-ecs',
       type: 'search' },
     { id: '089b85d0-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '12667040-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs',
       type: 'visualization' },
     { id: '1aae9140-1b93-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '2e224660-1b19-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '327417e0-8462-11e7-bab8-bd2f0fb42c54-ecs',
       type: 'visualization' },
     { id: '341ffe70-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '346bb290-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '3cec3eb0-f9d3-11e6-8a3e-2b904044ea1d-ecs',
       type: 'visualization' },
     { id: '3d65d450-a9c3-11e7-af20-67db8aecb295-ecs',
       type: 'visualization' },
     { id: '4b254630-f998-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '4d546850-1b15-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '51164310-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs',
       type: 'visualization' },
     { id: '5c7af030-fa2a-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '5dd15c00-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: '6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs',
       type: 'visualization' },
     { id: '78b74f30-f9cd-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: '825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs',
       type: 'visualization' },
     { id: '83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '855899e0-1b1c-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '8c071e20-f999-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '96976150-4d5d-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: '99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: '9c69cad0-f9b0-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'Container-Block-IO-ecs', type: 'visualization' },
     { id: 'Container-CPU-usage-ecs', type: 'visualization' },
     { id: 'Container-Memory-stats-ecs', type: 'visualization' },
     { id: 'Syslog-events-by-hostname-ecs', type: 'visualization' },
     { id: 'Syslog-hostnames-and-processes-ecs',
       type: 'visualization' },
     { id: 'System-Navigation-ecs', type: 'visualization' },
     { id: 'a30871f0-f98f-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'bb3a8720-f991-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: 'c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: 'd16bb400-f9cc-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: 'd2e80340-4d5c-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: 'd3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: 'd3f51850-f9b6-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'd56ee420-fa79-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'dc589770-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: 'e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'e121b140-fa78-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'e6e639e0-f992-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'f398d2f0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: 'fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' } ],
  installed_es:
   [ { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.auth', type: 'index_template' },
     { id: 'metrics-system.core', type: 'index_template' },
     { id: 'metrics-system.cpu', type: 'index_template' },
     { id: 'metrics-system.diskio', type: 'index_template' },
     { id: 'metrics-system.entropy', type: 'index_template' },
     { id: 'metrics-system.filesystem', type: 'index_template' },
     { id: 'metrics-system.fsstat', type: 'index_template' },
     { id: 'metrics-system.load', type: 'index_template' },
     { id: 'metrics-system.memory', type: 'index_template' },
     { id: 'metrics-system.network', type: 'index_template' },
     { id: 'metrics-system.network_summary', type: 'index_template' },
     { id: 'metrics-system.process', type: 'index_template' },
     { id: 'metrics-system.process_summary', type: 'index_template' },
     { id: 'metrics-system.raid', type: 'index_template' },
     { id: 'metrics-system.service', type: 'index_template' },
     { id: 'metrics-system.socket', type: 'index_template' },
     { id: 'metrics-system.socket_summary', type: 'index_template' },
     { id: 'logs-system.syslog', type: 'index_template' },
     { id: 'metrics-system.uptime', type: 'index_template' },
     { id: 'metrics-system.users', type: 'index_template' } ],
  es_index_patterns:
   { auth: 'logs-system.auth-*',
     core: 'metrics-system.core-*',
     cpu: 'metrics-system.cpu-*',
     diskio: 'metrics-system.diskio-*',
     entropy: 'metrics-system.entropy-*',
     filesystem: 'metrics-system.filesystem-*',
     fsstat: 'metrics-system.fsstat-*',
     load: 'metrics-system.load-*',
     memory: 'metrics-system.memory-*',
     network: 'metrics-system.network-*',
     network_summary: 'metrics-system.network_summary-*',
     process: 'metrics-system.process-*',
     process_summary: 'metrics-system.process_summary-*',
     raid: 'metrics-system.raid-*',
     service: 'metrics-system.service-*',
     socket: 'metrics-system.socket-*',
     socket_summary: 'metrics-system.socket_summary-*',
     syslog: 'logs-system.syslog-*',
     uptime: 'metrics-system.uptime-*',
     users: 'metrics-system.users-*' },
  name: 'system',
  version: '0.5.4',
  internal: false,
  removable: false,
  install_version: '0.5.4',
  install_status: 'installed',
  install_started_at: '2020-09-16T13:31:44.601Z' }
endpoint === system? false
createSetupSideEffects endpoint should be installed? false
createSetupSideEffects return { isIntialized: true }
awaitIfPending ok. result: { isIntialized: true }
awaitIfPending onResolve(result)
awaitIfPending isPending=false false
awaitIfPending end. return a promise which will resolve/reject Promise { { isIntialized: true } }

From scratch (no ES data): ES [3]

   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata_current-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata_current] for index patterns [metrics-endpoint.metadata_current-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.entropy] for index patterns [metrics-system.entropy-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.network_summary] for index patterns [metrics-system.network_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.raid] for index patterns [metrics-system.raid-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.service] for index patterns [metrics-system.service-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket] for index patterns [metrics-system.socket-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.users] for index patterns [metrics-system.users-*]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [.transform-internal-005] creating index, cause [auto(bulk api)], templates [.transform-internal-005], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [JFSIII.local] updating number_of_replicas to [0] for indices [.transform-internal-005]
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [.transform-notifications-000002] creating index, cause [auto(bulk api)], templates [.transform-notifications-000002], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [JFSIII.local] updating number_of_replicas to [0] for indices [.transform-notifications-000002]
   │ info [o.e.x.t.t.p.SchemaUtil] [JFSIII.local] Failed to deduce mapping for [agent.id], fall back to keyword.
   │ info [o.e.c.m.MetadataCreateIndexService] [JFSIII.local] [metrics-endpoint.metadata_current-default] creating index, cause [api], templates [metrics-endpoint.metadata_current], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [JFSIII.local] updating number_of_replicas to [0] for indices [metrics-endpoint.metadata_current-default]
   │ info [o.e.x.i.IndexLifecycleTransition] [JFSIII.local] moving index [metrics-endpoint.metadata_current-default] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [metrics]
   │ info [o.e.x.i.IndexLifecycleTransition] [JFSIII.local] moving index [metrics-endpoint.metadata_current-default] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] in policy [metrics]
   │ info [o.e.x.t.t.TransformTask] [JFSIII.local] [metrics-endpoint.metadata-current-default-0.16.0-dev.0] updating state for transform to [{"task_state":"started","indexer_state":"stopped","checkpoint":0,"should_stop_at_checkpoint":false}].
   │ info [o.e.x.t.t.TransformPersistentTasksExecutor] [JFSIII.local] [metrics-endpoint.metadata-current-default-0.16.0-dev.0] successfully completed and scheduled task in node operation
   │ info [o.e.x.i.IndexLifecycleTransition] [JFSIII.local] moving index [metrics-endpoint.metadata_current-default] from [{"phase":"hot","action":"unfollow","name":"wait-for-indexing-complete"}] to [{"phase":"hot","action":"unfollow","name":"wait-for-follow-shard-tasks"}] in policy [metrics]
   │ info [o.e.c.m.MetadataMappingService] [JFSIII.local] [.kibana_1/wYZsvH_rSSWfIaC7EbXHaA] update_mapping [_doc]

Restart Kibana (ES remains up w/ packages installed): [4]

setupIngestManager called
awaitIfPending called
awaitIfPending try
createSetupSideEffects called
createSetupSideEffects call ensureDefault*, etc in parallel. wait for result
ensureInstalledPackage system { installed_kibana:
   [ { id: 'system-0d3f2380-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'dashboard' },
     { id: 'system-277876d0-fa2c-11e6-bbd3-29c986c96e5a-ecs',
       type: 'dashboard' },
     { id: 'system-5517a150-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'dashboard' },
     { id: 'system-79ffd6e0-faa0-11e6-947f-177f697178b8-ecs',
       type: 'dashboard' },
     { id: 'system-CPU-slash-Memory-per-container-ecs',
       type: 'dashboard' },
     { id: 'system-Filebeat-syslog-dashboard-ecs',
       type: 'dashboard' },
     { id: 'system-Metrics-system-overview-ecs', type: 'dashboard' },
     { id: 'system-c431f410-f9ac-11e9-90e8-1fb18e796788',
       type: 'dashboard' },
     { id: '62439dc0-f9c9-11e6-a747-6121780e0414-ecs',
       type: 'search' },
     { id: '8030c1b0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'search' },
     { id: 'Syslog-system-logs-ecs', type: 'search' },
     { id: 'b6f321e0-fa25-11e6-bbd3-29c986c96e5a-ecs',
       type: 'search' },
     { id: 'eb0039f0-fa7f-11e6-a1df-a78bd7504d38-ecs',
       type: 'search' },
     { id: '089b85d0-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '12667040-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '19e123b0-4d5a-11e7-aee5-fdc812cc3bec-ecs',
       type: 'visualization' },
     { id: '1aae9140-1b93-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '26732e20-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '2e224660-1b19-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '327417e0-8462-11e7-bab8-bd2f0fb42c54-ecs',
       type: 'visualization' },
     { id: '341ffe70-f9ce-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '346bb290-fa80-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: '34f97ee0-1b96-11e7-8ada-3df93aab833e-ecs',
       type: 'visualization' },
     { id: '3cec3eb0-f9d3-11e6-8a3e-2b904044ea1d-ecs',
       type: 'visualization' },
     { id: '3d65d450-a9c3-11e7-af20-67db8aecb295-ecs',
       type: 'visualization' },
     { id: '4b254630-f998-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '4d546850-1b15-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '4e4bb1e0-1b1b-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '51164310-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '522ee670-1b92-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '590a60f0-5d87-11e7-8884-1bb4c3b890e4-ecs',
       type: 'visualization' },
     { id: '5c7af030-fa2a-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: '5dd15c00-fa78-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: '6b7b9a40-faa1-11e6-86b1-cd7735ff7e23-ecs',
       type: 'visualization' },
     { id: '78b74f30-f9cd-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: '7cdb1330-4d1a-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: '825fdb80-4d1d-11e7-b5f2-2b7c1895bf32-ecs',
       type: 'visualization' },
     { id: '83e12df0-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: '855899e0-1b1c-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: '8c071e20-f999-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: '96976150-4d5d-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: '99381c80-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: '9c69cad0-f9b0-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'Container-Block-IO-ecs', type: 'visualization' },
     { id: 'Container-CPU-usage-ecs', type: 'visualization' },
     { id: 'Container-Memory-stats-ecs', type: 'visualization' },
     { id: 'Syslog-events-by-hostname-ecs', type: 'visualization' },
     { id: 'Syslog-hostnames-and-processes-ecs',
       type: 'visualization' },
     { id: 'System-Navigation-ecs', type: 'visualization' },
     { id: 'a30871f0-f98f-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'ab2d1e90-1b1a-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'bb3a8720-f991-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'bfa5e400-1b16-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'c5e3cf90-4d60-11e7-9a4c-ed99bbcaa42b-ecs',
       type: 'visualization' },
     { id: 'c6f2ffd0-4d17-11e7-a196-69b9a7a020a9-ecs',
       type: 'visualization' },
     { id: 'd16bb400-f9cc-11e6-8115-a7c18106d86a-ecs',
       type: 'visualization' },
     { id: 'd2e80340-4d5c-11e7-aa29-87a97a796de6-ecs',
       type: 'visualization' },
     { id: 'd3166e80-1b91-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' },
     { id: 'd3f51850-f9b6-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'd56ee420-fa79-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'dc589770-fa2b-11e6-bbd3-29c986c96e5a-ecs',
       type: 'visualization' },
     { id: 'e0f001c0-1b18-11e7-b09e-037021c4f8df-ecs',
       type: 'visualization' },
     { id: 'e121b140-fa78-11e6-a1df-a78bd7504d38-ecs',
       type: 'visualization' },
     { id: 'e6e639e0-f992-11e9-90e8-1fb18e796788',
       type: 'visualization' },
     { id: 'f398d2f0-fa77-11e6-ae9b-81e5311e8cab-ecs',
       type: 'visualization' },
     { id: 'fe064790-1b1f-11e7-bec4-a5e9ec5cab8b-ecs',
       type: 'visualization' } ],
  installed_es:
   [ { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.auth-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.syslog-0.5.4', type: 'ingest_pipeline' },
     { id: 'logs-system.auth', type: 'index_template' },
     { id: 'metrics-system.core', type: 'index_template' },
     { id: 'metrics-system.cpu', type: 'index_template' },
     { id: 'metrics-system.diskio', type: 'index_template' },
     { id: 'metrics-system.entropy', type: 'index_template' },
     { id: 'metrics-system.filesystem', type: 'index_template' },
     { id: 'metrics-system.fsstat', type: 'index_template' },
     { id: 'metrics-system.load', type: 'index_template' },
     { id: 'metrics-system.memory', type: 'index_template' },
     { id: 'metrics-system.network', type: 'index_template' },
     { id: 'metrics-system.network_summary', type: 'index_template' },
     { id: 'metrics-system.process', type: 'index_template' },
     { id: 'metrics-system.process_summary', type: 'index_template' },
     { id: 'metrics-system.raid', type: 'index_template' },
     { id: 'metrics-system.service', type: 'index_template' },
     { id: 'metrics-system.socket', type: 'index_template' },
     { id: 'metrics-system.socket_summary', type: 'index_template' },
     { id: 'logs-system.syslog', type: 'index_template' },
     { id: 'metrics-system.uptime', type: 'index_template' },
     { id: 'metrics-system.users', type: 'index_template' } ],
  es_index_patterns:
   { auth: 'logs-system.auth-*',
     core: 'metrics-system.core-*',
     cpu: 'metrics-system.cpu-*',
     diskio: 'metrics-system.diskio-*',
     entropy: 'metrics-system.entropy-*',
     filesystem: 'metrics-system.filesystem-*',
     fsstat: 'metrics-system.fsstat-*',
     load: 'metrics-system.load-*',
     memory: 'metrics-system.memory-*',
     network: 'metrics-system.network-*',
     network_summary: 'metrics-system.network_summary-*',
     process: 'metrics-system.process-*',
     process_summary: 'metrics-system.process_summary-*',
     raid: 'metrics-system.raid-*',
     service: 'metrics-system.service-*',
     socket: 'metrics-system.socket-*',
     socket_summary: 'metrics-system.socket_summary-*',
     syslog: 'logs-system.syslog-*',
     uptime: 'metrics-system.uptime-*',
     users: 'metrics-system.users-*' },
  name: 'system',
  version: '0.5.4',
  internal: false,
  removable: false,
  install_version: '0.5.4',
  install_status: 'installed',
  install_started_at: '2020-09-16T13:31:44.601Z' }
ensureInstalledPackage endpoint { installed_kibana: [],
  installed_es:
   [ { id: 'logs-endpoint.alerts-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.file-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.library-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metadata-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.metrics-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.network-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'metrics-endpoint.policy-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.process-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.registry-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.events.security-0.16.0-dev.0',
       type: 'ingest_pipeline' },
     { id: 'logs-endpoint.alerts', type: 'index_template' },
     { id: 'logs-endpoint.events.file', type: 'index_template' },
     { id: 'logs-endpoint.events.library', type: 'index_template' },
     { id: 'metrics-endpoint.metadata', type: 'index_template' },
     { id: 'metrics-endpoint.metadata_current',
       type: 'index_template' },
     { id: 'metrics-endpoint.metrics', type: 'index_template' },
     { id: 'logs-endpoint.events.network', type: 'index_template' },
     { id: 'metrics-endpoint.policy', type: 'index_template' },
     { id: 'logs-endpoint.events.process', type: 'index_template' },
     { id: 'logs-endpoint.events.registry', type: 'index_template' },
     { id: 'logs-endpoint.events.security', type: 'index_template' },
     { id: 'metrics-endpoint.metadata-current-default-0.16.0-dev.0',
       type: 'transform' } ],
  es_index_patterns:
   { alerts: 'logs-endpoint.alerts-*',
     file: 'logs-endpoint.events.file-*',
     library: 'logs-endpoint.events.library-*',
     metadata: 'metrics-endpoint.metadata-*',
     metadata_current: 'metrics-endpoint.metadata_current-*',
     metrics: 'metrics-endpoint.metrics-*',
     network: 'logs-endpoint.events.network-*',
     policy: 'metrics-endpoint.policy-*',
     process: 'logs-endpoint.events.process-*',
     registry: 'logs-endpoint.events.registry-*',
     security: 'logs-endpoint.events.security-*' },
  name: 'endpoint',
  version: '0.16.0-dev.0',
  internal: false,
  removable: false,
  install_version: '0.16.0-dev.0',
  install_status: 'installed',
  install_started_at: '2020-09-16T13:31:44.573Z' }
createSetupSideEffects ensureDefault*, etc worked
system === system? true
createSetupSideEffects system should be installed? true
createSetupSideEffects system is installed? true
endpoint === system? false
createSetupSideEffects endpoint should be installed? false
createSetupSideEffects return { isIntialized: true }
awaitIfPending ok. result: { isIntialized: true }
awaitIfPending onResolve(result)
awaitIfPending isPending=false false
awaitIfPending end. return a promise which will resolve/reject Promise { { isIntialized: true } }

cc @neptunian & @nnamdifrankie in the hopes these give more insight into what's happening

  1. kibana_remote_initial_error_log.txt
  2. kibana_local_initial_success_log.txt
  3. es_local_initial_success_log.txt
  4. kibana_local_second_cold_start_success_log.txt
jfsiii picture jfsiii on 16 Sep 2020

Update: We have two changes pending which should fix the error on load.

1) https://github.com/elastic/endpoint-package/pull/80 will remove the file that we're struggling to handle. It needs to be pushed to the Registry but Cloud and others will pick it up soon after it's available there.

2) https://github.com/elastic/kibana/pull/77640#issuecomment-693711219 fixes the cause of the 400 which seems to be a missing leading slash. It might not be available for Thursday's snapshot but (1) above will be. Worst case this is available Friday and we can try restoring the file from (1)

jfsiii picture jfsiii on 17 Sep 2020

An additional option to get testing going is to run the following curl command after your 8.0 snapshot build completes and you get your Kibana URL. Do the below BEFORE going to the Ingest page.

  • Get the username <user>, it's usually elastic
  • Get your password <pass>, the long string that's randomly generated
  • Get your Kibana URL <kibana-url>, you can either "Copy Endpoint" on the cloud screen or just copy the URL including the port from your browser's address bar.

Open a terminal, fill in the above, run this command. After that Ingest should load.

curl -X POST -H 'kbn-xsrf: xxx' --user <user>:<pass> \
 -H "Content-Type: application/json" -d '{"force" : true}' \
 <kibana-url>/api/ingest_manager/epm/packages/endpoint-0.15.0

The above is only necessary until we get another 8.0 SNAPSHOT build. The root issue should be fixed there.

kevinlog picture kevinlog on 17 Sep 2020
❤2 👍2

I just tried tried this on a few deployments, and it seems that it only works on an instance which has never tried to setup before.

That makes sense now but just confirming (for myself as much as anyone).

Existing deployment - /setup had previously failed.

endpoint 0.16 was installed

curl --user elastic:somepass 'https://1aef124d8bc9424ea7f6e2ca049ca7e0.us-central1.gcp.foundit.no:9243/api/saved_objects/_find?type=epm-packages&search=endpoint&fields=version&fields=name' | jq .
{
  "page": 1,
  "per_page": 20,
  "total": 1,
  "saved_objects": [
    {
      "type": "epm-packages",
      "id": "endpoint",
      "attributes": {
        "name": "endpoint",
        "version": "0.16.0-dev.0"
      },
      "references": [],
      "updated_at": "2020-09-17T14:26:13.894Z",
      "version": "WzMzNywxXQ==",
      "score": 0.6931471
    }
  ]
}

POST 0.15.0 failed

curl --user elastic:somepass -H 'kbn-xsrf: xxx' -H "Content-Type: application/json" -d '{"force" : true}' -X POST https://1aef124d8bc9424ea7f6e2ca049ca7e0.us-central1.gcp.foundit.no:9243/api/ingest_manager/epm/packages/endpoint-0.15.0
{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "Bad Request"
}

Existing deployment - never started

no endpoint installed

curl --user elastic:somepass 'https://aef1d915fdf74db9ade2688b9045e4a9.us-east-1.aws.staging.foundit.no:9243/api/saved_objects/_find?type=epm-packages&search=endpointz&fields=version&fields=name' | jq .
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}

POST 0.15 succeeds

 curl --user elastic:somepass 'https://aef1d915fdf74db9ade2688b9045e4a9.us-east-1.aws.staging.foundit.no:9243/api/saved_objects/_find?type=epm-packages&search=endpoint&fields=version&fields=name' | jq .
{
  "page": 1,
  "per_page": 20,
  "total": 1,
  "saved_objects": [
    {
      "type": "epm-packages",
      "id": "endpoint",
      "attributes": {
        "name": "endpoint",
        "version": "0.15.0"
      },
      "references": [],
      "updated_at": "2020-09-17T14:37:47.645Z",
      "version": "WzIxLDFd",
      "score": 0.6931471
    }
  ]
}

endpoint 0.15 installed

curl --user elastic:somepass 'https://03ebbf213bb44c5d827f28afb4a0e7a3.us-central1.gcp.foundit.no:9243/api/saved_objects/_find?type=epm-packages' | jq .
{
  "page": 1,
  "per_page": 20,
  "total": 1,
  "saved_objects": [
    {
      "type": "epm-packages",
      "id": "endpoint",
      "attributes": {
        "installed_kibana": [],
        "installed_es": [
          {
            "id": "logs-endpoint.alerts-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.file-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.library-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "metrics-endpoint.metadata-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "metrics-endpoint.metrics-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.network-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "metrics-endpoint.policy-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.process-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.registry-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.security-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.alerts",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.file",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.library",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.metadata",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.metadata_current",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.metrics",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.network",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.policy",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.process",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.registry",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.security",
            "type": "index_template"
          }
        ],
        "es_index_patterns": {
          "alerts": "logs-endpoint.alerts-*",
          "file": "logs-endpoint.events.file-*",
          "library": "logs-endpoint.events.library-*",
          "metadata": "metrics-endpoint.metadata-*",
          "metadata_current": "metrics-endpoint.metadata_current-*",
          "metrics": "metrics-endpoint.metrics-*",
          "network": "logs-endpoint.events.network-*",
          "policy": "metrics-endpoint.policy-*",
          "process": "logs-endpoint.events.process-*",
          "registry": "logs-endpoint.events.registry-*",
          "security": "logs-endpoint.events.security-*"
        },
        "name": "endpoint",
        "version": "0.15.0",
        "internal": false,
        "removable": false,
        "install_version": "0.15.0",
        "install_status": "installed",
        "install_started_at": "2020-09-17T14:58:05.433Z"
      },
      "references": [],
      "updated_at": "2020-09-17T14:58:12.593Z",
      "version": "WzE2LDFd",
      "score": 0
    }
  ]
}

New deployment - call API before visiting

no packages installed

 curl --user elastic:somepass 'https://03ebbf213bb44c5d827f28afb4a0e7a3.us-central1.gcp.foundit.no:9243/api/saved_objects/_find?type=epm-packages' | jq .
{
  "page": 1,
  "per_page": 20,
  "total": 0,
  "saved_objects": []
}

POST 0.15 succeeds

curl --user elastic:somepass -H 'kbn-xsrf: xxx' -H "Content-Type: application/json" -d '{"force" : true}' -X POST https://03ebbf213bb44c5d827f28afb4a0e7a3.us-central1.gcp.foundit.no:9243/api/ingest_manager/epm/packages/endpoint-0.15.0
{
  "response": [
    {
      "id": "logs-endpoint.alerts-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.events.file-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.events.library-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "metrics-endpoint.metadata-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "metrics-endpoint.metrics-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.events.network-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "metrics-endpoint.policy-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.events.process-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.events.registry-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.events.security-0.15.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-endpoint.alerts",
      "type": "index_template"
    },
    {
      "id": "logs-endpoint.events.file",
      "type": "index_template"
    },
    {
      "id": "logs-endpoint.events.library",
      "type": "index_template"
    },
    {
      "id": "metrics-endpoint.metadata",
      "type": "index_template"
    },
    {
      "id": "metrics-endpoint.metadata_current",
      "type": "index_template"
    },
    {
      "id": "metrics-endpoint.metrics",
      "type": "index_template"
    },
    {
      "id": "logs-endpoint.events.network",
      "type": "index_template"
    },
    {
      "id": "metrics-endpoint.policy",
      "type": "index_template"
    },
    {
      "id": "logs-endpoint.events.process",
      "type": "index_template"
    },
    {
      "id": "logs-endpoint.events.registry",
      "type": "index_template"
    },
    {
      "id": "logs-endpoint.events.security",
      "type": "index_template"
    }
  ]
}

endpoint 0.15 installed

curl --user elastic:somepass 'https://03ebbf213bb44c5d827f28afb4a0e7a3.us-central1.gcp.foundit.no:9243/api/saved_objects/_find?type=epm-packages' | jq .
{
  "page": 1,
  "per_page": 20,
  "total": 1,
  "saved_objects": [
    {
      "type": "epm-packages",
      "id": "endpoint",
      "attributes": {
        "installed_kibana": [],
        "installed_es": [
          {
            "id": "logs-endpoint.alerts-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.file-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.library-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "metrics-endpoint.metadata-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "metrics-endpoint.metrics-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.network-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "metrics-endpoint.policy-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.process-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.registry-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.events.security-0.15.0",
            "type": "ingest_pipeline"
          },
          {
            "id": "logs-endpoint.alerts",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.file",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.library",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.metadata",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.metadata_current",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.metrics",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.network",
            "type": "index_template"
          },
          {
            "id": "metrics-endpoint.policy",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.process",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.registry",
            "type": "index_template"
          },
          {
            "id": "logs-endpoint.events.security",
            "type": "index_template"
          }
        ],
        "es_index_patterns": {
          "alerts": "logs-endpoint.alerts-*",
          "file": "logs-endpoint.events.file-*",
          "library": "logs-endpoint.events.library-*",
          "metadata": "metrics-endpoint.metadata-*",
          "metadata_current": "metrics-endpoint.metadata_current-*",
          "metrics": "metrics-endpoint.metrics-*",
          "network": "logs-endpoint.events.network-*",
          "policy": "metrics-endpoint.policy-*",
          "process": "logs-endpoint.events.process-*",
          "registry": "logs-endpoint.events.registry-*",
          "security": "logs-endpoint.events.security-*"
        },
        "name": "endpoint",
        "version": "0.15.0",
        "internal": false,
        "removable": false,
        "install_version": "0.15.0",
        "install_status": "installed",
        "install_started_at": "2020-09-17T14:58:05.433Z"
      },
      "references": [],
      "updated_at": "2020-09-17T14:58:12.593Z",
      "version": "WzE2LDFd",
      "score": 0
    }
  ]
}

jfsiii picture jfsiii on 17 Sep 2020

@jfsiii

I just tried tried this on a few deployments, and it seems that it only works on an instance which has never tried to setup before.

Yes like I mentioned in our chat. This is because of how Ingest Manager currently handles downgrades where it leave the index templates even when you remove the package. I believe there is some marked difference between 0.15.0 to 0.16.0, which I think has to do with changes to datastream that prevents the downgrade after setup. However once we fix the build we should not be looking back after setup.

nnamdifrankie picture nnamdifrankie on 17 Sep 2020

@rahulgupta-qasource

Summary: we believe this has been fixed by https://github.com/elastic/kibana/pull/77640. There are some unrelated issues on staging at the moment which require a workaround to deploy.

The Kibana 8.x SNAPSHOT builds are failing on the default 1GB memory. However, if you increase that to 2GB, they will behave as expected

Select 8.x

Screen Shot 2020-09-18 at 9 43 00 AM

Click Customize

Screen Shot 2020-09-18 at 9 43 12 AM

Increase to 2GB

Screen Shot 2020-09-18 at 9 43 31 AM

jfsiii picture jfsiii on 18 Sep 2020

I concur - I see this fixed in latest 8.0 snapshot. shall we close it out?

EricDavisX picture EricDavisX on 18 Sep 2020

Hi @jfsiii / @EricDavisX

Thank you for sharing the feedback.

On increasing default 1GB memory to 2GB under 'Kibana 1 configuration', user is successfully able to deploy 8.0.0-SNAPSHOT Kibana cloud environment on staging cloud platform and no error is displayed on clicking left hand menu 'Ingest Manager' link (Fixed).

Screenshot:

  1. Successful deployment on increasing memory to 2GB
    image
  1. No error is displayed on clicking left hand menu 'Ingest Manager' link
    #77133_closed_2

Hence, we are closing this bug.

rahulgupta-qasource picture rahulgupta-qasource on 21 Sep 2020

Bug Conversion:

Below 01 Testcase already exists for this ticket:
https://elastic.testrail.io/index.php?/cases/view/15250

rahulgupta-qasource picture rahulgupta-qasource on 25 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

treussart picture treussart  ·  3Comments
timmolter picture timmolter  ·  3Comments
ynux picture ynux  ·  3Comments
stacey-gammon picture stacey-gammon  ·  3Comments
cafuego picture cafuego  ·  3Comments