Kibana: [EPM] install/update package side effects in order

[neptunian]

Installing and updating logic is going to be similar due to the fact that even when installing a package for the first time, there could be data coming (eg. the user is already using the agent Standalone)
If any one thing fails in the installation or update, it (and all side effects) should not happen.

Order of install / update

1. Check saved objects to see if the package is installed

• Created/update package saved object that installation has started and to what version in case of failure in upgrade process

2. Delete Kibana saved objects assets

• delete all kibana assets if this is an update to start fresh in case some assets were removed

3. Install Pipeline, ILM, Kibana Index Patterns, Kibana Assets

Pipeline

• A new pipeline with a unique name (type-datasetName-version) is created and sent to ES
• This must be created and exist before the index template gets updated which references the new pipeline in the settings

ILM Policy

• Currently on the base package has a global ILM policy that we never plan to update, but at some point packages could have ILMs
• Do not update the ILM policy (what about a package level ILM policy?)
• During an install, check if the ILM already exists, install if so, continue otherwise

Kibana Index Patterns

Kibana Assets (visualizations, maps, dashboards, etc)

• all assets should be removed first and then saved again, in the event that an upgrade removes an asset. are assets ever shared between packages?

4. Index template

• Generate/Update a new index template per dataset with updated settings (referencing the new pipeline) and mappings and send to ES
• This happens before updating the current index’s mappings and settings to account for the edgecase that a rollover happens before the index is updated (next step)

5. Update the current write index’s mappings and settings

• The current write index's mappings and settings need to be updated instead of triggering a rollover ( this is inefficient because it could potentially create a lot of indices and we want to avoid that) if possible.
• Update the mappings. It is possible at this point that the data comes in when the mappings has been updated but settings has not yet been. We assume if the mappings update was successful then the "old pipeline" being referenced in the not yet updated settings will also be compatible and that this is OK.
• If the mappings update is successful, update the settings. The settings may reference a new pipeline PUT in a previous step
• If the PUT mapping is incompatible (except for supported mapping parameters, you can’t change the mapping or field type of an existing field), it will return an error. If the new mappings are incompatible with this index we need to trigger a rollover.
  
  
  
  • The new index created from the rollover will be created with the correct mappings and settings from the already updated index template in step 5
  
  

6. delete the old pipeline

• old pipelines must be deleted after the the template has been updated

7. save package to saved objects

• if successful, update the saved object to new version and updating installing status to installed

Updating the Agent Configuration

• have spoken with @ruflin about this and the thinking is we will not update the configuration automatically (which references package versions) after updating a package. Instead will inform them some way through the UI to see if they want to update it.

Edgecases

• If we decide to turn off dynamic mapping, there could possibly be negative side effects in step 3. If the mapping did not exist previously, and then it was updated and added, but the settings has not yet been updated to reference the new pipeline which may need to do something with that new field, during that window of time, some data could be incorrect
• in step 3, it's possible the current write index has no documents. Attempting to update mappings that are incompatible on an index that has no documents will still fail. In this scenario we'd have an empty index hanging around.

https://github.com/elastic/kibana/issues/56359

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[ruflin]

@neptunian Could you share a bit of background on why the order is as proposed above?

[jen-huang]

If all of the above is successful save to saved objects PACKAGES_SAVED_OBJECT_TYPE

Will this be a new saved object, or an update to the saved object of the previous version?

we will not update the configuration automatically (which references package versions) after updating a package

++ I agree with this approach. On the agent config UI side, we should surface package version that is being used by the agent config.

[neptunian]

@ruflin In step 3, I want to confirm in the scenario where we update the mappings and data comes in before we update the settings, that we are not concerned that new mappings could have been added and that data is not being handled (but is supposed to be), by the new ingest pipeline.

For triggering the rollover in the event that mappings is incompatible, how is this meant to work currently when we only have one index and are not using an index alias?

[ruflin]

As the mapping and ingest pipeline are always updated before any agent config, it should not be possible that new data comes in before we updated all of it?

For triggering the rollover: I guess it will not work in alpha1 then :-( We will need data streams for that. Lets assume we don't have breaking mapping changes during alpha1. But at the same time, we should already build all the code and testing for it to make sure it will work.

[neptunian]

This was completed for alpha and is more of a meta issue to document how installation works

[ph]

@neptunian @ruflin What is the goal of that issue, should we keep it open, move it into our documentation?

[ruflin]

++ moving this into docs and for the outstanding open new issues.