Kibana: Kibana's system indices

[kobelb]

Legacy multi-tenancy

Kibana currently supports "legacy multi-tenancy" by allowing users to configure the Elasticsearch indices that should be used to persist Kibana specific entities. It's rather common to hear developers refer to the ".kibana" index, but this is rather imprecise as there are a number of dot-indices that are used, not just ".kibana":

name	kibana.yml setting	default indices	default alias	category
saved-objects	kibana.index	.kibana_${integer}	.kibana	system
reporting	xpack.reporting.index	.reporting-${date}	N/A	system
task manager	xpack.task_manager.index	.kibana_task_manager_${integer}	.kibana_task_manager	system
security session	derived from kibana.index	.kibana_security_session_${integer}	N/A	system
event log	derived from kibana.index	.kibana-event-log-${version}-${integer}	.kibana-event-log-${version}	hidden

There are also some indices that Kibana reads from and writes to, but they aren't configurable in the kibana.yml, so they aren't changed when users are implementing legacy multi-tenancy.

name	default indices	default alias	category
APM agent configuration	.apm-agent-configuration	N/A	system
APM custom link	.apm-custom-link	N/A	system
monitoring	.monitoring-*		hidden
beats central management	.management-beats		deprecated

Starting in 8.0, this method of multi-tenancy will no longer be supported. Users will no longer be able to specify their kibana.index, xpack.reporting.index and xpack.task_manager.index settings in their kibana.yml. For more details, please see https://github.com/elastic/kibana/issues/82020

Upgrade Paths

According to our documentation, the following upgrade paths are supported with a rolling upgrade:

• Between minor versions
• From 6.last to 7.x
• From 7.last to 8.x

And the following upgrade paths are supported with a full cluster restart:

• From 6.not-last to 7.x
• From 7.not-last to 8.x

When upgrading Kibana, users are instructed to first upgrade Elasticsearch and then shut down all instances of Kibana prior to starting up a new version. If users experience an issue with their minor version upgrade, they will commonly roll-back to the previous version. I don't believe this is documented as supported, but I know a number of users do this and Cloud does this as well.

Kibana's system-index patterns

The following patterns will be registered as Kibana's system-indices. As such, end-users will be unable to access them using the standard ES document APIs starting in 8.0. If end-users access them during 7.x, a deprecation will be logged.

• .kibana_*
• .reporting-*
• .apm-*

ES API changes

When Kibana makes API calls to Elasticsearch, all API calls are prefixed with _kibana/. Besides that, these are standard API calls. Some example API calls:

## Create index
PUT _kibana/.kibana_1
{
  "mappings": {
    "properties": {}
  }
}

## Clone index
PUT _kibana/.kibana_1/_clone/.kibana_2

## Create alias referencing index
PUT _kibana/.kibana_1/_alias/.kibana

## CRUD and search documents using the index-name
POST _kibana/.kibana_1/_doc/
{
   "type" : "visualization",
   "visualization": {

 }
}

POST _kibana/_bulk
{ "index" : { "_index" : ".kibana_1", "_id" : "1" } }

POST _kibana/.kibana_1/_search
{
   "query": {
   "bool" : {
     "must" : {
       "term" : { "type" : "visualization" }
     }
   }
 } 
}

## CRUD and search documents using the alias
POST _kibana/.kibana/_doc/
{
   "type" : "visualization",
   "visualization": {

 }
}

POST _kibana/_bulk
{ "index" : { "_index" : ".kibana", "_id" : "1" } }

POST _kibana/.kibana/_search
{
   "query": {
   "bool" : {
     "must" : {
       "term" : { "type" : "visualization" }
     }
   }
 } 
}

7.x

During 7.x, since users will still be able to specify kibana.index, xpack.reporting.index and xpack.task_manager.index settings in their kibana.yml. When these settings have been changed from their defaults, Kibana should not communicate with them using the _kibana prefixed paths. However, when these settings have not been changed, Kibana will communicate with them using the Kibana specific endpoints which are prefixed by _kibana.

Once Kibana has transitioned to accessing these indices using the _kibana endpoints, Elasticsearch will log a deprecation warning whenever Kibana's system indices are accessed using standard ES APIs.

8.0

Users will no longer be able to specify kibana.index, xpack.reporting.index orxpack.task_manager.index in the kibana.yml. If they do, they won't be able to start Kibana. Users will be instructed as part of the 8.0 major upgrade assistant to migrate to using Spaces for multi-tenancy.

Only Kibana's system indices will be able to be accessed via the _kibana prefixed endpoints. Additionally, Kibana's system indices will not be able to be accessed via the standard ES APIs.

Implementation

• [ ] Kibana's system-indices will no longer be able to be communicated with directly using the Elasticsearch client(s), as these endpoints are intended to be hidden from other users of Elasticsearch. There are potentially some tricks that we can play here by changing the "path" of Elasticsearch to include _kibana, but this will potentially lead to run-time errors if an unsupported API is called. https://github.com/elastic/kibana/issues/82716
• [ ] ESArchiver + System Indices == 🥰 or 😭 #83592
• [ ] Reporting system-index retention policy https://github.com/elastic/kibana/issues/81544
• [ ] Explicitly delete all Kibana security session https://github.com/elastic/kibana/issues/81941

[kobelb]

@jaymode is going to be investigating whether or not we have other options besides wrapping the existing APIs under a new endpoint in ES, so we should hold off on implementing the approach outlined in this issue for the time being.

/cc @elastic/kibana-core