Elasticsearch: Default action.auto_create_index to false?

[jasontedor]

Today we default action.auto_create_index to true. There is potential for this to be a dangerous default. In production, it means that indices might not be created with the appropriate number of primary shards and replicas, and that analyzers and mappings might not be set up before any data is indexed. It means that a wide range of PUT and POST requests against the cluster create indices (I've seen vulnerability scanners accidentally create too many indices, hurting the cluster). The question put forth here is: should we change the default to false?

Note: this issue is not a proposal, this issue is being opened for discussion to solicit feedback from the community.

[jpountz]

For the record, there is a discussion at #20640 about moving index auto creation control to templates.

[jasontedor]

Indeed, where there is a proposal to ship with a default template matching * that has action.auto_create_index set to true. If there is agreement here to default action.auto_create_index to false, I think that the response to #20640 would be to not ship with a default template matching * with action.auto_create_index set to true.

[clintongormley]

If we were to change the default to false, the outcomes would be:

• You have to explicitly create an index (even an empty index) before indexing a document
• Or you have to have a template which matches the index

The downside of removing auto-create during development is small: you just run PUT test before continuing.

In production, you almost never want a completely unmapped index to spring into existence (and if you did want that, putting an empty template which matches * would resolve that).

I'm leaning towards changing the default to false.

[javanna]

Does this imply that we would still do the index creation automatically but only in case there is a matching template? So the index creation step will not be mandatory at all times?

[clintongormley]

Does this imply that we would still do the index creation automatically but only in case there is a matching template? So the index creation step will not be mandatory at all times?

Correct - if there is a matching template, the index can be created automatically (although we have spoken about setting in the template which allows this behaviour to be disabled).

Regarding the above para: why would you have a template if you don't want indices to be created automatically? One possible reason: the rollover API. Any others?

[javanna]

why would you have a template if you don't want indices to be created automatically?

Templates and indices may be created/managed by different people? Don't we want to still apply templates to indices that are manually created (although what is in the request has the precedence)?

[clintongormley]

fair enough

[neuroticnetworks]

How would this work with the rollover API?

Another possibility (sake of argument, no idea if this makes sense or not): What if this were handled much like the whitelisting of backup locations? You could whitelist patterns you approve of auto-creating and disapprove of everything else.

Not really sure that would buy very much though.....

[jasontedor]

@evanv This is already possible (specifying a pattern) and solves the rollover problem too.

[dakrone]

It appears that a consensus was reached on this issue, does this need further discussion @jasontedor or can it be marked "adopt me"?

[elasticmachine]

Pinging @elastic/es-core-infra

[tmenier]

Any updates on this? I understand changing the default would be breaking, but one way to avoid that _and_ put a little more control in the hands of developers would be to allow specifying/overriding the behavior directly in calls to the Index API. In my case I control the code that builds and queries indexes but not necessarily the cluster deployments, and I can't trust that action.auto_create_index is always false. So I do an extra call to be sure, and cache the result (to avoid lots of extra calls), which works fine, but it would sure be a lot more convenient if the index call would just fail and tell me when the index doesn't exist.