Elasticsearch: Deprecate extracting _routing and _id from document fields

[clintongormley]

Currently routing and ID values can be passed in the query string and url respectively, but they can also be extracted from fields within the document.

This has a significant performance impact because each doc needs to be parsed on the node which receives the index/get/update/delete request in order to know which node should handle the request.

On top of that, there are clashes between (eg) routing values set in fields and parent settings.

We should deprecate the functionality to extract these values from fields, and make it the responsibility of the client code instead.

It should still be possible to set _routing to required. Perhaps we should set this automatically if the user ever passes in a routing or parent value at index time?

Relates to #8870

[jpountz]

+1

[javanna]

+1

[clintongormley]

This change needs to be made in a backwards compatible way.

From 2.0:

• refuse to create new indices with the path setting in _routing or _id
• for bwc, maintain the ability to support these settings on old indices
• make the upgrade API complain about these settings (see #8682)

From 3.0:

• remove support completely

Users should still be able to set _routing.required as it is a useful safety check. Perhaps we should set this value automatically if the user indexes a document with a custom routing value?

[jpountz]

We should deprecate extracting the _timestamp field as well.

[clintongormley]

We should deprecate extracting the _timestamp field as well.

@jpountz I'm not sure about this, but I think it requires a separate discussion, so I've opened https://github.com/elasticsearch/elasticsearch/issues/9058

[clintongormley]

Also see #5558

[ostefano]

Can someone confirm me that this is why copy_to inside _id mappings does not currently work?

[rjernst]

This change was made only to master. As far as I can tell, copy_to was never supported for the _id field.

[celesteking]

I had to dig through the bloat over internet in order to get to this information. You should really DOCUMENT that there will be no substitution for path and that copy_to can't be used to "extract" _id.

[itsnavee]

is there an alternative to using your own ID to avoid duplicate record publishing

[jpountz]

@itsnavee No, there isn't. In the future please ask questions at discuss.elastic.co as we want to keep this repo about feature requests and bug reports.