Elasticsearch: Inner hits _source is now always wrapped in path to parent

[Mpdreamz]

Prior to 6.x the _source of an inner hit was always rooted to the child object that you were referencing.

Given the following super simple document where children is nested

POST /sample/doc
{
  "children": [
    {
      "startTime": 37230000000,
      "endTime": 73840000000
    }
  ]
}

The inner hits for children used to look like this:

"_source" : {
  "startTime" : 37230000000,
  "endTime" : 73840000000
}

Now in 6.x the context of source has changed the root of the child. Note the additional children key. If it was N levels deep the actual source of the inner hit would be N levels deep too.

"_source" : {
  "children" : {
   "startTime" : 37230000000,
   "endTime" : 73840000000
  }
}

This changes a lot of code that assumes the current context of the inner hit is inner hit and not the root of the document.

Another side effect is that the returned inner hit _source json, that is now rooted to the original doc, can no longer be mapped back into the doc type because children no longer is an array.

So now _source neither maps back to the original doc nor child and always acts as if it was source filtered.

I spoke with @martijnvg and I realize this change relates to:

https://github.com/elastic/elasticsearch/issues/23090
https://github.com/elastic/elasticsearch/issues/16653

Which I am 100% behind, full paths should be specified always. I don't agree with the way these changes changed the way _source is presented back to user though.

cc @elastic/es-clients

[clintongormley]

Just chatted to @martijnvg about this. It does feel more practical to return the nested _source without the prefix. Either way, it's a breaking change (which already landed in 6.0.0-beta2 in https://github.com/elastic/elasticsearch/pull/26102). I really wouldn't want to break this again in 7.0. My inclination is to fix this in rc2.

/cc @mattweber