Kibana: [DocViewer/Table] empty date fields are listed

[spalger]

Kibana version: 7.8 - master

Elasticsearch version: 7.8 - master

Original install method (e.g. download page, yum, from source, etc.): source

Describe the bug:

If you have multiple date fields in your index (metricbeats created dozens of them) but documents use only a handful of those date fields at a time, Discover renders all of the date fields even when they are not found in the _source of the document.

Steps to reproduce:

1. Index two test documents in console:

POST /test/_doc
{
  "date1": "2020-06-18T01:02:03Z",
  "date2": "2020-06-18T04:05:06Z"
}
POST /test/_doc
{
  "date1": "2020-06-18T01:02:03Z"
}

1. Create a test index pattern
2. Open up discover and see the extra date field is rendered when it shouldn't be

Expected behavior:

Discover should only show fields that are relevant to a specific document.

Screenshots (if relevant):

[bvader]

Just to add context for filebeat-* and metricbeat-* many, perhaps 10-20 or more empty fields are displayed

[timroes]

@kertal I marked this as a blocker for 7.8.1

[kertal]

Elasticsearch in 7.8 seems to have changed the docvalue_fields of search result. So the following request with the data @spalger provided returns:

Request in 7.8:

POST /test/_search
{
  "docvalue_fields": [
    {
      "field": "date1",
      "format": "date_time"
    },
    {
      "field": "date2",
      "format": "date_time"
    }
  ]
}

Hits in response in 7.8

 [
      {
        "_index" : "test",
        "_id" : "tofp3XIBeEof-q9jHaWL",
        "_score" : 1.0,
        "_source" : {
          "date1" : "2020-06-18T01:02:03Z",
          "date2" : "2020-06-18T04:05:06Z"
        },
        "fields" : {
          "date2" : [
            "2020-06-18T04:05:06.000Z"
          ],
          "date1" : [
            "2020-06-18T01:02:03.000Z"
          ]
        }
      },
      {
        "_index" : "test",
        "_id" : "wYfp3XIBeEof-q9jOqWi",
        "_score" : 1.0,
        "_source" : {
          "date1" : "2020-06-18T01:02:03Z"
        },
        "fields" : {
          "date2" : [ ],
          "date1" : [
            "2020-06-18T01:02:03.000Z"
          ]
        }
      }
    ]

Hits in 7.7:

[{
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "nCPu3XIB3tjGouxM1KJr",
        "_score" : 1.0,
        "_source" : {
          "date1" : "2020-06-18T01:02:03Z",
          "date2" : "2020-06-18T04:05:06Z"
        },
        "fields" : {
          "date1" : [
            "2020-06-18T01:02:03.000Z"
          ],
          "date2" : [
            "2020-06-18T04:05:06.000Z"
          ]
        }
      },
      {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "niPu3XIB3tjGouxM46I3",
        "_score" : 1.0,
        "_source" : {
          "date1" : "2020-06-18T01:02:03Z"
        },
        "fields" : {
          "date1" : [
            "2020-06-18T01:02:03.000Z"
          ]
        }
      }]

So 7.7 returns also empty fields. If this is an intended change from ES side, we need to change the way we prepare the data. Because currently we display what ES returns

[kertal]

This is a regression in ES, and the PR is already in the works to fix it, will link it here once available

[kertal]

Should by fixed by https://github.com/elastic/elasticsearch/pull/58418

[spalger]

Confirmed this is fixed in 7.8+

Thanks all!