Kibana: Field type conflict information missing from field editor

[jen-huang]

20245 unintentionally removed detailed information about field type conflict in 6.4. Need to update the field editor to restore this information:

Reported from this Discuss thread.

[weslambert]

Hi @jen-huang ,

Was wondering if there was any update on this, or if there was a somewhat quick and easy alternative to get this info, other than manually searching and comparing?

Thanks!
Wes

[remd]

I am also very interested in an update on this as this makes fixing field datatype conflicts even more difficult.

In the meantime, for others who rely on this tool for finding and fixing indices with mapping conflicts, this data can be retrieved from Kibana's REST API at /api/saved_objects/_bulk_get.

Example curl (TLS and Security are enabled on my cluster, so I must provide the -u option and specify https://, and I am using 6.4.2 at the time of this writing, so YMMV):

curl -XPOST -u myusername 'https://mykibanahost/api/saved_objects/_bulk_get' -H 
"Content-Type: application/json" -H "kbn-version: 6.4.2" -d @request_body.json

Example request_body.json:

[{"id": "my-index-*", "type": "index-pattern"}]

The response JSON will contain a nested JSON string which contains an array of fields across the indices matched by the index pattern. Fields that conflict across indices will be marked as type "conflict" and contain a "conflictDescriptions" key, whose value is an object containing the different types associated with the field and the indices in which each type is present. This information is roughly equivalent what used to be displayed on this page prior to this regression.

For example:

{
  "name": "host",
  "type": "conflict",
  "count": 4,
  "scripted": false,
  "searchable": false,
  "aggregatable": false,
  "readFromDocValues": false,
  "conflictDescriptions": {
    "text": [
      "my-index-monthly-2017.10",
      "my-index-monthly-2017.11",
      "my-index-monthly-2017.12",
      "my-index-monthly-2018.01",
      "my-index-monthly-2018.02",
      "my-index-monthly-2018.03"
    ],
    "keyword": [
      "my-index-monthly-2017.08",
      "my-index-monthly-2017.09",
      "my-index-monthly-2018.04",
      "my-index-monthly-2018.05",
      "my-index-monthly-2018.06",
      "my-index-monthly-2018.07"
    ]
  }
}

If you have jq available to parse the response you can retrieve just the fields with conflicts like so:

curl -XPOST -u myusername 'https://mykibanahost/api/saved_objects/_bulk_get' -H 
"Content-Type: application/json" -H "kbn-version: 6.4.2" -d @request_body.json |
 jq -r '.saved_objects[].attributes.fields' | jq '.[] | select(.type=="conflict")'

[vpavlushkov]

Hi @jen-huang,
Is there any update on this issue or at least some estimates when it could be addressed? I just checked that the latest stable Kibana 6.5.4 is still affected.
Thanks!

[Dumbaz]

Kibana 6.6.0 is also affected. Hopefully this can be fixed everywhere.

[drtobbe]

Kibana 6.6.1 is also affected. Hopefully this can be fixed everywhere.

[webmat]

Having this information displayed again would be very helpful for people performing a Beats upgrade from 6 to 7.

Context:

With Beats moving to ECS, they are moving hundreds of fields around. The Beats upgrade procedure uses ES field aliases to smooth out the transition.

However currently the Kibana index pattern view:

• shows the old field type + the alias field as a conflict
• no longer gives information about the field conflicts

The combination of these two make the upgrade process quite painful. As now users are potentially shown tens or hundreds of conflicts, with most of them being acceptable (aliases) and perhaps a few hidden gems in there that need to be understood/addressed (actual type changes).

[remd]

@webmat Are you stating that this issue also affects Kibana 7.x?

[timroes]

@remd This issue also effects 7.x so far. We're planning to fix this over the 7.x roadmap, but don't have a PR up yet, thus can't give any guarantee yet in which version that will be released.

[webmat]

Correction to my comment above: if the old field and new field (once the alias is resolved) are of the same datatype, they will not show as conflicted in Kibana.

They only show up as a conflict if there is an actual change in datatype.

[kertal]

Solution is on the way, so here's the first adaption, please check dear @elastic/kibana-design, using EuiFormRow(fullWidth) + EuiBasicTable

Note that the list of indices shown in the table columns will usually be much longer

So this just displays the conflicts, is there a way to help users fixing these problems?

[cchaos]

@kertal Just watch your usage of icon color vs text color. In fact, the text you have should probably just be a small EuiCallOut

[kertal]

@cchaos thanx, working on the PR