Kibana: Advanced setting to show fields regardless of conflict type

[alexfrancoeur]

In 5.5 we no longer show field conflicts for "like" types and allow you to select a field for a visualization that resolve to the same Kibana type. This is best described in this comment https://github.com/elastic/kibana/pull/13070#issuecomment-317829553

Today, Kibana will not allow you to select a field for a visualization if there is any kind of conflict. While the adjustments made in 5.5 makes this experience a bit better, there are still scenarios in which field conflicts occur for dissimilar types. For example a field who had text type and the mapping changed to integer.

We should provide an outlet for users to override the assumptions Kibana makes about field types. If there is a conflict, a user should be able to toggle an advanced setting to ignore these conflicts and allow a user to select this field regardless. If the visualization attempted to aggregate on a non-supported or conflicted type, Elasticsearch should throw an error and surface this in Kibana.

[alexfrancoeur]

cc: @epixa @ppf2 @spalger @chrisronline

[chrisronline]

There might be a possible workaround that involves using scripted fields.

Consider the following dataset:

PUT my-data-2017-11-13
{
  "mappings": {
    "doc": {
      "properties": {
        "name": {
          "type": "text"
        },
        "time": {
          "type": "date"
        },
        "count": {
          "type": "keyword"
        }
      }
    }
  }
}
PUT my-data-2017-11-14
{
  "mappings": {
    "doc": {
      "properties": {
        "name": {
          "type": "text"
        },
        "time": {
          "type": "date"
        },
        "count": {
          "type": "integer"
        }
      }
    }
  }
}
PUT my-data-2017-11-15
{
  "mappings": {
    "doc": {
      "properties": {
        "name": {
          "type": "text"
        },
        "time": {
          "type": "date"
        },
        "count": {
          "type": "integer"
        }
      }
    }
  }
}

POST my-data-2017-11-13/doc
{
  "name": "Michael Scott",
  "time": "2017-11-13T14:00:48.365Z",
  "count": "2"
}

POST my-data-2017-11-14/doc
{
  "name": "Jim Halpert",
  "time": "2017-11-14T14:00:48.365Z",
  "count": 3
}

POST my-data-2017-11-15/doc
{
  "name": "Pam Beesley",
  "time": "2017-11-15T14:00:48.365Z",
  "count": 4
}

If I created an index pattern my-data-*, I'd see a mapping conflict for count. However, I can create a scripted field using:

try {
  return Integer.parseInt(doc['count'].value);
}
catch (Exception e) {
  return doc['count'].value;
}

I can now use that scripted field to build visualizations, dashboards, etc.

[alexfrancoeur]

Thanks @chrisronline. While this is certainly possible, it's definitely not ideal. The purpose of this issue is to have the option natively in Kibana. For anyone looking to solve this issue today, @chrisronline's comment above is a workaround today

[epixa]

To be clear, what @chrisronline posted and what we're proposing to add to Kibana are apples and oranges. The proposal in Kibana is to give people the ability to opt out of assumptions Kibana makes based on field types and just let people try to do whatever they want to any field, which is a considerably worse experience than seamlessly handling the conflicts in the query itself, which is what the scripted field is doing.

[ppf2]

If there are documents without the field (or multiple indices in the index pattern where one or more indices do not contain the field), the workaround script will fail at the ES level, so I have added a pre-check to it:

if (doc.containsKey('count')) {
try {
  return Integer.parseInt(doc['count'].value);
}
catch (Exception e) {
  return doc['count'].value;
}
}

[alexfrancoeur]

@chrisronline @spalger @epixa is this still something we plan to address in a 6.x release? I remember some early brainstorm conversations around this and have heard similar requests from some of our larger customers. Think we can revisit?

[epixa]

@alexfrancoeur I think it's probably too early to tell.

[ppf2]

This is another example of a type conflict Kibana does not handle today (concrete vs. object field): https://github.com/elastic/kibana/issues/26583