Kibana: Clicking "Visualize" from Discover field pane results in java.lang.IllegalStateException error

[tbragin]

I get this in any indexed field in a time-based index, such as the logstash-* index from the "Getting Started" tutorial. Doesn't happen on the "shakespeare" and "bank" indices which are not time-based. Haven't tested whether this is present in earlier versions.

Video:

Error:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on machine.os"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2015.05.18","node":"2zGW0gguQlC3OIWKSVER_A","reason":{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on machine.os","caused_by":{"type":"unchecked_execution_exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on machine.os","caused_by":{"type":"illegal_state_exception","reason":"Field data loading is forbidden on machine.os"}}}}]}}
    at http://localhost:5601/bundles/kibana.bundle.js?v=9682:78996:39
    at Function.Promise.try (http://localhost:5601/bundles/commons.bundle.js?v=9682:61111:23)
    at http://localhost:5601/bundles/commons.bundle.js?v=9682:61080:31
    at Array.map (native)
    at Function.Promise.map (http://localhost:5601/bundles/commons.bundle.js?v=9682:61079:31)
    at callResponseHandlers (http://localhost:5601/bundles/kibana.bundle.js?v=9682:78968:23)
    at http://localhost:5601/bundles/kibana.bundle.js?v=9682:78475:17
    at processQueue (http://localhost:5601/bundles/commons.bundle.js?v=9682:42357:29)
    at http://localhost:5601/bundles/commons.bundle.js?v=9682:42373:28
    at Scope.$eval (http://localhost:5601/bundles/commons.bundle.js?v=9682:43601:29)

[spalger]

I think this is actually expected behavior. Kibana currently tries to guess when something can/can't be aggregated on, but this new criteria is not a part of the equation. #3335 was supposed to expand the definition in a different way, and this new criteria (fieldata.format = disabled) is starting to pop up a lot. When discussing how we would handle #3335 we decided that https://github.com/elastic/kibana/pull/5806 was the right way to go. We should rely on elasticsearch to tell us when something is/isn't aggregatable (via structured exceptions).

This specific failure is caused by logstash disabling field data for its analyzed fields. In order to aggregate on this field you must use the .raw version.

[spalger]

That said I'm not a fan of this user experience, and I'm seriously wondering if we should try harder to define "aggregatable" and help the user earlier rather than after sending the agg request to elasticsearch.

[rashidkpc]

I personally don't think we should define aggregatable, we can't keep up with elasticsearch changing what that means. Elasticsearch needs a better way of telling us if something can be aggregated on. Perhaps we could ask for the validate API to be extended such that we could fire a pre-flight _validate request to determine if the request would succeed, and determine if we should show the button or not?

[tbragin]

The problem is that for many operational use cases, what you will see in discover are these analyzed, and not raw fields. So they will be non-aggregatable and user will get this exception when clicking on Visualize. Can we automatically select the .raw version of the field upon clicking this button, if one exists?

[rmoff]

Can we automatically select the .raw version of the field upon clicking this button, if one exists?

This would certainly help for those coming to Kibana from a non-Elasticsearch/Lucene background. It'd be useful to extend it in general to failed visualisations attempted against a field when it should be the .raw version used.

[hkarpf]

Agree with rmoff, would be nice to automatically select the .raw when trying to visualize.

[benmadin]

I'm no expert here (although I can get this error pretty reliably!), but at the very least if the error message could suggest using the .raw version, it would save some sadness...

[spalger]

I agree @benmadin. The error message displayed currently is coming directly from elasticsearch. We should do more to handle this specific error message better though.

[skearns64]

FWIW, starting in Elasticsearch 5.0, when it auto-detect a string in field foo, ES creates a text field called foo (analyzed string, not aggregate-able), and a keyword field called foo.keyword (not-analyzed, great for aggregations).

Now that this is a built-in feature of ES, rather than a convention used by Logstash templates, I would imagine that this issue/feature is a bit more important, as it will impact a lot more users.

This may be a simplistic/short-sighted way of thinking of it </disclaimer> :) , but that "visualize" button on the Discover page could do a check to see if field is a text field, and field.keyword exists, use field.keyword when redirecting to the visualization page. Same logic could control the presence of the little warning icon on the visualize button.

[Bargs]

This should be trivial to fix now that we have searchable and aggregatable flags for all fields. https://github.com/elastic/kibana/pull/8421