Kibana: Build a Better Search Experience

[Bargs]

Motivation

Kibana currently provides four different search mechanisms with overlapping responsibilities:

1. Lucene query syntax in the query bar
2. Query DSL in the query bar
3. Filters created via the UI (which could include custom query DSL if edited)
4. Console

Exposing the lucene query syntax and the query DSL to users creates a few problems. Since we don't control the query syntax we can't implement features that would require introspection into a user's query. This includes things like:

1. Safe and seamless migrations of saved searches when ES search APIs change
2. Typeahead/autocomplete in the query bar
3. Dynamic help text

We could solve these problems by building a model in Kibana to represent raw Elasticsearch queries, but there are other advantages to building our own query language:

1. We can support query types that are available in the ES query DSL that are not supported by the lucene query syntax
2. We can implement functionality that is beyond the scope of the lucene query syntax, e.g. support for aggregations and visualizations in the query language
3. We can provide finer grain controls for admins to restrict access to expensive queries, e.g. leading wildcards or regexes
4. We can add support for scripted fields to the language
5. We can unify the query bar and the filter bar, eliminating confusion about when to use one or the other

How

This is a meta ticket that we'll break down into smaller tasks. At a high level, here's what we're thinking:

• [x] Create a feature toggle to support multiple query languages https://github.com/elastic/kibana/pull/12624
• [x] Develop new kibana query language as experimental feature https://github.com/elastic/kibana/issues/12282 https://github.com/elastic/kibana/pull/12624
• [ ] Build a better query editor https://github.com/elastic/kibana/issues/13621
• [ ] Consider adding support for filter features (pin/unpin, activate/deactivate) to the query language, possibly with something like VisualSearch.js. https://github.com/elastic/kibana/issues/13564
• [ ] Add inline help
• [ ] Remove support for lucene query syntax and the filter bar, filtering via the UI will update the query (think Github filters) when new query experience is enabled

[rashidkpc]

Here's a 5 minute-ish video of my first experiment into a language that unifies search, real-time field extraction, aggregations and multi-phase querying in Kibana

https://www.youtube.com/watch?v=L5LvP_Cj0A0#t=32m28s

The repo for it is here, though I haven't updated it to work in the latest version of Kibana: https://github.com/rashidkpc/kable

The wildly simple grammar for the language is here: https://github.com/rashidkpc/kable/blob/master/server/lib/kable.peg

Hope it helps, or at least gives you some ideas of what not todo :-)

[rashidkpc]

Also, sometime when I started working on that it was a pipe-like syntax. Here's the grammar for that, back in the first commit: https://github.com/rashidkpc/kable/blob/76e6ff5539de9cf5d9979543e47cf351d260aae0/server/lib/kable.peg

[cjcenizal]

Kibana currently provides three different search mechanisms with overlapping responsibilities:

1. Lucene query syntax in the query bar
2. Query DSL in the query bar
3. Filters created via the UI (which could include custom query DSL if edited)

There's also sort of a 4th option, which is ES queries built in Console (https://discuss.elastic.co/t/how-can-i-use-dev-tools-query-in-kibana-discover-query-bar-is-it-possible/79752/2). We might also want to consider tools that make it easier to translate ES queries into the query language.

[Bargs]

@cjcenizal that will be difficult because the query language will be a subset of the query DSL, so a Console crafted query might not have a valid query string representation. Having some sort of console integration in the query bar is an interesting idea though.

[rashidkpc]

Oh, btw, https://github.com/rashidkpc/kable does work on the latest kibana now. Though watch the video if you want to play with it.

[trevan]

@Bargs, is there any documentation on what this new language is going to look like?

[Bargs]

@trevan nope, ideas are still just rattling around in my head. We don't have any specs written out (I'd post them here if we did). If you have any ideas I'd love to hear them.

[darkmoon03]

I'll second the request for multi-phase queries. (do a query, take the result set and do this to it)

Also sub queries and client side joins. ^^

[Bargs]

This issue was a good starting point but it's a bit out of date now that we've shipped the new query language and started iterating on it. We're keeping track of enhancement ideas in https://github.com/elastic/kibana/issues/13650