Kibana: [GS] Advanced search syntax to fine tune results returned

[alexfrancoeur]

Assuming we end up adding more granular search results (https://github.com/elastic/kibana/issues/72680, https://github.com/elastic/kibana/issues/74283) we'll likely want a way to narrow down the search even further. Adding some sort of simple syntax will likely be necessary to fine tune navigation. Some examples below:

• metrics: aws-host2
• cases: malware and cases:open
• dashboards: soc

I think this will only be necessary as we introduce new result providers to navigational search. We'll also need to make sure we get the UX / syntax right.

Improvements / follow-ups:

• [ ] add the filter suggestion option https://github.com/elastic/kibana/issues/83888
• [ ] should we translate the search syntax? https://github.com/elastic/kibana/issues/83887
• [ ] add type alias support in type filter (canvas-workpad -> canvas / workpad)https://github.com/elastic/kibana/issues/83889

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-core-ui (Team:Core UI)

[pgayvallet]

Functionally, this makes sense imho. technically, we will need a definitive syntax before starting to implement a parser. Depending on the requested syntax features, this may be non-trivial.

[joshdover]

Depending on the requested syntax features, this may be non-trivial.

I vouch for using PEG.js to generate a parser/lexer from a formal grammar spec. We have prior art here in the expressions / interpreter code which uses the same library for parsing canvas expressions.

[ryankeairns]

We've been doing some work to line up all the GS requests thus far in anticipation of prioritizing work beyond 7.10. We'll obviously need your input as we have those discussions. Related, I've started a GH project to track ongoing work for this feature: https://github.com/elastic/kibana/projects/65

[pgayvallet]

the syntax

So, after thinking a little about it, maybe we should be using EUI's Query syntax here. The two arguments I got are:

• This is the only search syntax widely used on Kibana at the moment. Users would not have to learn a 'new' syntax (For example selecting a type to filter on in the SO management section updates the search field using this Query syntax)

• The lexer already exists. we would 'only' have to implement the parser and to use the parsed query from within the result providers (mostly the SO provider)

Note that this syntax is slightly different than the one suggested

• cases: malware and cases:open would be cases:(malware and open) (or, eventually cases:malware cases:open)
• dashboards: soc would be dashboards:soc (no space allowed between the field/value and the : separator)

the features / filters we can provide

FYI, The query documentation is available here: https://elastic.github.io/eui/#/forms/search-bar.

Technically, for the SO result provider, we have two options:

(The easy one) using the search, type, references searchField options of the find API.

This limits our filtering options to:

• search only for specific types (using the same search term) type:(dashboard OR visualization) my term or dashboard:"my term" (but NOT dashboard:myterm OR visualization:another-term)
• search only for objects assigned to the given tags (using the same search term) tag:(tag-1 OR tag-2) my term
• search only on specific fields - description:(my term)
• inclusions of the previous options, i.e type:(dashboard OR visualization) tag:(tag-1 OR tag-2) my term

(The hard one) convert the client-side ast to KQL

For more advanced search needs, the find SO api allow consumers to pass a KQL query to be used for the search/filtering. If we can convert from the Query AST to a KQL node, we could add more advanced features, such as searching for different terms depending on the type of object, i.e dashboard:my-term OR visualization:another-term.

However it's important to note that:

• converting a syntax to another syntax is a very challenging task. The work here would be significant
• EUI's Query only supports per-field filtering (field:value another-field:(value1 OR value2 ) my search term). It would not allow us to parse things such as dashboard:my-term OR visualization:another-term. For this solution, we would need to implement our own lexer.

what do we want exactly

So, before digging any further on the technical level, I would like to know what we expect (ideally short and mid term) from this feature.

Regarding the issue's description, it seems the only need would be to be able to filter on a specific type using mytype:myterm or mytype:(myterm OR anotherterm)

So my questions are:

• Would the EUI Query syntax be acceptable, or should we directly exclude this option and create our own lexer
• Apart from type:term, what other kind of filtering do we expect for a MVP of the feature

Side note

type:term syntax

the search syntax interpretation will be at the GS API level, meaning that when a user type mytype:(myterm OR otherterm) search term, the result providers will receive an object that could look like

{
   term: 'search term',
   raw: 'mytype:(myterm OR otherterm) search term',
   fields: {
    mytype: ['myterm', 'otherterm']
  }
}

I would really like to avoid the type:term (i.e dashboard:flight) syntax in possible, in favor of the type:dashboard flight one (unless this is the ONLY type of advanced filter we would provide, which I doubt)

The reasoning is that mixing filters such as type:term (dashboard:flight) with field:value (tag:my-tag) will introduce a lot of complexity for providers to interpret the query.

For example, as the applications result provider, I know that if I receive { type: 'dashboard'}, I must not return any result, as type would be a statically defined field in the official GS documentation. On the other hand, If I receive { dashboard: 'some term'}, this is a dynamic field that depends on the list of existing saved objects. The provider would not have any way to understand that the user is searching only for dashboard.

We could, eventually, process this type:search syntax in, and pass down a processed object to the providers. That way, dashboard:flight would be returned as

{
   term: 'flight',
   raw: 'dashboard:flight',
   fields: {
    type: ['dashboard']
  }
}

However, this transformation feels non natural. Also, What are we suppose to understand from a query that would be dashboard:flight another search term than flight?

Result providers implementation overhead

As already said, the search syntax interpretation will be at the GS API level, meaning that when a user type mytype:(myterm OR otherterm) search term, the result providers will receive a structured object resulting from the query parsing.

however, the actual usage of this object will be delegated at the result provider's level. Meaning that any additional result provider we would be adding (or any solution team would be adding) would have to also be able to support the filtering, which is additional (significant) work, to be integrated into the GS results.

How do we educate our users

Advanced search syntax are something that can be very hard for users to lean, or even to understand.

• How do we keep that simple enough
• How do we educate our users, and document the filtering possibilities of the navigational search
• How do we handle syntax errors when the user types something that is invalid for the lexer

[ryankeairns]

My prevailing thought, as I read through your comment, was that we should lean towards a simple filtering experience as opposed to a query-writing experience. I believe the latter would generate confusion with the existing search query UIs and, frankly, introduces what feels like an unnecessary learning curve to this search UI.

Perhaps there's a future state where these are one in the same but, until then, I think it would be wise to maintain our current navigation-oriented focus and not mix purposes.

Given that, I favor (The easy one) using the search, type, references searchField options of the find API. option.

---

I would really like to avoid the type:term (i.e dashboard:flight) syntax in possible, in favor of the type:dashboard flight one (unless this is the ONLY type of advanced filter we would provide, which I doubt)

+1 for avoiding this

How do we educate our users

Definitely an important topic. We can do some design work around this.

[pgayvallet]

ping @alexfrancoeur I would like your opinion before starting a POC.

[alexfrancoeur]

Thanks for detailing out @pgayvallet and I apologize for the delayed response. I can understand how my original examples raised some additional questions. I'd like to ensure that the foundational steps we take for the initial implementation can grow to support additional result providers (https://github.com/elastic/kibana/issues/72680, https://github.com/elastic/kibana/issues/74283).

Search today, includes applications, saved objects and one feature (lens). In the future, we'll have more features and ideally, navigational entities. It feels to me like type should roughly translate to result provider or types within a provider. If that's the case, should we be more restrictive or provide stricter guidelines for adding new providers?

Using EUI syntax and with some of the assumptions made above, I'm going to try and put together some simplistic examples of syntax that help highlight the types of navigation I'd like to see navigational search support.

| Release | Provider | Intended navigation | Query | Results |
|-|-|-|-|-|
| 7.10 | application | Navigate to the Security overview page | type:application security overview | Return any application that matches "security overview" |
| 7.10 | saved object | Navigate to dashboard or visualization by AlexF | type:(dashboard OR visualization) AlexF | Return any dashboard or visualization with "AlexF" in the title |
| 7.11 | saved object | Navigate to dashboard with AlexF in the title and a production tag | type:dashboard tag:prod AlexF | Return dashboards tagged with "prod" that include AlexF in the title |
| 7.x | feature | Navigate to index management views in Stack Management | type:feature index | Return any feature that has index in it. May extend to results outside of "Stack Management" |
| N/A | entity | Navigate to pre-prod hosts in the Metrics UI | type:host pre-prod | Return any navigation results for host that match "pre-prod", potentially providing navigation to the Security applications as well |
| N/A | entity | Navigate to case with critical in the title | type:case critical | Return any case with "critical" in the title |

I'll answer some questions below, and am generally in favor of the easy one, but have an open question around filtering on multiple tags.

Would the EUI Query syntax be acceptable, or should we directly exclude this option and create our own lexer

For the most part, I think it's a good first step. But we'll eventually want to support more complex queries like dashboard:my-term AND tag:another-term. Given the examples provided, it's unclear to me if tags are being treated differently or if those were examples of field:value filtering.

Apart from type:term, what other kind of filtering do we expect for a MVP of the feature

type:term string gets us pretty far for MVP. There is boolean logic, and we don't need to filter by numeric value or ranges. Once we have additional metadata I could see filters on author or modified_date being helpful, but we're not there yet.

would really like to avoid the type:term (i.e dashboard:flight) syntax in possible, in favor of the type:dashboard flight

I agree that this would work, but how or can this work with tags? Do we believe that tags will be its own filterable component or an additional type? Given that we can support multiple tags, I'd imagine the need for searches like tag:(prod AND pre-prod) will be necessary. With the type:term could we do something like type:tag prod AND type:tag pre-prod or type:tag (prod AND pre-prod)?

Advanced search syntax are something that can be very hard for users to lean, or even to understand.

In addition to a quick link to documentation, a user specific product tour (all future features) could help with education. Could / should we support autocomplete? Maybe not initially, but in the future, that's a good way to learn syntax.

[pgayvallet]

release: 7.10

You probably meant 7.11, even for the first rows, right?

Search today, includes applications, saved objects and one feature (lens)

lens is today technically registered as an application, using its own provider that returns only this result. Should be resolved by https://github.com/elastic/kibana/issues/72680 though.

Using EUI syntax and with some of the assumptions made above, I'm going to try and put together some simplistic examples of syntax that help highlight the types of navigation I'd like to see navigational search support.

The first 3 would be possible to implement for 7.11 with the proposed implementation. The 4th one would be, as long as the feature result provider is also implemented on time.

For the most part, I think it's a good first step. But we'll eventually want to support more complex queries like dashboard:my-term AND tag:another-term

Not sure what this query is supposed to do? Return all dashboard assigned to tag another-term that matches the my-term clause? If so, it would be possible with proposed syntax, except it would be using type:dashboard tag:another-term my-term.

If that was meant to return dashboards matching my-term and tags matching another-term, it would not be possible atm, and would require some (impacting) changes on the saved objects find API (Not sure this would even be working atm by using KQL queries, but maybe)

I agree that this would work, but how or can this work with tags? Do we believe that tags will be its own filterable component or an additional type? Given that we can support multiple tags, I'd imagine the need for searches like tag:(prod AND pre-prod) will be necessary. With the type:term could we do something like type:tag prod AND type:tag pre-prod or type:tag (prod AND pre-prod)

tags will be its own filterable component or an additional type

From the discussions in https://github.com/elastic/kibana/issues/81846, tags are only going to be a filter, not an additional type. Where would we be redirecting the user to if we had tag results anyway? We don't have a 'physical' representation of tags atm (a page that represents the tag)

type:tag (prod AND pre-prod) would then not make any sense, as tag is not a type of result. If you want to search objects that are assigned to the prod and pre-prod tags, You would be using tag(prod AND|OR pre-prod) instead.

type:term string gets us pretty far for MVP. There is boolean logic, and we don't need to filter by numeric value or ranges. Once we have additional metadata I could see filters on author or modified_date being helpful, but we're not there yet.

👍

[alexfrancoeur]

tags will be its own filterable component or an additional type

This clears up most of the questions I had. Specifically around requirements for supporting filtering on different values by type.

I'm good with these requirements to kick off a POC. It will be interesting to see how this can evolve to support future requirements for search, but I love that we're providing the capabilities to fine tune the results. Thanks for leading the charge here @pgayvallet !

[pgayvallet]

Ok great, I will take a shot at it then

[pgayvallet]

After a very quick look and test, my first observation is that advanced search syntax does not really play well with the 'suggestion' type searchbar we are currently using for the navigational search (the fact that we refresh the results live on keypress, instead of requiring the user to press enter to trigger the search).

A quick example: the user want to type tag:prod foobar:

• the user start typing
• tag. GS returns objects matching the tag term
• tag:. GS returns nothing as the term is an invalid query (field:%NOVALUE%) and cannot be parsed. Note that in case if invalid Query syntax, we could just return the results matching the 'raw' query (that was gmail is going for example)
• tag:prod. GS returns all the SO objects that have the prod tag
• tag:prod foobar. GS actually returns the expected results

The main significant problems are step 1 and 2 imho.

That made me look a little on other products that are using/allowing advanced syntax in a single search field / search bar. I did not find any that was using complex syntax in workflows that also use 'live' results (refresh results while the user is typing).

That makes me wonder if we are really going with the right functional decision here.

Alternatives I could see:

• Disable the 'live' results from navSearch bar, to let the user type its entire query before we interpret it and fetch the results. I kinda don't like it though, as live results are a great functionality.
• Have a specific 'advanced search' page. When opening the navbar, we could add a 'advanced search' button / link that could redirect to a specific page where we could use a more 'traditional' search that would not be dynamic (such as the SO management page). It would also allow us to add, in addition to 'inlined' search syntax in the searchbox, new fields for all the filter we allow. We could have a tag multiselect and a type multiselect for now.

I think the second option would make sense, as it sounds acceptable for users wanting to perform some 'complex' search to be redirected to a specific page to do so, and as it would allow us to keep the KISS implementation of the navSearch bar while handling the 'more complex searches' need.

Note that I'm also good with just going with the current poposal. But maybe this 'advanced search' page could be a great (later?) improvement to the nav search.

@ryankeairns @alexfrancoeur @MichaelMarcialis @joshdover wdyt?

[MichaelMarcialis]

@pgayvallet: I personally don't think the live, on keypress results are a bad thing in conjunction with the proposed advanced search syntax. If the user were to type tag or tag: with no query value, I think it's perfectly logical and expected on our part to show results for those raw strings, similar to Google and Gmail as you indicated.

[alexfrancoeur]

I agree with Michael's response. It seems we've taken note of the same experiences. My mind went immediately to Google Drive. They have found a nice balance between live results and advanced syntax. For discoverability and curated filter experiences, I think there are other UI/UX we might evolve to support (advanced search, type filters listed below) but live results + query syntax feels like it's accepted behavior.

As you type in Google Drive, results are returned in real time. But as soon as you've completed your filter, the expected results are shown below. The largest difference I see here is the speed at which the results are returned, it feels pretty seamless with each keystroke. As an end user, I imagine that performance plays a large role in this UX.

They handle discoverability of the features with the "types" at the bottom. Though, one difference that's been previously called out is that they do have a dedicated results view. I could see this working for our use case without a dedicated view.

There is also an inline advanced search option that can build out a query. Another option to explore and a way to learn the syntax.

Unrelated / related question - have we done any "load testing" on query results? If there are thousands of saved objects, what does performance look like?

[myasonik]

One other thing we could explore is what I see VS Code doing.

CMD+P brings up a search box which, when you begin typing, searches for files.

CMD+SHIFT+P brings up the same search box with a leading character which changes what is being search on (in this case, it will perform an action rather than open a file).

I could see us extending this pattern for ourselves where just starting typing into the searchbox does a naive string match across pages.

A slightly different keyboard-shortcut and/or a leading character changes the searchbox to expect advanced filtering. (This would still allow a plain string but would parse things like tag:{name} instead of doing a string search on it and would also allow us to change how and when we load results to better optimize performance.)

And then if/when we got to it, scales pretty naturally (imo) to a third option for performing actions (e.g., create a new dashboard from the searchbox).

[pgayvallet]

Got it, will continue with the initial proposal then.

I made some progress on the implementation, however I'm currently stuck due to the searchbar's internal EUI component we are using. See https://github.com/elastic/kibana/pull/83422#issuecomment-728793229

[pgayvallet]

FYI, https://github.com/elastic/kibana/pull/83422 is ready for review.

[pgayvallet]

Created:

• https://github.com/elastic/kibana/issues/83888 about the tag:XXX suggestion option
• https://github.com/elastic/kibana/issues/83887 to discuss if we should translate the search syntax
• https://github.com/elastic/kibana/issues/83889 for type aliases (canvas-workpad -> canvas)

[pgayvallet]

Closing this as https://github.com/elastic/kibana/pull/83422 has been merged and as the follow-ups are listed in the Navigational Search project.