Kibana: Do any type of query from a saved objects client

[majagrubic]

Some background:
When adding a visualization to the dashboard, we'd like to hide this visualization from the end user. From the end user perspective, this visualization doesn't exist in the Visualize app, but only in the context of the dashboard.

Plan:
To do that, I'd like to add a new flag to visualization saved object, called visible. All visualizations would have this flag set to true by default, except the ones created by the dashboard, which would have this set to false. If the flag is false, we wouldn't show this in the visualize listing. (Draft PR for the flow above)

Problem:
I thought that filtering items on the client side would be enough. This won't work in the saved objects management console, where we rely on the total number of hits in the ES response to do the pagination:
https://github.com/elastic/kibana/blob/7df981fbf8f81e1df205082a883a74ca99839e22/src/legacy/core_plugins/kibana/public/management/sections/objects/components/objects_table/objects_table.js#L226
It could happen that ES finds 170 total saved objects, out of which 168 are visualizations, and out of those we only need to show 37, so the pagination would be messed up.

Which leads to changing the ES query itself. Query is issued through the saved_object_client.find method:
https://github.com/elastic/kibana/blob/8e9a8a84dccfa7965ce8a22362885e6cdef8b51f/src/legacy/core_plugins/kibana/server/routes/api/management/saved_objects/find.js#L88

which transforms the options to simple_query_string and looks on specific fields (ie. title).

"must": [
   {
     "simple_query_string":
        {
          "query": "bla*" ,
           "fields": ["config.title", "maps.title", "visualization.title"....]
         }
   }
]

Is there a way to make the saved object client accept any type of query?
I think the query I need would look something like:

{
  "query": {
    "bool": {
      "must": {
       "simple_query_string": {
         "query": "bla*"
       }
      },
      "should": [
        { "term": { "type": "visualization "}},
        { "term": { "type": "dashboard"}},
        { "term": { "type": "maps"}}
      ], 
      "must_not": {
        "term": { "visualization.visible": "false" }
      }
    }
  }
}

Alternatively, there's conditional clauses, but that can't be done through the client if I understand things correctly.

@pgayvallet suggest adding a visible field to all the saved objects as a mitigation for this issue, but it seems like an overkill for what I'm trying to do.

Would appreciate any ideas/thoughts/comments on how to proceed with this.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

Saved Objects management is primarily used for importing and exporting saved objects and I think the primary user would be an administrator making backups or importing configuration from i.e. a staging environment where it was tested into a production environment.

So while the dashboard/visualizations app could hide or combine saved objects to simplify it for the user, in the data persistence layer (for which saved objects management is the ui) we could have a complex network of objects.

In a sense, this UI allows you to peek "under the hood" of Kibana and there could be many objects here that you wouldn't otherwise see directly in the UI. Does it matter that these dashboard visualisations are visible there? Apart from the technical challenges you mentioned it can also create unexpected behaviour where you select to export all objects (i.e. 10), but your resulting export file has more objects than what you selected.

[jportner]

In a sense, this UI allows you to peek "under the hood" of Kibana

To add to this, the saved objects client relies on the repository to generate the query for find operations. When it comes to Spaces, we add query clauses to filter out and make sure you only get results for the appropriate space. Exposing customization for queries would open up a potential vulnerability to bypass our access controls in the secure saved object client wrapper.

[majagrubic]

Does it matter that these dashboard visualisations are visible there?

From product perspective it would be best if those objects are not there. From the end user point of view, they don't exist. Additionally, we will be saving them with some random generated title - the idea is that the users won't need to enter the title anymore. Having a bunch of visualizations with some random title pop up in the management console when the users wouldn't expect them there could lead to a confusing user experience.

In a sense, this UI allows you to peek "under the hood" of Kibana and there could be many objects here that you wouldn't otherwise see directly in the UI

This is actually a really good point. I guess if the complexity around not showing those objects in the UI is too high, it is an acceptable tradeoff to make.

[joshdover]

Having a bunch of visualizations with some random title pop up in the management console when the users wouldn't expect them there could lead to a confusing user experience.

We could make the title in UI show up as something different for these "embedded" visualizations. For instance they could be displayed as "Visualization in Dashboard X"

[legrego]

It looks like the _find endpoint accepts a filter query parameter, which in turn expects a Kuery string. If you can represent your criteria as a Kuery, then this might already work for you.

https://github.com/elastic/kibana/blob/d61ef267d530554cd048b5e602a0b6c2e362185f/src/core/server/saved_objects/service/lib/repository.ts#L507-L517

Implemented in https://github.com/elastic/kibana/pull/41136

[legrego]

Just for my own education -- is there a reason to create these embedded visualizations as their own saved objects? If they are only meant to be consumed within the context of a single dashboard, could the visualizations just be directly embedded into the dashboard's saved object?

I suppose you could also consider (and maybe you already have) creating a hidden_visualization saved object type, which has the hidden flag set to true on the saved object schema. That would prevent it from being used anywhere, unless the calling code explicitly opted into "seeing" this hidden saved object type.

[rudolf]

Administrators are (rightfully) very scared when they're exporting objects for backups, if the numbers don't match it would reduce their confidence and they would probably not be comfortable to continue. So seeing some an embedded visualization without knowing where it came from is better than making a backup and suddenly seeing more objects in your export than what you selected, or restoring and then seeing less objects in Kibana than what you imported. So although a KQL filter will solve the technical problems I'm still not sure this is an improvement for the user experience.

I like @joshdover suggestion for naming the "embedded" visualizations. We should probably just use id references instead of title references otherwise these could get out of sync.

Directly embedding the visualization in the dashboard as per @legrego's suggestion is worth exploring, this would avoid any confusion whether for a dashboard user or a saved objects management user like a system admin. The downside is that it requires keeping the mappings between visualizations and embedded visualizations in sync and it complicates migrations if we ever change the format of visualizations - we would need to remember to migrate both dashboards and visualizations saved objects. It would also require changes to telemetry since we'll have to also search inside dashboards. So all in all it introduces more technical complexity for improved UX.

creating a hidden_visualization saved object type, which has the hidden flag set to true on the saved object schema

This would also hide the object from the Saved Objects REST API which will then require creating a custom REST endpoint.

[majagrubic]

@rudolf we already explored the option of embedding the visualization in the dashboard config directly. This is something that we want to do long-term, but for now are going with the approach of simply hiding the vis from the user.
See this doc for more info. It also contains links to PoC PRs we did.

PS. great point about the export and not being surprised by the sudden increase in the objects

[rudolf]

@majagrubic What's the status on this, is there something more you'd like to discuss or anything we should action?