Kibana: [Meta] Embeddable types

[stacey-gammon]

Let's use this meta issue to track all our current embeddable types, embeddables currently being implemented, and embeddables that different teams would like to implement. This way the App Arch team can be sure to give support where needed!

Current Embeddable Types

• Maps
• Visualization
• Search
• Dashboards
• Lens
• ML Swimlane
• In progress Endpoint Resolver

Embeddables being used in

• Dashboard
• Done Canvas. Embeddable support was merged but has some bugs:
  
  
  
  • Trigging filters action with embedded visualizations in Canvas causes exception
  
  
  • Embeddable Expression should reflect embeddable state
  
  
• Done Map pew pew in SIEM
  
  
  
  • Provide ability to know when map/embeddable is loading
  
  
  • [x] [Provide ability to edit actions displayed in context menu and on map](https://github.com/elastic/kibana/issues/43643)
  
  

cc @streamich

[elasticmachine]

Pinging @elastic/kibana-app-arch

[andrew-goldstein]

@stacey-gammon thanks for your help during the SIEM team's spike on embeddables for the Overview page! 🙏

As detailed in the issue linked above, the following issues are relevant to the SIEM team's plan to use embeddables in a future release:

• #52680 - [Embeddable] Dashboard by reference embeddable

• #53196 - [Embeddable] Add Export to PDF and Export to PNG UI Actions to the Embeddable Dashboard

• #52806 - [Lens] Trigger a filter action on click in embedded visualizations DONE!

• #16917 - Dashboard level index patterns

• #33496 - Allow plugins to register default saved objects that will be created for every space

[stacey-gammon]

Issuing a call for more use cases of users/devs wanting "UI things" from other apps/solutions to be addable to a dashboard or canvas workpad.

@AlonaNadler - you know of any more to add here that are being considered?
@tbragin - any requests for this coming from Observability?
@epixa - any known use cases from security (aside from the Resolver one being worked on already)?

We are rethinking the add panel flow and specific information about embeddables that devs want to add will be useful (https://github.com/elastic/kibana/issues/64383)

[epixa]

@spong @XavierM Can either of you think of anything here? Or perhaps you could ping some folks from the endpoint side if you think they have ideas.

[tbragin]

@stacey-gammon Here are some initial thoughts, some of them previously discussed with @AlonaNadler

Happy to chat further to sync up on priorities between these.

APM charts:

• Transactions per second/response times
• Top transactions table including impact column
• Distributed trace timeline

Metrics:

• Waffle map

Logs:

• Embeddable Logs UI

cc @roncohen @sqren @jasonrhodes

[AlonaNadler]

@tbragin, I'm assuming these are all types of observability charts you want users to be able to add to a Dashboard or a Canvas. https://github.com/elastic/kibana/issues/64528

@stacey-gammon we already have also ML that have a PR to add ML job using a swimlane into a dashboard and Canvas https://github.com/elastic/kibana/pull/64056

[stacey-gammon]

thanks @AlonaNadler - just fyi though, the ML job swimlane will not be addable to Canvas without additional work. The way Embeddables in Canvas is built, every new type has to be manually supported. This is probably a topic for another issue though. (cc @crob611 )

[mchopda]

@spong @XavierM Can either of you think of anything here? Or perhaps you could ping some folks from the endpoint side if you think they have ideas.

@MikePaquette @andrew-goldstein ^^

[spong]

There was a recent discuss post (item 5) that was requesting ML job anomalies be surfaced on the SIEM Overview page. So I think we on the security side could leverage the new ML Swimlane embeddable for that, so it's good to see 🙂

As far as new embeddables for SIEM to consume, not too much comes to mind outside of perhaps some of the Alerting UI's? Though I'm not sure how much sense it makes for them to be exposed as embeddables vs a generic re-usable UI? For instance, we've recently added Notifications to our Detection Engine Rules, and used the connector and notification configuration UI from Alerting. We'll be adding Threshold Based Rules here soon as well, and are looking at the Alerting Metric Threshold UI to see if we could leverage that. Any future Activity Monitor UI Alerting provides could be of use as well. Again, not sure if they make sense as embeddables, but wanted to bring them up in case we're moving to a direction where shared components between plugins should be done via embeddables.

Lastly, I'd be curious to see what people think about exposing the SIEM Timeline as an embeddable? This has come up before, but with the recent Timeline re-design effort, it might be a good idea to revisit this as part of that effort.

[jasonrhodes]

@stacey-gammon I would just like to clarify the embeddable use case for myself as I think about some of this (and go through some old notifications that got lost!). If the Logs app wanted to embed a map on one of its pages, what would be the advantage of using a Kibana Embeddable as opposed to a React component provided by the Maps team? My initial thought was that Kibana Embeddables would be more about making items available to users to embed on the fly, the way they can embed any viz they want into any dashboard they like, but are we also thinking of using them to share components across Kibana apps and teams as well?

Thanks!

[stacey-gammon]

Oh gosh, thanks for the ping! I meant to follow up on this. It's a great question!

Some clarifying questions, and then I'll make educated guesses on them with my answers but correct me if I'm wrong.

You are saying "Maps Kibana Embeddable" vs "Maps React component" , but the Maps Kibana Embeddable _is_ a react component. I'm going to assume you mean the react component that maps uses _inside_ the embeddable react component.

One advantage to using Embeddables is that you get things like attached ui actions (drilldowns), and, depending on how you render the embeddable, you can get the "panel wrapper" which exposes the gear icon which opens the context menu panel and shows all registered actions that can act on that embeddable (even things that aren't map specific actions like "customize time range").

If you want to use this functionality, you should still grab the element directly from the map plugin itself instead of going through the generic embeddable registry (a recommendation that we don't have documented anywhere but should).

The other big reason to use embeddables is for dynamic usage, when you don't know what type of embeddable you have at runtime. You do, because you are saying "maps", so you shouldn't need to go directly through the embeddable api.

e.g.

Prefer:

 const MapEmbeddable = mapsStart.MapEmbeddable;
 return <MapEmbeddable props={...} />

to:

// Don't use generic access when you know the type at compile time.
const mapEmbeddable = embeddableStart.getEmbeddableFactory(MAP).create(input);
mapEmbeddable.render();

However, if you don't want any drilldown or action functionality, you can definitely just use any functionality or react components they share off their plugins contract, or exported statically.

Let me know if this helps!

cc @ppisljar

[jasonrhodes]

@stacey-gammon this is fantastic context, thanks! I plan on playing with these some over the next few weeks to see how Observability UIs can leverage them or provide our own to the registry. Thanks again!

[stacey-gammon]

FYI

@Dosant just merged https://github.com/elastic/kibana/pull/67783 which adds a new developer example for embedding dashboards by value. I believe a "by reference" one is up next, as a lightweight wrapper around the "by value" version.

Just go to Developer examples app when running Kibana with yarn start --run-examples and click the Dashboard container section.

@tsg @andrew-goldstein it would be helpful to determine what are actual blockers to adopting embeddable dashboards in SIEM, and what is a nice to have, as I think almost everything is just about there.

52680 - [Embeddable] Dashboard by reference embeddable

Coming soon, but could still use the "by value" one to get around it.

53196 - [Embeddable] Add Export to PDF and Export to PNG UI Actions to the Embeddable Dashboard

Nice to have?

52806 - [Lens] Trigger a filter action on click in embedded visualizations

Done!

16917 - Dashboard level index patterns

Nice to have?

33496 - Allow plugins to register default saved objects that will be created for every space

I think this is the last one that is required left? I added a comment in the issue, there may be a simple solution for this.

[andrew-goldstein]

FYI

@Dosant just merged #67783 which adds a new developer example for embedding dashboards by value. I believe a "by reference" one is up next, as a lightweight wrapper around the "by value" version.

Just go to Developer examples app when running Kibana with yarn start --run-examples and click the Dashboard container section.

@tsg @andrew-goldstein it would be helpful to determine what are actual blockers to adopting embeddable dashboards in SIEM, and what is a nice to have, as I think almost everything is just about there.

52680 - [Embeddable] Dashboard by reference embeddable

Coming soon, but could still use the "by value" one to get around it.

I tried running the Dashboard container example via yarn start --no-base-path --run-examples to see whether or not it's been updated to support reads and writes in edit mode, but per the screenshot below, I don't think the new example provides an edit mode (like the previous example did):

@stacey-gammon would you happen to know if the "by value" example above or the upcoming "by reference" example supports reads and writes in edit mode?

53196 - [Embeddable] Add Export to PDF and Export to PNG UI Actions to the Embeddable Dashboard

Nice to have?

I would put Export to PDF in the "Nice to have" bucket. CC: @MikePaquette

52806 - [Lens] Trigger a filter action on click in embedded visualizations

Done!

Awesome! 🎉

16917 - Dashboard level index patterns

Nice to have?

@stacey-gammon please see the SDH related to the Maps embeddable on the SIEM Network page for an example of the confusion caused by Synchronization Issues With Kibana Index Patterns Referenced by Embeddables. This still feels like a blocker.

33496 - Allow plugins to register default saved objects that will be created for every space

I think this is the last one that is required left? I added a comment in the issue, there may be a simple solution for this.

The pattern of creating things on demand when a user visits a page in an app (like the Detections page) is associated with open issues, community posts, and bugs:

• There's still an open issue for how to handle read-only users when they visit the Detections page

• This community post is an example of this pattern producing a less-than ideal OOTB experience. Novice users, who may not be familiar with Elasticsearch security privileges, must navigate the following documentation, which again, is less than ideal.

• This recent fix in another part of the SIEM app was necessary because there's no guarantee a user with proper permissions visited the Detections page before using another part of the app. When there are no guarantees that things exist, branching logic must exist throughout the app to compensate.

This still feels like a capability that should be provided by the platform.

[andrew-goldstein]

@stacey-gammon, @XavierM, @ppisljar and I met today to review the issues referenced in the previous comment above, and discuss solutions:

• https://github.com/elastic/kibana/issues/52680 is still a blocker due to the complexity of translating from the saved object representation of a dashboard (and the visualizations it references) to the configuration / props required to render a dashboard via Embeddables. To resolve it:

  • Reads: The platform team would provide an API or utility function that reads the saved object representation of a dashboard ("by reference") and translates it to the format required by embeddable dashboards. Implementing reads is the minimum work required to remove the blocker, but ideally writes would be implemented at the same time, per the next bullet.
  • Writes: To persist changes made by users when the dashboard is in Edit mode, the platform would provide an API or utility function that allows changes made by users to the contents of the dashboard (and it's visualizations, if they have also been changed) to be written back to their respective saved object representations. Note: since only users with permissions to update the saved objects may save these changes, the Edit action in the UI should be disabled if the user lacks permissions to save them, and the API should enforce this server-side.

• 52806 We're keeping Export to PDF in the "Nice to have" bucket

• https://github.com/elastic/kibana/issues/52806 Is unblocked, as previously noted 🎉

• https://github.com/elastic/kibana/issues/16917 can be removed as a blocker _if_ the SIEM app implements Kibana Index patterns. (This would not be a trivial change to the SIEM app, but it would eliminate the impedance mismatch that's a blocker today.)

  • The effort estimate on this item is "XL"

• https://github.com/elastic/kibana/issues/33496 looks like it can be unblocked without changes to Embeddables, with the following caveats / assumptions:

  • The base assumption is we can create a new server side route in the SIEM app that is invoked on application start the first time a user visits a newly-created Kibana space. (This assumption must be verified.)
  • The implementation of this route would have all the information necessary to populate the space, thus avoiding the need to send ndjson descriptions of saved objects to the client, which can be very large.
  • Since this route will be invoked via start, it _should_ be possible to ensure the route has the privileges necessary to populate the new Kibana space, regardless of the privileges of the logged-in user. (This assumption must be verified.)
  • @stacey-gammon highly recommends the implementation of the route utilize the Import / Export APIs for populating the space, as opposed to invoking the Saved Object APIs directly, because the Import / Export APIs will handle migrations when formats change.
  • There may be some additional "unknowns" here, so the effort estimate on this item is "large"

Recommendations

We recommend @hmnichols and @MikePaquette follow-up with:

• @stacey-gammon re: the planning / logistics of implementing at least the "Reads" part of https://github.com/elastic/kibana/issues/52680 , and ideally, the "Writes" part, so the SIEM team may consume it in a future release
• the largerTeam:AppArch re: Export to PDF, which is not a blocker, but required to fully realize our vision
• @XavierM on the two "large" / "XL" items, which are no longer considered blockers from the perspective of Embeddables, but still rank high in terms of effort / complexity:
  
  
  
  • Index patterns, as related to https://github.com/elastic/kibana/issues/16917
  
  
  • Seeding spaces with dashboards and visualizations as saved objects, as related to https://github.com/elastic/kibana/issues/33496
  
  

[andrew-goldstein]

@stacey-gammon, @XavierM, @ppisljar and I met today to review the issues referenced in the previous comment above, and discuss solutions:

@hmnichols @MikePaquette For completeness, it's worth noting that we also discussed the impediment that saved objects, which are used to persist cases and timelines, don't support aggregations, as @kobelb documented here: https://github.com/elastic/kibana/issues/64137

Although not a limitation of embeddables themselves, some of the embeddable dashboards we would like to implement cannot be realized until Lens, charts, etc can perform aggregations on saved objects.

[XavierM]

@stacey-gammon, @XavierM, @ppisljar and I met today to review the issues referenced in the previous comment above, and discuss solutions:

@hmnichols @MikePaquette For completeness, it's worth noting that we also discussed the impediment that saved objects, which are used to persist cases and timelines, don't support aggregations, as @kobelb documented here: #64137

Although not a limitation of embeddables themselves, some of the embeddable dashboards we would like to implement cannot be realized until Lens, charts, etc can perform aggregations on saved objects.

This PR will allow us to do aggs https://github.com/elastic/kibana/pull/64002