Beats: Support for Kibana Spaces in Beats modules

[alexfrancoeur]

We are currently targeting Kibana Spaces for 6.5 (https://github.com/elastic/kibana/issues/18948). With this implementation, Kibana will support multi-tenancy. If we change nothing today, every user that follows the add data instructions will end up importing the module dashboards into the default space (assuming they even have access to this). As an end user, if I followed these instructions, I would expect the dashboards to be available in the space that I'm currently in.

As modules currently utilize the dashboard import/export API (https://github.com/elastic/kibana/pull/10858), I want to make sure we support Spaces out of the box. This actually works today as long as the URL is formatted appropriately.

Today
API endpoint
/api/kibana/dashboards/export?dashboard=7adfa750-4c81-11e8-b3d7-01146121b73d

With Spaces
Default Space API endpoint (same as above)
/api/kibana/dashboards/export?dashboard=7adfa750-4c81-11e8-b3d7-01146121b73d

Specific Space API endpoint
/s/space-id/api/kibana/dashboards/export?dashboard=7adfa750-4c81-11e8-b3d7-01146121b73d

I've tested this with a Space ID and it does work. Are there any changes that we need to add to support this? Should we have users add the default Kibana URL and add an option to add one or multiple Spaces from CLI or the config yaml? How important is it to have a single data source (the beat) and dashboards in multiple Kibana Spaces?

We should also either intelligently add the URL / space ID to the UI tutorials or add some descriptive text around using the Space URL generically.

cc: @ruflin @tsg @legrego @kobelb @acchen97

[ruflin]

In any case we should add support for this in Beats. So the user could run something like:

metricbeat setup dashboards -E setup.dashboards.soace=space-id

I assume Kibana knows in which space the users is when looking at the add data ui so this could be dynamically generated?

Does Kibana have any capabilities to copy / make available the same dashboards across multiple spaces?

[ruflin]

cc @jsoriano

[kobelb]

I assume Kibana knows in which space the users is when looking at the add data ui so this could be dynamically generated?

That's correct, we'll have to make sure that this is exposed in such a manner that the add data UI can take advantage of it.

Does Kibana have any capabilities to copy / make available the same dashboards across multiple spaces?

For the initial implementation of Spaces, a Dashboard/Visualization will only exist in a single Space and there's not a mechanism to make something available in every space.

[alexfrancoeur]

@nreese you might be interested in following this thread as well

[webmat]

To expand on @kobelb's answer, my understanding of the spaces presentation is that it will be possible to export and re-import dashboards across spaces, just like one could export/import across clusters.

But you won't be able to "make available" per se, as he said. And from what I understood, it seems like the approach of having Kibana Objects 100% independent from one space to another is rooted pretty deeply in the implementation of spaces.

[ycombinator]

@alexfrancoeur said:

We should also either intelligently add the URL / space ID to the UI tutorials or add some descriptive text around using the Space URL generically.

Poking around the tutorials code in Kibana I found that the Kibana UI requests tutorials from the Kibana server via the GET /api/kibana/home/tutorials endpoint, which simply returns the array of registered tutorials. There is no provision, at the moment, for passing any sort of runtime contextual information (such as the Space ID) to tutorials so they can dynamically alter their content based on such information.

So if we want to add more dynamic-ism / intelligence to the tutorial content, this will require a bit of work on the Kibana side. @nreese thoughts?

[nreese]

@ycombinator Have you considered moving saved object loading into Kibana? Checkout the APM tutorial. There is a button to load all saved objects directly from within kibana. The objects are loaded via kibana's savedObjectClient so it should be naturally space aware.

Maybe beats should no longer load kibana saved objects?

[ruflin]

@nreese It's definitively one of the directions we should be heading long term. I would treat removing dashboards from our repos as a breaking change so I don't think we should do it in a minor.

[nreese]

Started a PR to show what one potential solution would look like
https://github.com/elastic/kibana/pull/22260

[legrego]

Started a PR which builds on Nate's PR above to supply the spaceId to all tutorial providers when Spaces is enabled: https://github.com/elastic/kibana/pull/22760

[chrisronline]

I have a PR in kibana to address this: https://github.com/elastic/kibana/pull/22998

[legrego]

@ycombinator @ruflin

We've merged Spaces into Kibana's master and 6.x branches.
We are still working through a couple of issues, but what we have merged is stable at this point, if you want to move forward with your work.

Let us know what we can do to help, and of course, let us know if you run into any issues integrating with Spaces.

[ycombinator]

Thanks @legrego. I'll be merging https://github.com/elastic/beats/pull/8045 shortly.