Kibana: APM Sample Data

[graphaelli]

In order to improve the getting started experience, we'd like to add sample data to Kibana to power APM ~+ Infra + Logs~ so users can get hands on with them. Edit: leave infra/logs/uptime for a follow on effort.

elastic/kibana#17807 originally added sample data into Kibana, so could help as a reference for implementation.

~Logs UI already supports the sample data index (kibana_sample_data_logs) by default, so populating that index should just work.~

~Infra UI also includes the logging sample index by default but not one for metrics.~

~APM UI does not include a sample data index by default. elastic/kibana#39741 could fit in nicely to enable that.~ edit: use an index name that matches the exist pattern

See https://github.com/elastic/kibana/issues/40704#issuecomment-547198210 for more implementation details

cc @tbragin @nehaduggal

[elasticmachine]

Pinging @elastic/apm-ui

[nreese]

Sample data sets can load multiple Elasticsearch indices so perhaps the existing web logs sample data set could just be enhanced with the indices needed by apm.

[jasonrhodes]

Perhaps obvious to say but it'd be excellent if the data was planned out so that it looked like it was all connected, so we can demonstrate the interconnected features of our various apps. (That means we also need to be able to easily update it when we add new connections between apps.) Can sample data have dynamic dates applied?

Another reason to consider building a simple data generator to use for dev that could export this kind of data... space time someday!

[nreese]

Can sample data have dynamic dates applied?

Yes. When you install sample data sets, the timestamps identified with timeFields in the schema are shifted into the present.

[jasonrhodes]

@nreese brilliant! Thanks

[graphaelli]

Narrowing the scope on this for now, let's only tackle APM data and tie in logs and metrics connections as a follow on.

The plan is now:

• [ ] Import sample APM data into a new randomly generated index name that matching the default configuration, eg apm-7.4.1-transaction-sample-data-laishjvl
• [ ] Set sample data service.environment to apm-sample-data to clearly distinguish it even when overlapping data is loaded
• [ ] Allow import of sample data from the 'no services' page right in the APM UI
• [ ] Use the same mechanism from the sample data page
• [ ] Stretch goal - drop sample data button. This will phase out over time anyway, would be nice to have.

Open question - can we work backwards from the pattern to a random name that matches? Alternatively, can we warn the user / hide the create sample data flow if the user has modified apm_oss.*Indices?

Per discussion, we do not wish to have a workflow that is special to sample data - eg pass sample data index name around in query params or otherwise.

[nreese]

Import sample APM data into a new randomly generated index name that matching the default configuration, eg apm-7.4.1-transaction-demo-laishjvl

Kibana's sample data implementation prefixes index names with kibana_sample_data_ and then the id of the sample data set

[sqren]

Kibana's sample data implementation prefixes index names with kibana_sample_data_ and then the id of the sample data set

@nreese Yes we talked about this and agreed it was better to ingest sample data into the APM specific indices. This way the APM app doesn't have to do anything to handle sample data - it's just normal data. It'll still be in a separate index which makes it easy to wipe again.

[sqren]

can we work backwards from the pattern to a random name that matches?

@graphaelli I don't think so. I think we'll just have to pick an index name eg apm-{version}-transaction-sample-data-{uuid}. If it doesn't match the index (pattern) specified by the user (eg apm-*) I'm okay with either hiding the "Add sample data" button or showing the user a warning.

An alternative approach could be to take advantage of runtime index configuration settings and overwrite them with the sample data index name: apm-{version}-transaction-sample-data-{uuid}.
The (big) downside to this approach is that users will not be able to see anything but sample data until they change their settings. Which probably makes this a show-stopper.
EDIT: clicking the "Delete sample data" button could reset the runtime index configuration indices. This would make it easier to revert back to the defaults.

Stretch goal - drop sample data button. This will phase out over time anyway, would be nice to have.

For us to delete the sample data again we'll need to know the index name. Since the index name is dynamic (contains a uuid) we'll need to store this somewhere (eg. saved objects) when the sample data index is created.

[graphaelli]

I don't think so. I think we'll just have to pick an index name eg apm-{version}-transaction-sample-data-{uuid}

Fair enough. I like the change from demo to sample-data and edited my comment accordingly.

For us to delete the sample data again we'll need to know the index name. Since the index name is dynamic (contains a uuid) we'll need to store this somewhere (eg. saved objects) when the sample data index is created.

I was thinking we would delete apm-{version}-transaction-sample-data-* but I like the idea of being more precise. I think we should still leave this as a stretch goal unless saving that information (probably in saved object, as you noted) is useful otherwise.

[bmorelli25]

@graphaelli Where is the sample APM data for this project going to come from? Opbeans?

[graphaelli]

That's the plan, planning to pull from one of the kubernetes clusters.

[graphaelli]

Spent some time on this

: https://github.com/elastic/kibana/compare/356e3a47768701740ae99863f8e0089258874a20...graphaelli:obs-sample-data?expand=1 and have some notes for the implementer:

• Sample data is intended show off kibana viz/dashboards. I'm concerned that showing prospective users sample dashboards without the curated UIs would be confusing - instead is it straightfowrad to only make the sample obs data available when the Observability curated uis are available? IIUC, the presence of the saved objects indicates that the sample data is already loaded - I expect we can find a workaround for that.
• Addition of "open in APM App" link wasn't clear, probably b/c i'm out of the legacy/NP loop
• Sample data indexes are always prefixed kibana_sample_data_ - one workaround would be to create an alias to apm-kibana_sample_data_observability to make it show up in the UI. Alternatively, perhaps a custom index name could be passed in.

I would be happy to continue working on the sample data content if someone could work on wiring all of this up.