Kibana: [APM] Excluded 6.x legacy data is not apparent

[tylersmalley]

In 7.0+, users may have upgraded and will have no pre 7.0 data until it's been migrated to the new ECS format.

Currently, there are two places in APM where we mention the need to migrate:

In the APM tutorial, it mentions "You can also migrate your 6.x data with the migration assistant found in Kibana's management section.". This could also use a link to the upgrade assistant.

If there is NO data, we mention that it's possible there is old data that needs to be migrated.

My feeling is we should actively check to see if there is missing legacy data within the current context. Take APM in Cloud as an example, once the stack has been upgraded to 7.0 it will begin producing data and users will only see the messaging in the tutorial.

[elasticmachine]

Pinging @elastic/apm-ui

[jasonrhodes]

Just a few notes for context:

• We tried to include a link from the setup instructions page but the declarative setup of that section only allows raw text there so it wasn’t possible.

• As far as we know, there isn’t a simple and performant way to check for 6.x data on a per query basis, or even once overall on load. @tylersmalley if you have suggestions there, please let us know. We currently filter out 6.x data from queries so that it doesn’t break the UI.

• We were also concerned about users who want to keep old data around but not reindex it. Those users will be notified about old indices for … all of 7.x? AIUI, renaming the old indices isn’t easy or possible so they have to delete, reindex, or live with the constant notifications.

FWIW I’d prefer if these kinds of things could be done in one central place where users are encouraged to go after a major upgrade, a sort of upgrade health check to make sure you’ve done all the upgrade steps without having to pollute the code with checks and notifications. If that doesn’t exist and we think there are more users who would benefit from the notification to migrate, who wouldn’t otherwise know about index migration, then we probably need some help thinking through how to get that info in a way that makes sense.

[jasonrhodes]

Ping @makwarth @alvarolobato

[sqren]

This could also use a link to the upgrade assistant.

As mentioned by @jasonrhodes we cannot add a link in the getting started guide, since the tutorial schema is created server side, and then rendered client side. We would need some kind of DSL to allow creating links, since rendering arbitrary html is dangerous.

My feeling is we should actively check to see if there is missing legacy data within the current context.

We should be able to do this. We can make a request that filters for 6.x data in the given time range. If any is returned we show a toast or callout about missing data that needs to be migrated to show up. If the toast is dismissed it will still show up next time the user navigates to the page.
We can do this check on every page (on full page load) or just on the services overview (list of services).

[sqren]

FWIW I’d prefer if these kinds of things could be done in one central place where users are encouraged to go after a major upgrade, a sort of upgrade health check to make sure you’ve done all the upgrade steps without having to pollute the code with checks and notifications.

Big +1. What is the current approach for eg. dashboards and visualisation? How are users informed about data that needs migration?

[tylersmalley]

Ideally, this migration would have been done in 6.7 allowing any checks or duplicated functionally to be removed starting in 7.0. This is similar to what the stack had to do in 5.6 to support single and multi-type indices. Unfortunately, we didn't have time for this approach - just something to keep in mind going forward.

I don't think we should have actions that need to be done by the user post upgrade to restore usage to our products. Preferably, these sort of things should be done prior to the upgrade so the products function afterward.

As for dashboards and visualizations, it's possible these can be done automatically through Kibana migrations.

[tylersmalley]

There are a few ways that I can think of to make the check for legacy data.

• when searching for data, perform instead us an msearch to also do a count for legacy data within the time filter
• when performing aggregations, could you use a filter aggregation for legacy data?
• check the index names as we do in the upgrade assistant. This doesn't have the benefit of being constrained by the time filter.

[jasonrhodes]

I don't think we should have actions that need to be done by the user post upgrade to restore usage to our products. Preferably, these sort of things should be done prior to the upgrade so the products function afterward.

This is interesting and something I've never heard before. With breaking changes across a major version, I'm not sure how it would be possible to migrate data before the upgrade is complete. Having to perform an action to restore service seems like an acceptable part of a major version upgrade, but maybe I'm misunderstanding something?

There are a few ways that I can think of to make the check for legacy data.

Thanks for these! We'll look into this and come up with a plan.

[tylersmalley]

This is interesting and something I've never heard before. With breaking changes across a major version, I'm not sure how it would be possible to migrate data before the upgrade is complete. Having to perform an action to restore service seems like an acceptable part of a major version upgrade, but maybe I'm misunderstanding something?

We want to be able to perform a rolling upgrade, even across major versions. In order to accomplish this, we require users to upgrade to the last minor version before performing a rolling upgrade. During this version, we can manage any compatibility required for the current and future version. For data changes, like we have done here, it should be possible to account for both data types and drop all handling of legacy data in the next major version. This is how it has been done in the past, for example with the removal of mapping types. In 5.6 most products supported data with both single and multi-types. This allowed us to migrate the indices to a single type in 5.6 so they would be compatile with 6.x.

[graphaelli]

While making these changes, please also update the copy in the setup instructions to match the language in the APM UI to make it more assertive. It currently reads:

You can also migrate your 6.x data with the migration assistant found in Kibana's management section.

substituting can also for also need to would be a start.

In addition, please ensure these instructions will show up on ESS too.

[jasonrhodes]

substituting can also for also need to would be a start

Let's chat about this in person tomorrow too. My understanding was that users _don't need to_ migrate old indices, and that many of them will in fact choose not to and rather just move forward with new data.

[tylersmalley]

Had a discussion live over a Zoom call, here are the bullet points:

• Starting in 7.1.0, APM will augment the search requests to detect legacy data (count using terminate_after=1 and add a notice based on this.
• The wording on the notice can be changed to let the user know there is missing data and how to resolve it if they wish.