Kibana: Allow additional solutions to take advantage of the telemetry opt-in

[kobelb]

The Security solution team would like to send anonymous telemetry about the effectiveness of their detection rules, and to do so they'd like to take advantage of the existing telemetry opt-in mechanism. The existing telemetry plugin will not be responsible for shipping the telemetry data, it will only expose to the Security solution's plugin that the user has opted into telemetry being sent.

Additionally, we should change the example of the usage data we collect to include this information:

/cc @alexfrancoeur @stevewritescode @rylnd

[alexfrancoeur]

I'm sure @stevewritescode or @cosiomoises are already working with legal, but if it's a different type of telemetry we'll have to check and see if we have to update our privacy statement as that's what we're referencing in the notice.

[stevewritescode]

@alexfrancoeur our product privacy statement already covers security telemetry since it needs to address the Endgame/Elastic integration that is current sold: https://www.elastic.co/legal/product-privacy-statement#information-we-collect. However, we may need to tweak the wording slightly - IIRC pre-built detection rules didn't exist at the time that the privacy statement was last updated so it may not be clear in the currently wording how we collect data about those. I'll make sure legal is engaged long before we release any changes to security telemetry in the stack.

[cosiomoises]

agree with @stevewritescode. We have good wording in the current Endgame EULA that enabled us to bring telemetry data for improvement of our products. It seems we might need to add an example of the detection rules. We work before with Carl Spataro and he might be able to help or point us into the right direction

[stevewritescode]

May also need a minor text change to https://github.com/elastic/kibana/issues/74690.

[cosiomoises]

met with @alexfrancoeur and @stevewritescode and agreed on the following:

We will adopt the same paradigm that current usage telemetry follows in terms of opt-out options

• cloud deployments wont have an opt-out options
• for on-prem deployments we will update the current text to enable/disable usage telemetry to make sure we include the "Product performance" telemetry which will include this event data (alert data)

[TinaHeiligers]

Additionally, we should change the example of the usage data we collect to include this information

@kobelb The example data that's shown is a snippet of the payload that Kibana sends to the remote Telemetry service. Since Security won't be shipping data using the telemetry setup in Kibana, that data won't be in the payload for the POST request to /api/telemetry/v2/clusters/_stats.

[TinaHeiligers]

The existing telemetry plugin will not be responsible for shipping the telemetry data, it will only expose to the Security solution's plugin that the user has opted into telemetry being sent.

The simplest way to check if telemetry is opted in or not is to read the value of the telemetry.optIn flag either in the kibana.yml or in the .kibana telemetry saved object. One reads it as one would any other saved object in Kibana.

Examples where this is done are:

1. Kibana Welcome Screen
2. telemetry section of the advanced settings

If that's all that's needed, then the majority of the work here is figuring out how to add non-telemetry payload data to the example data. One would need to add it to the data that's rendered in the panel

[kobelb]

@kobelb The example data that's shown is a snippet of the payload that Kibana sends to the remote Telemetry service. Since Security won't be shipping data using the telemetry setup in Kibana, that data won't be in the payload for the POST request to /api/telemetry/v2/clusters/_stats.

Agreed. We'll likely want to add a new section to the existing fly-out, or add a new fly-out which includes this information.

The simplest way to check if telemetry is opted in or not is to read the value of the telemetry.optIn flag either in the kibana.yml or in the .kibana telemetry saved object. One reads it as one would any other saved object in Kibana.

We should be treating where we store the opt-in status as an implementation detail, and not requiring solutions to rely on these implementation details. Instead, the telemetry plugin should be exposing on its start contract a method to determine whether or not the user has opted into telemetry. I believe this is being done client-side using TelemetryService#isOptedIn but I don't believe it's available server-side.

[tsg]

To record another request, in addition to the opt-in status, it would be useful to get access to a bit more information:

• the configured Telemetry URL. The Security telemetry will most likely post to a different path under the same URL.
• get cluster metadata about the cluster (at the minimum the cluster UUID)

[TinaHeiligers]

We'll likely want to add a new section to the existing fly-out, or add a new fly-out which includes this information

My vote goes for a new flyout for two reasons:

1. The data won't be 'lost' in the huge payload
2. The shipping mechanism is different

In addition, we have to think about the UI for users who don't have sufficient privileges to see the data and when the security solution isn't used at all.

[stevewritescode]

There's also the matter of UI changes in addition to the plugin API. @Bamieh shall I re-open this ticket for that or would you prefer I make a new one?

[stevewritescode]

Re-opening this so we can continue to track the work for enhancing the opt-in / opt-out text in the Kibana UI.

[stevewritescode]

@Bamieh or @TinaHeiligers have you done any codework yet toward the completion of the UI changes described in the ticket? The part about adding a flyout to describe the security telemetry data.

I think either myself or @pjhampton will pick up this work for 7.10 if you don't have time. But I certainly don't want to conflict if you're plans if you've already started it or plan to complete it soon.

[TinaHeiligers]

@stevewritescode We haven't started any work for adding the flyout. It'll be better if your team handles it because you know what data you'll be showing. We'd be guessing at best 😉

[pjhampton]

I'm looking at this on behalf of @elastic/security-data-engineering

[stevewritescode]

Here's some notes on adding that additional flyout. I recommended a few text changes to match the language we use in our product privacy policy page.

Changes to the settings section:

Text for the endpoint security flyout:

[stevewritescode]

Here's a screenshot of the first-run banner, I think that text is ok to stay the same.

[pjhampton]

Sounds good. I will submit a PR soon