Kibana: [Ingest Manager] Provide the ability to register a callback for creation and/or editing of Endpoint Datasources

[paul-tavares]

Describe the feature:
The Endpoint security functionality need to be able to intercept and possibly manipulate the Endpoint Datasource at creation and/or editing time.

Describe a specific use case for the feature:
Endpoint requires that some additional custom data be provided along with the Policy configuration that are specific to the Endpoint executable. This data is not user controlled and is generated at runtime in kibana, thus it can not be included in the endpoint package definition. The Datasource is currently used to include and deliver the Endpoint Policy data and we would like to use that same delivery mechanism and include/inject this data into it.

Automated Test Cases on the Ingest Side To Be Documented:

• [ ] Validate when create api handler and external callbacks are registered that it should call external callbacks in expected order
• [ ] Validate when create api handler and external callbacks are registered that it should feed datasource returned by last callback
• [ ] Validate when create api handler and external callbacks are registered that it should create with data from callback
• [ ] Validate when create api handler and external callbacks are registered and it throws an exception error that it should skip over callback exceptions and still execute other callbacks
• [ ] Validate when create api handler and external callbacks are registered and it throws an exception error that it should log errors
• [ ] Validate when create api handler and external callbacks are registered and it throws an exception error that it should create datasource with last successful returned datasource

Test Case:

• [ ] Validate when a policy is created the policy information is included in the response from the Package Config API (C15633 - manual)

[elasticmachine]

Pinging @elastic/endpoint-management (Team:Endpoint Management)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/endpoint-response (Team:Endpoint Response)

[elasticmachine]

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

[kevinlog]

We need:

1. Callback from Ingest
2. Trigger from our side to alert that manifest has changed (artifact version increase)

[ph]

@jen-huang Can you take a look at this, from a discussion with @ruflin I think you already had an idea for managing this.

[paul-tavares]

Team,
Here is what I'm thinking for design around this issue:

Register Callbacks

1. Change the server plugin to expose a register(type: string, callback: function) method. This method will take in two arguments - the first being the type of callback being registered. For this initial implementation, the supported type will be datasourceCreate.
   _NOTE: I initially thought that we also needed one for Edit datasource, but after thinking about the use case in Endpoint a little more, I no longer see that need_
2. Store a Map<String, Set<Function>> in the Plugin instance with all callbacks. The map's string key will equal the callback type defined in the registration
3. Change the Datasource Create Route to check if callbacks are registered for the create action, and call those in sequence (await'ing their response) with the NewDatasource input before calling the service that actually creates them. Each callback will receive the NewDatasource from the prior one in the list.
4. The callback for the Create Datasource will be provided the NewDatasource object as input, and is expected to return a Promise that resolves with a NewDatasource object

Q. How should we handle failures in the callbacks?
Should we:

• cause the REST API call to fail on first callback failure?
• Ignore the Failures and log it - but still allow the create of the Datasource using the last good returned Datasource object (which could be the one provided on input to the REST route)?

Expose Datasource services to Server Plugins

The Endpoint plugin will need to make changes to datasources whenever certain endpoint specific artifacts are changed. In order to support those use cases:

1. Change the server plugin to expose methods out of the Datasource service, so that other plugins can use them in their processes. Currently, I think Endpoint only needs Datasource#list() and Datasource#update()

[paul-tavares]

Another Question: Should the register() method return an unregister() callback itself?
In endpoint, I can see us needing it, but it may be useful to ensure external concerns have a way to either unregister themselves or clean up during a shutdown (.stop())

[jen-huang]

Change the Datasource Create Route to check if callbacks are registered for the create action, and call those in sequence (await'ing their response) with the NewDatasource input before calling the service that actually creates them. Each callback will receive the NewDatasource from the prior one in the list.

Can you elaborate on why we should execute the callbacks prior to creating the data source saved object itself? I was thinking that we would create the data source SO first, then pass the SO to each callback.

I was also thinking that the callback executions should happen at datasource service level, in the create() and bulkCreate() methods (instead of at the route handler level).

How should we handle failures in the callbacks?

My preference is the second option - ignore & log it. This is even safer if we create the data source SO prior to callbacks.

Change the server plugin to expose methods out of the Datasource service

👍 I wouldn't mind just exposing the entire data source service (not just a few methods). It was always our intention that the services would be consumed by other plugins.

Should the register() method return an unregister() callback itself?

No strong preference on my end, but I'm not sure what the Endpoint use case is, could you elaborate on the type of cleanup actions you might need?

[paul-tavares]

Thanks @jen-huang for the review/feedback.

Re:

Can you elaborate on why we should execute the callbacks prior to creating the data source saved object itself? I was thinking that we would create the data source SO first, then pass the SO to each callback.

We're trying to avoid Endpoint (the .exe) failures for the cases where the Endpoint integration is added to an existing Agent Configuration that has already Agents enrolled. Since the creation of the Datasource triggers an update to the Agent Configuration and thus delivers the update to the enrolled agents, we would be delivering an incomplete endpoint configuration - thus why we would need to intercept the payload before its actually created

Re:

I was also thinking that the callback executions should happen at datasource service level, in the create() and bulkCreate() methods (instead of at the route handler level).

We can do that, but I wonder if that will actually limit us or make the code more complicated and fragile. If the service is being used by other apps (as Endpoint will), having the service invoke the external callbacks could potentially cause an infinite circular loop. We would have to put in place code that detected such loops or it would place a larger burden on the logic inside of the callbacks that are registered.

Re:

...could you elaborate on the type of cleanup actions you might need?

Endpoint does not have need for it at the moment (that I can see). I just wanted to ask the question to see if anyone saw a need for it. I can leave that off for now.

Thanks again for the feedback. I should have some code (PR) up for review soon on this 😄