Logstash: Logstash Centralized pipeline Manage pipelines via Gui

[toddferg]

I propose an enhancement to the Logstash centralized pipeline management. Currently the method for enabling a pipeline is to add it to the logstash.yml file and restart the logstash instance (or reload the configuration).

Would it be possible to start assigning the pipelines via the Hostname (or unique identifier configured in the yml) and logstash will automatically pickup the new pipelines or accept the configuration changes.

Proposal:

In logstash.yml
xpack.management.pipeline.logstash.id (This is the unique setting for that instance or "set of instances"

In the .logstash index we would have a new field listed as logstash.id which the logstash would use to see the available configurations assigned to it.

OR

Utilize the hostname of the box, and be able to assign different pipelines that way.

I feel this allows for a more cohesive and centered management approach to the centralized pipeline management.

[toddferg]

Might be related: https://github.com/elastic/kibana/issues/18155

[C0FFEEC0FFEE]

We would also greatly benefit from this feature since we're adding new pipelines on a regular basis.

Since we're also regularly deploying new Logstash nodes I suggest implementing it in a way that one can easily add new Logstash nodes without having to add them to each pipeline individually (i.e. use option 1 / set of instances or extend the GUI to assign new Logstash instances to a set of pipelines).