Kibana: Feature Request: Ingest pipeline management in Kibana

[danielkasen]

It would be pretty cool if we could control the ingest pipelines in kibana the same way we have centralized mgmt for the logstash pipelines.

[chrisronline]

@ycombinator Any thoughts on this?

[ycombinator]

@danielkasen Just for my clarity, I assume you are using some Filebeat module and would like to visualize the Elasticsearch ingest node pipeline it creates, right?

[danielkasen]

Actually, this is more around duplicating the experience of logstash centralized pipelines with ingest pipelines. So, instead of having to issue an ingest api PUT to update the pipeline I can do it though Kibana.

[ycombinator]

Okay, thanks for the clarification. I'm going to update the issue title to reflect this (by decoupling Filebeat from it).

We did prototype an "ES ingest node pipeline builder and management" project once but it never got shipped. This is something we might want to rebuild but it would depend on how much of a need it is vs. other features in Kibana. We can use this issue for users to express their interest/need in this feature so we can then prioritize it accordingly relative to other features.

[danielkasen]

Ok, sounds good. It's just something that would be nice to allow other users to build the pipeline for their app so I don't have to manage everyone's pipelines.

[timroes]

This issue was brought to my attention, and I think I labelled it wrongly for the Beats team, while it should be under the responsibility of the Elasticsearch UI team. I add both labels for now, please feel free to remove your team label if you think it's def outside the scope of your team.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[P1llus]

After bringing this topic up with Tim, I want to shine some more information on this subject, and why I believe it is something that is worth a second look.

With the introduction and increased focus on SIEM and security, the current ingest pipeline implementation with beats (but also outside beats territory) is causing some unnecessary issues.

What we might see happen with the current implementation

• If beats do not have access to ES directly, the process of importing ingest pipelines or templates can become tedious.

• If we need access to an ingest pipeline or template from winlogbeat but only have linux environments, we would need a separate Windows VM spin up just to get the newest version.

• Managing these deployments on MSSP scenarios with multiple clusters becomes harder, there is no way to synchronise these pipelines between clusters.

• Having similar functionality for Logstash already implemented makes it more confusing on why a similar approach was not taken on beats or ingest pipelines in general.

• If there is a bigger focus on SIEM in the future, the amount of ingest pipelines will grow, and it will end up being harder to manage for the customers.

Possible solutions, in no specific order

• Ingest pipelines and templates from beats should be easily available somewhere online.

• Add the possibility to add/remove/list/manage ingest pipelines from the Kibana UI. And the possibility to enable/disable them.

• Create metrics similar to Logstash pipeline management, showing time taken on each step of the process.

• Move all ingest pipelines and templates to Kibana, so there is never any need for beats to directly communicate with ES. Rather available from a dropdown menu or something similar? Does not have to remove the current functionality built into the beats, but should allow the option to choose between them.

• Possibility to export/import collections of ingest pipelines from the UI, to make it easier to manage between clusters (maybe a possibility through API as well would be nice?)

[cjcenizal]

We shipped an Ingest Node Pipelines app in 7.8, with substantial improvements in 7.9 and 7.10 (including debugging tools). Similarly, Ingest Manager shipped in 7.8 with substantial improvements in subsequent releases, which addresses some of the concerns around Beats asset management.