Fluent-bit: in_kafka plugin

[cainelli]

Hi all, I was wondering if there is any plans to support input Kafka as done with out_kafka?

[ymartincds]

Is it for technical reason that there is out and not in Kafka binding ?

Regards

[kinghuang]

Now that fluent-but 1.1 has stream processing, I’m super interested in the ability to use Kafka topic(s) as input. It’d be a good alternative to KSQL for certain tasks.

At the moment, I’m using kafkacat (also built on librdkafka) to consume messages and pipe them to fluent-bit.

[marcosflobo]

Long time since this thread was opened. Any news on this? Is there any plan to support Kafka as input for FluentBit?

[tarunwadhwa13]

We too forward raw logs and use logstash for parsing and storing to ES. Since Logstash consumes too much resources, planning to move to fluent bit for this. But unfortunately, couldn't find any In Kafka support. Is there any workaround to achieve this?

[edsiper]

Yes, it will be implemented during Q3-Q4 this year.

[tarunwadhwa13]

Gentle Reminder !!
Can we add this to 1.6 milestone?

[edsiper]

good idea. Added.

[tarunwadhwa13]

@edsiper We are eagerly waiting for this :smile: Is there expected date for 1.6 release?

[edsiper]

Moved to v1.7 release (Feb 2021)