Graylog2-server: Support consumer group management in kafka plugin

[jozefbarcin]

Support consumer group management group.id in graylog kafka native plugin

Why?

This feature is necessary if you want to consume data (logs) to more than one consumer (PROD| STAGE|DEV environments), from the same kafka topic.

It is supported by kafka from version 2.0 at least.

Your Environment

• Graylog Version: 3.0.2, 3.2.2
• Elasticsearch Version: 6.8.6
• MongoDB Version: 4.0.13
• Operating System: Ubuntu 18.04
• Browser version: Firefox 73.0.1

[bernd]

@jozefbarcin You can already configure the group.id in the input configuration:

Or do you mean something else?

[jozefbarcin]

@bernd I see, that is cool. I missed that, sorry

Is there possibility to connect input directly to kafka brokers, not via zookeepers
Our case is: only Kafka brokers has IPs, which are available for consumers (graylog). Zookepers are in private IP, they are not accessible for consumers.

Also we are missing Allow ssl communication with kafka broker support

Should I open another issue & close this one?

[jozefbarcin]

I'm closing this issue, created new one for new features: https://github.com/Graylog2/graylog2-server/issues/7617