Graylog2-server: Create Global Consumer Group for Kafka Input in Multi-node Setups

[jalogisch]

Expected Behavior

If you have a multi node Graylog Setup and configure a global Kafka Input all Graylog Nodes should build a Global/Consumer Group to be able to consume messages with all nodes from a Kafka queue that is configured with Partitions.

References: https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/

Current Behavior

One node consumes all messages from the Kafka queue and the messages are not balanced by partition.

Steps to Reproduce (for bugs)

1. Have Graylog Multi Node Environment
2. Create Global Kafka Input
3. Have a Kafka Queue with the same number of Partitions as your Graylog Nodes
4. Messages are consumed only by one Node

Context

If your run Kafka as some kind of Buffer, having Partitions the same amount as you have Graylog Nodes the assumption is that each Graylog Node is getting the messages from one Partition.

Your Environment

• Graylog Version: 2.2.3

[landonix]

This issue can be resolved by using a Kafka topic with four (4) partitions. Each Graylog node will then enter the Kafka Consumer Group (assuming that 2 processors are selected, default). If changes are made to an existing system, the Kafka Producer will need restarted to recognize the additional partitions are available. After that, both Graylog nodes will ingest in parallel.

[flavioleggio]

I had the same issue and @landonix answer was the correct solution. This should be closed.

[iankoetter]

@landonix but if one of the graylog nodes goes down, doesn't the associated kafka partition(queue) fill up?