Beats: Metricbeat kafka output not working.

Created on 6 Jun 2018 · 3Comments · Source: elastic/beats

Version: 7.0.0-alpha
Operating System: centos-7

the version 7.0.0 alpha (compiled from master) is not working with kafka output.
I've compuiled the metricbeat and the log output is

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/broker/1 state change to [open] on remote-metricbeat/2

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/broker/1 state change to [open] on remote-metricbeat/8

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/broker/1 state change to [open] on remote-metricbeat/5

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/5 selected broker 1

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/2 selected broker 1

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/2 state change to [flushing-3]

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/2 state change to [normal]

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/8 selected broker 1

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/8 state change to [flushing-3]

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/8 state change to [normal]

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/5 state change to [flushing-3]

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/leader/remote-metricbeat/5 state change to [normal]

2018-06-06T09:42:28.891-0400    INFO    kafka/log.go:36 producer/broker/2 state change to [closing] because EOF

Kafka version is : 0.10.2.1
metricbeat.yml is :

output.kafka:
  hosts: ["kafka.95:9092", "kafka.96:9092", "kafka.97:9092"]
  topic: 'network-metricbeat'
  partition.round_robin:
  reachable_only: false
  required_acks: 1
  compression: none
  max_message_bytes: 1000000

I want to mention that it is working with metricbeat version 6.2.4

Source

marian-craciunescu

Most helpful comment

Add version: 0.10 to output.kafka. In master the new default version is 1.0.0. The wire protocol protocol of kafka is a little different between versions. While the kafka server can deal with old protocol versions, an old kafka can not deal with newer protocol versions. By adding version: 0.10, you tell the output to use an older protocol version.