Cmak: Kafka Configuration to Enable JMX

[nipunarora]

One of the aspects that Kafka-manager can use is JMX-Polling. What kind of configuration needs to be done in Kafka to enable metrics reporting to Kafka-Manager.

I have a system with multiple agents (kafka producers), which send logs to different topics. I would like to know how many messages are sent by the agents to each topic. I believe kafka-manager fits the bill.

[patelh]

Make sure JMX_PORT env var is exported before you run the script to start kafka broker:
export JMX_PORT=

[aamirfayaz]

in which file inside kafka we have to add export JMX_PORT= property, and while starting kafka-manager it asks for JMX username and password, what should we provide these?? thanks in advance

[tejasmitha]

To enable JMX Monitoring for Kafka broker, please follow below instructions:

Edit kafka-run-class.sh and modify KAFKA_JMX_OPTS variable like below (please replace <> with your Kafka Broker hostname)

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=

Add below line in kafka-server-start.sh

export JMX_PORT=${JMX_PORT:-9999}

[aamirfayaz]

+1

[mluis]

Awesome @tejasmitha that worked for me! Thanks!

[drazenzubovic]

@tejasmitha: yes this is what was missing.

However, I would try to avoid modifying OOTB scripts like kafka-run-class.sh if not necessary. Instead, you can rather export the full KAFKA_JMX_OPTS="..." string into environment, like:
$ export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname= $ export JMX_PORT=
and then start your Kafka

[chandanatalef]

Is it possible to run jmx exporter independently and monitor kafka metrices?
Like, I do not(can't) want to edit my kafka scripts.

[montana-ua]

@chandanatalef you can send all kafka metrics (or any other java based app) via JMX to a metric collection system (e.g. graphite) via jmxtrans-agent (moreover you don't need to enable JMX port on a kafka broker side in this side).

[chandanatalef]

I'm confused. Apologies if I sound too naive.
I have JMX prometheus jar with dependencies, built from
https://github.com/prometheus/jmx_exporter

My kafka broker is running on remote host say, 1.2.3.4:9092
I'm running below command locally:

java -jar jmx_prometheus_httpserver-0.3.2-SNAPSHOT-jar-with-dependencies.jar 7071 kafka.yaml

kafka.yaml is taken from: https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

_I am able to see few metrics on localhost:7071_

# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.519737032
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 1.0
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.3.2-SNAPSHOT",name="jmx_prometheus_httpserver",} 1.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0

Now, I wonder how can I get jmx to know my kafka metrics - I have no where specified kafka broker address in jmx conf.

[aandis]

If https://github.com/yahoo/kafka-manager/issues/187#issuecomment-261467734 doesn't help try appending -Dcom.sun.management.jmxremote.rmi.port=9999 to KAFKA_JMX_OPTS. Worked for me.

[kaisa316]

JMX_PORT=8081 bin/kafka-server-start.sh -daemon config/server.properties > /dev/null 2>&1
可以通过这种方式指定JMX_PORT的值，而且不需要修改kafka-run-class.sh ，已经在本地验证。