Cmak: Broker Metrics not being shown up in the UI

[suhas22]

Hi Team,
First of all thanks for putting up a great application, it's really useful to understand what's going with 100 odd brokers that we have in our cluster :)

I am however facing a few issues while trying to view the broker metrics.
1) I have setup the latest version of kafka manager, i.e 1.3.0.4.
2) Was able to add our prod zk/brokers successfully.
3) But when i try to view the broker information such as combined metrics, nothing shows up, but i go to the topics page, i can view the different topics available and the size of the data coming in.

Brokers home page:

Topics home page :

I browsed through a couple of issues related to this, and it was said that the issue was resolved in the master build, but looks like it isn't.(https://github.com/yahoo/kafka-manager/issues/69)
Any help here would be greatly appreciated.

[rphillips]

I ran across the same issue with a 3 node cluster.

[patelh]

Do you have JMX enabled for kafka brokers? If so, did you enable JMX polling in the configuration of the cluster in Kafka Manager?

[rphillips]

I do not have JMX enabled. Is this a requirement?

[suhas22]

Yes, I have JMX enabled for the brokers and also have enabled jmx polling
while configuring the cluster on kafka manager.

On Sun, Feb 14, 2016 at 6:34 AM, Ryan Phillips [email protected]
wrote:

I do not have JMX enabled. Is this a requirement?

—
Reply to this email directly or view it on GitHub
https://github.com/yahoo/kafka-manager/issues/186#issuecomment-183783393
.

[patelh]

@rphillips Yes, you need to enable JMX for metrics. No other way right now.

@suhas22 What version of Kafka?

[suhas22]

Hi,
I am using Kafka 0.8.2

On Mon, Feb 15, 2016 at 10:14 AM, patelh [email protected] wrote:

@rphillips https://github.com/rphillips Yes, you need to enable JMX for
metrics. No other way right now.

@suhas22 https://github.com/suhas22 What version of Kafka?

—
Reply to this email directly or view it on GitHub
https://github.com/yahoo/kafka-manager/issues/186#issuecomment-184064685
.

[patelh]

@suhas22 Do you see any errors in the logs? Are you using 0.8.2.0, 0.8.2.1, or 0.8.2.2 ?

[suhas22]

Hi,
Sorry, It was a typo while mentioning the version, I am currently on
0.8.2.1.
I don't see any specific errors to this issue.
Part information is see though like Topic related information is seen such
as offsets, leaders etc, but the metrics is not be seen as a whole.

On Tue, Feb 16, 2016 at 5:52 AM, patelh [email protected] wrote:

@suhas22 https://github.com/suhas22 Do you see any errors in the logs?
Are you using 0.8.2.0, 0.8.2.1, or 0.8.2.2 ?

—
Reply to this email directly or view it on GitHub
https://github.com/yahoo/kafka-manager/issues/186#issuecomment-184450556
.

[patelh]

Are you using Oracle JDK8 or OpenJDK8 ?

[suhas22]

Oracle JDK8

[patelh]

Hm, it is odd that you don't see any exceptions related to connecting to JMX or something similar. Are you able to connect to the broker and see the metrics using JConsole ? Did you configure a custom metrics reporter in your kafka server.properties ?

[suhas22]

I have a distributed system setup all on cloud, am running kafka manager on
Zookeeper cluster where i have oracle java 8 installed, but my brokers are
on java 7.
Jconsole just doesn't start on the zookeeper.
I did not understand this statement"Did you configure a custom metrics
reporter in your kafka server.properties" What do we need to do to
configure custom metrics reporter ?

When i access the kafka manager UI, it lists all the brokers that the
zookeepers are connected to and gives the details of all the topics like
offsets, partitions etc but gives nothing for brokers.
So is the java version a issue here ?

On Wed, Feb 17, 2016 at 12:57 PM, patelh [email protected] wrote:

Hm, it is odd that you don't see any exceptions related to connecting to
JMX or something similar. Are you able to connect to the broker and see the
metrics using JConsole ? Did you configure a custom metrics reporter in
your kafka server.properties ?

—
Reply to this email directly or view it on GitHub
https://github.com/yahoo/kafka-manager/issues/186#issuecomment-185073022
.

[suhas22]

Hi,
I did a brand new setup, and saw that we have a new version now, but again I am seeing the same issue .
Here are the errors that's showing up now , it says connection refused to port 9997.
But I have jmx port 9997 opened up and Kafka is listening at this port.

[error] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://kafka-2-broker-24:9997/jmxrmi
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[na:1.8.0_74]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_74]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_74]
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) ~[na:1.8.0_74]
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) ~[na:1.8.0_74]
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) ~[na:1.8.0_74]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) ~[na:1.8.0_74]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_74]
at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:50) ~[kafka-manager.kafka-manager-1.3.0.4-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerTopicPartitionsSize$1$$anonfun$apply$29$$anonfun$apply$4.apply$mcV$sp(BrokerViewCacheActor.scala:382) [kafka-manager.kafka-manager-1.3.0.4-sans-externalized.jar:na]
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_74]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_74]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_74]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_74]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_74]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_74]
at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_74]
at java.net.Socket.(Socket.java:434) ~[na:1.8.0_74]
at java.net.Socket.(Socket.java:211) ~[na:1.8.0_74]
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_74]

And here is the screenshot of my kafka process and port 9997 opened up and kafka is listening to it.

[suhas22]

The more I see the error, I get to think it's due to the Java version difference.
I have my broker's and ZK's running on java 7, where as just for Kakfa manager I installed Java 8, as it won't install with Java 7.
And the error also points to java 1.8 RMI connector stuff, Is my assumption correct ?

[xavierpayne]

I had a similar problem. The cause is the initial port 9997 is just the port that starts the connection. After the initial connection JMX negotiates a random port and uses that for the rest of the communication.

This article explains it better than I can:
https://olegz.wordpress.com/2009/03/23/jmx-connectivity-through-the-firewall/

Unfortunately the solution is to open the entire range that it picks it's random port numbers from.

[keithjpaulson]

JDK7_45(?) and later should support -Dcom.sun.management.jmxremote.rmi.port=xxxx so the RMI port is not randomly chosen (I have not yet tried this).

[krishna81m]

Same here, as a newbie, setup a Open JDK 1.7.0_101 Kafka 0.9.0.1 cluster and started kafka manager with JMX enabled as I can see kafka-manager picked up the JMX port but the bytes is 0 everywere.
kafka 18040 1 0 06:59 pts/4 00:00:50 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/kafka/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka.broker.hostname -Dcom.sun.management.jmxremote.port=6980 -Dkafka.logs.dir=/home/kafka/kafka/bin/../logs -Dlog4j.configuration=file:/home/kafka/kafka/bin/../config/log4j.properties -cp :/home/kafka/kafka/bin/../libs/* kafka.Kafka /home/kafka/kafka/config/server.properties

tcp6 0 0 :::6980 :::* LISTEN

Ok, found a problem, although the kafka-manager was showing the JMX_PORT does not necessarily mean it can connect and get metrics, I verified this by trying to connect manually from JVisualVM. I had to make this fix to get JMX working on JVisualVM.

~/kafka/bin/kafka-run-class.sh
# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<actual host address>"
fi

[1ambda]

the JMX server which is running on broker will open the RMI port, you might need to specify RMI port also if firewall is your concern (-Dcom.sun.management.jmxremote.rmi.port=10000) otherwise, JMX server will use random port

For example,

JMX_PORT=9999 KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=10000 -Djava.rmi.server.hostname=$HOSTNAME" ./bin/kafka-server-start.sh -daemon ./config/server.properties

[shades198]

adding -Djava.rmi.server.hostname option didn't change anything. It seems as if this option is not taking any effect. Please help. I also have the same scenario wherein I have 3 brokers and kafka-manager is able to read metrics only from the server on which it is running.

[jimymodi]

@shades198 Did you found a fix around it ?

[thammuio]

I have the same issue https://github.com/yahoo/kafka-manager/issues/571 ; not sure why it is not working....

[TarikChrouki]

@shades198, @jimymodi what do you put in KAFKA_JMX_OPTS ?

[funkol2007]

I enabled KAFKA_JMX_OPTS and the settings are followings:
JMX_PORT='23311'
KAFKA_JMX_OPTS='-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=23311'

But I still can't get Combined Metrics