Cmak: Metrics are always Zero

Created on 4 Apr 2016  路  19Comments  路  Source: yahoo/CMAK

I created a new cluster as follows

image

i specified the zookeeper hosts as per my system(not shown in this image)

I can see

image

then, if i go to a particular topic, i see

image

why are all the metrics zero? this is the case for all topics.

Strangely, in the list view for all the topics

image

i see some valid stats. So, how can the metrics be zero? Does it have something to do with the JMX port?

picture abtpst
👍2

Most helpful comment

I solve the problem now.
In kafka-server-start:
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
export JMX_PORT="9999"
export "KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=falsei -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT -Djava.rmi.server.hostname=xxx.xxx.xxx.xxx(ip) -Djava.net.preferIPv4Stack=true"
And modify iptables and restart iptables

picture sanguinar on 18 Aug 2017
👍8

All 19 comments

Deselect "Display Broker and Topic Size" unless you actually applied the linked patch to Kafka. Check your logs to see if there are any exceptions and post them here.

patelh picture patelh on 4 Apr 2016

how do i 'apply' the patch? i see the file log_segments_dir_jmx_info.patch. where should i put this?

abtpst picture abtpst on 4 Apr 2016

also, do i still have to apply the patch for version 0.9.x.x? on the jira, it says

abtpst picture abtpst on 4 Apr 2016

image

abtpst picture abtpst on 4 Apr 2016

Not sure if it will work with 0.9.x.x. I suggest you de-select that feature for now and just get jmx working without segment size info.

patelh picture patelh on 4 Apr 2016

if i deselect, i see nothing in the topic list view either

abtpst picture abtpst on 4 Apr 2016

Can you check logs and see if there are any exceptions/errors ?

patelh picture patelh on 4 Apr 2016

you mean kafkamanager logs? or my local kafka logs?

abtpst picture abtpst on 4 Apr 2016

kafka manager logs.

patelh picture patelh on 4 Apr 2016

here is what i see in the kafka manager logs

2016-04-04T13:38:13.454-0400[App/0]OUT java.lang.IllegalArgumentException: requirement failed: No jmx port but jmx polling enabled!
2016-04-04T13:38:13.454-0400[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0-internal]
2016-04-04T13:38:13.454-0400[App/0]OUT at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply(BrokerViewCacheActor.scala:355) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.11.7.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.11.7.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at scala.Predef$.require(Predef.scala:219) ~[org.scala-lang.scala-library-2.11.7.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at java.lang.Thread.run(Unknown Source) ~[na:1.8.0-internal]
2016-04-04T13:38:13.454-0400[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0-internal]
2016-04-04T13:38:13.454-0400[App/0]OUT at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply(BrokerViewCacheActor.scala:355) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.11.7.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at scala.Predef$.require(Predef.scala:219) ~[org.scala-lang.scala-library-2.11.7.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0-internal]
2016-04-04T13:38:13.454-0400[App/0]OUT at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply(BrokerViewCacheActor.scala:355) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
2016-04-04T13:38:13.454-0400[App/0]OUT[error] k.m.a.c.BrokerViewCacheActor - Failed to get broker metrics for BrokerIdentity(4,hostName,9092,-1)
2016-04-04T13:38:13.454-0400[App/0]OUT at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply(BrokerViewCacheActor.scala:355) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
2016-04-04T13:38:13.455-0400[App/0]OUT at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.11.7.jar:na]
2016-04-04T13:38:13.455-0400[App/0]OUT at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply(BrokerViewCacheActor.scala:355) ~[kafka-manager.kafka-manager-1.3.0.7-sans-externalized.jar:na]
2016-04-04T13:38:13.455-0400[App/0]OUT at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0-internal]
2016-04-04T13:38:13.455-0400[App/0]OUT at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0-internal]

abtpst picture abtpst on 4 Apr 2016

is this helpful?

abtpst picture abtpst on 4 Apr 2016

Yea, u need to enable JMX on Kafka.

patelh picture patelh on 5 Apr 2016

thanks, i was looking into this. so is JMX enabling just specifying the port? do i have to install any other packages? i tried to set the port in multiple ways but i always get the above error.

abtpst picture abtpst on 5 Apr 2016

you just need to run export JMX_PORT=9999 before you start kafka. Check the quick start guide https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013316

patelh picture patelh on 8 Apr 2016

thanks, that worked

abtpst picture abtpst on 8 Apr 2016

Hi everyone, I set the JMX_PORT=9999 but still have some errors. So my metrics are all zeros

[error] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
    javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_31]
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
    at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.11.8.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.11.8.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
Caused by: javax.naming.CommunicationException: null
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) ~[na:1.8.0_31]
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
    at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
    at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
    javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
    at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) ~[na:1.8.0_31]
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ~[na:1.8.0_31]
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
    at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:980) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
    at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
    at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]

and 

[error] k.m.a.c.BrokerViewCacheActor - Failed to get topic metrics for broker BrokerIdentity(0,localhost,9092,9999,false)
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
    javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_31]
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
    at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na]
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_31]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_31]
Caused by: javax.naming.CommunicationException: null
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) ~[na:1.8.0_31]
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
    at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
    at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
    at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
    javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
    at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) ~[na:1.8.0_31]
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ~[na:1.8.0_31]
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
    at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:980) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
    at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
    at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
Angryrou picture Angryrou on 18 Nov 2016

SAME AS THE ERROR:
[^[[31merror^[[0m] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://xxx.110.8.153:9999/jmxrmi
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[na:1.8.0_121]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_121]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_121]
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) ~[na:1.8.0_121]
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227) ~[na:1.8.0_121]
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179) ~[na:1.8.0_121]
at com.sun.proxy.$Proxy6.newClient(Unknown Source) ~[na:na]
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430) ~[na:1.8.0_121]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) ~[na:1.8.0_121]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_121]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_121]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_121]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_121]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_121]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_121]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_121]
at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_121]
at java.net.Socket.(Socket.java:434) ~[na:1.8.0_121]
at java.net.Socket.(Socket.java:211) ~[na:1.8.0_121]
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_121]
[^[[31merror^[[0m] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://xxx.110.8.153:9999/jmxrmi

sanguinar picture sanguinar on 17 Aug 2017

I solve the problem now.
In kafka-server-start:
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
export JMX_PORT="9999"
export "KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=falsei -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT -Djava.rmi.server.hostname=xxx.xxx.xxx.xxx(ip) -Djava.net.preferIPv4Stack=true"
And modify iptables and restart iptables

sanguinar picture sanguinar on 18 Aug 2017
👍8

Thanks a lot. Resolved my problem.
I was facing two different issues on dev and prod clusters regarding jmx and rmi connectivity from KafkaManager to KafkaBrokers. java.net.ConnectException: Connection timed out and java.net.SocketException: Permission denied (connect failed).

Both got resolved with following addition at broker startup.
Add following in your kafka broker startup script:-
export JMX_PORT="9999" # mention port # you want to use for JMX. Ex. 9093 or 9999,
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"

Note that:- rmi.port is important inside System Property -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT".

As kafka-run-class.sh is adding JMX_PORT to jmxremote.port only. Not to jmxremote.rmi.port. Having rmi.port enabled on broker is important for Kafka Manager to kafka Broker's RMI JMX connectivity. Internally Kafka Mgr try to connect using : service:jmx:rmi:///jndi/rmi://brokerHost:port/jmxrmi.
kafka's bin/kafka-run-class.sh script contains # JMX port to use
if [ $JMX_PORT ]; then
KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

AshokPokharkar picture AshokPokharkar on 17 Oct 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tanuj83 picture tanuj83  路  5Comments
koldrid picture koldrid  路  3Comments
KaimingWan picture KaimingWan  路  9Comments
jmax01 picture jmax01  路  4Comments
mat1010 picture mat1010  路  5Comments