Cmak: Kafka Manager API timing out

[josephfrancis]

I have deployed kafka manager in several environments, while it works in all non production environments it keeps timing out when we try to access any topic level information in the production environment.

2015-10-21 17:16:46,276 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2 
error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [90000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [90000 ms]
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[org.scala-lang.scala-library-2.11.7.jar:na]
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[org.scala-lang.scala-library-2.11.7.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[org.scala-lang.scala-library-2.11.7.jar:na]
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[com.typesafe.akka.akka-actor_2.11-2.3.10.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]

• I can manually connect to kafka and zookeeper fine from the box where kafka manager is installed. I have also seen kafka manager to work a couple of times without timeouts.
• I'm trying to connect to a 10 node kafka cluster + 5 node zookeeper cluster.
• Kafka manager also gets into a stuck state if I try to update the cluster information.
• Also tried with a timeout of 2 mins as against the default 5 secs, but no joy.

I can't seem to understand why there would be timeouts especially since the kafka manager and kafka nodes are all in the same internal network. Any pointers in finding the root cause?

[josephfrancis]

Enabling the Active Offset Cache fixed the issue for me!
I think there was a lot of data returned causing the timeouts. Still don't understand why increasing the timeout to like 5 or 10 mins wouldn't return any data.
But fixed for now!

[josephfrancis]

[radutanasa]

I'm experiencing the exact same problem and unfortunately enabling the active offsetcache doesn't fix it. Is there another workaround?

[myusuf3]

@radutanasa any luck here?

[1ber]

I'm having the same issue, tried all the proposed solutions without any success, someone have other ideas?
I double checked my zk (telnet to the port in all zk hosts, all opened)
Installed manager in another machine.
The cluster is created i can see it in another machine with manager installed, but when i click on it (in any manager installation) i get the same error:
Yikes! Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager/cluster3/kafka-state)]] after [2000 ms]