Cmak: Zookeeper connection error Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]]

Created on 9 Feb 2015  路  12Comments  路  Source: yahoo/CMAK

2015-02-08 16:23:49,016 - [INFO] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7-SendThread(10.65.196.37:2181)
Opening socket connection to server 10.65.196.37/10.65.196.37:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-08 16:23:49,312 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2
Could not read WebJar RequireJS config for: dustjs-linkedin 2.4.0
Please file a bug at: http://github.com/webjars/dustjs-linkedin/issues/new

2015-02-08 16:23:49,354 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2
Could not read WebJar RequireJS config for: json 20121008
Please file a bug at: http://github.com/webjars/json/issues/new

2015-02-08 16:23:49,378 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2
Could not read WebJar RequireJS config for: requirejs 2.1.10
Please file a bug at: http://github.com/webjars/requirejs/issues/new

2015-02-08 16:23:58,829 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2
error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[akka-actor_2.11-2.3.7.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[scala-library-2.11.4.jar:na]
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[akka-actor_2.11-2.3.7.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40-MS]

2015-02-08 16:24:04,065 - [ERROR] - from org.apache.curator.ConnectionState in kafka-manager-system-akka.actor.default-dispatcher-7
Connection timed out for connection string (10.65.196.35:2181,10.65.196.36:2181,10.65.196.37:2181) and timeout (15000) / elapsed (15114)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.7.0.jar:na]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [curator-client-2.7.0.jar:na]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) ~[curator-client-2.7.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:671) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) ~[curator-framework-2.7.0.jar:na]
at kafka.manager.KafkaManagerActor$$anonfun$6.apply(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at kafka.manager.KafkaManagerActor$$anonfun$6.apply(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at scala.util.Try$.apply(Try.scala:191) ~[scala-library-2.11.4.jar:na]
at kafka.manager.KafkaManagerActor.(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_40-MS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_40-MS]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_40-MS]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_40-MS]
at akka.util.Reflect$.instantiate(Reflect.scala:66) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.Props.newActor(Props.scala:252) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.newActor(ActorCell.scala:552) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.create(ActorCell.scala:578) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231) ~[akka-actor_2.11-2.3.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.11.4.jar:na]

picture honghaic

Most helpful comment

For mine this command was working -
bin/kafka-manager -Dconfig.file=conf/application.conf -Dkafka-manager.zkhosts="localhost:2181"

make sure , u are using both parameter -Dconfig.file and -Dkafka-manager.zkhost .

If I was trying with individual parameter , it was giving same error .
Try with use of both parameter .

picture vijaygupta358 on 19 Jan 2017
👍4

All 12 comments

2015-02-08 16:40:44,135 - [INFO] - from org.apache.zookeeper.ZooKeeper in kafka-manager-system-akka.actor.default-dispatcher-7
Client environment:user.dir=D:\app\KafkaManager.A\siphon

2015-02-08 16:40:44,137 - [INFO] - from org.apache.zookeeper.ZooKeeper in kafka-manager-system-akka.actor.default-dispatcher-7
Initiating client connection, connectString=10.65.196.35:2181,10.65.196.36:2181,10.65.196.37:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4c0dfaf6

2015-02-08 16:40:44,142 - [DEBUG] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7
zookeeper.disableAutoWatchReset is false

2015-02-08 16:40:44,159 - [INFO] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7-SendThread(10.65.196.36:2181)
Opening socket connection to server 10.65.196.36/10.65.196.36:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-08 16:40:45,054 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2
error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[akka-actor_2.11-2.3.7.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[scala-library-2.11.4.jar:na]
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[akka-actor_2.11-2.3.7.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40-MS]

2015-02-08 16:40:45,395 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6
Could not read WebJar RequireJS config for: dustjs-linkedin 2.4.0
Please file a bug at: http://github.com/webjars/dustjs-linkedin/issues/new

2015-02-08 16:40:45,431 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6
Could not read WebJar RequireJS config for: json 20121008
Please file a bug at: http://github.com/webjars/json/issues/new

2015-02-08 16:40:45,454 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6
Could not read WebJar RequireJS config for: requirejs 2.1.10
Please file a bug at: http://github.com/webjars/requirejs/issues/new

honghaic picture honghaic on 9 Feb 2015

you can use the comand as below:
kafka-manager -Dkafka-manager.zkhosts="server-01:2181".
the shell can't load the rigth config file.

rycoxiao picture rycoxiao on 9 Feb 2015

Unfortunately I confirm this problem I assume it's because of incorrect Play/sbt-native-packager settings in kafka-manager.

How to reproduce

A kafka-manager packaged and deployed via a zipfile (./sbt clean dist) is apparently not loading conf/application.conf after a deployment.

$ ./sbt clean dist    # >>> creates a zipfile
$ unzip kafka-manager-1.0-SNAPSHOT.zip
$ cd kafka-manager-1.0-SNAPSHOT 

# Now modify `conf/application.conf` to e.g. configure your ZK connection string via the
# config setting `kafka-manager.zkhosts`.  Say, set `kafka-manager.zkhosts="myzookeeper:2181".
$ vi conf/application.conf

Now, when you launch kafka-manager without any additional CLI options you will run into the following error:

$ bin/kafka-manager
Play server process ID is 57526
[info] play - Application started (Prod)
[info] play - Listening for HTTP on /0.0.0.0:9000
[INFO] [02/10/2015 10:13:53.250] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] Starting curator...
[error] o.a.c.f.i.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.net.UnknownHostException: kafka-manager-zookeeper: kafka-manager-zookeeper
        at java.net.InetAddress.getAllByName0(InetAddress.java:1348) ~[na:1.7.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1257) ~[na:1.7.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1182) ~[na:1.7.0]
        at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[org.apache.zookeeper.zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) ~[org.apache.zookeeper.zookeeper-3.4.6.jar:3.4.6-1569965]
[INFO] [02/10/2015 10:13:53.275] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] zk=kafka-manager-zookeeper:2181
[INFO] [02/10/2015 10:13:53.275] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] baseZkPath=/kafka-manager
[error] k.m.ApiError - error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[com.typesafe.akka.akka-actor_2.11-2.3.7.jar:na]
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[com.typesafe.akka.akka-actor_2.11-2.3.7.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[org.scala-lang.scala-library-2.11.4.jar:na]
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[org.scala-lang.scala-library-2.11.4.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[org.scala-lang.scala-library-2.11.4.jar:na]
[error] o.a.c.ConnectionState - Connection timed out for connection string (kafka-manager-zookeeper:2181) and timeout (15000) / elapsed (15053)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
        at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) [org.apache.curator.curator-framework-2.7.0.jar:na]
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) [org.apache.curator.curator-framework-2.7.0.jar:na]

Note the java.net.UnknownHostException: kafka-manager-zookeeper error: kafka-manager-zookeeper is the default host for kafka-manager.zkhosts. So kafka-manager is not reading the app configuration file, because if it had, it would have used myzookeeper as the ZK host.

Workaround

Explicitly provide the location of the application.conf file by setting the config.file System property on the command line when launching kafka-manager:

$ bin/kafka-manager -Dconfig.file=/path/to/kafka-manager/conf/application.conf
miguno picture miguno on 10 Feb 2015
👍3

+1 for the work around.

mindscratch picture mindscratch on 18 Feb 2015

I'm also running into this problem on both ubuntu 14.04 and mac os x. Running the latest commit as of right now.

a9fbba0cc9cb7e68f55e8a1b678b5b5437d6d4d6

Specifying the config directory is the only way I can get this to work.

spuder picture spuder on 2 Mar 2015

Duplicate of #11

spuder picture spuder on 9 Mar 2015

i download the newest version of kafka-manager. and try to start it by the command above which said by miguno. but I still meet the error of load conf file error.

only I set the zookeeper in the command : -Dkafka-manager-zkhosts="127.0.0.1:2181".

I can start the kafka-manager in a right way now .

so I think this issue had no been fixed yet.

lewison picture lewison on 17 Aug 2015

@lewison Same problem, do you know where the default zk urls was stored when sbt clean dist

zheolong picture zheolong on 11 Nov 2015

i hit error "Bad application path: -Dconfig.file=../config/application.conf" when i use command line to start it. help~

guangping0215 picture guangping0215 on 11 Apr 2016
😕3

I have just started to see this today. Sometimes the page renders, sometimes I get this error.

memelet picture memelet on 16 Jul 2016

For mine this command was working -
bin/kafka-manager -Dconfig.file=conf/application.conf -Dkafka-manager.zkhosts="localhost:2181"

make sure , u are using both parameter -Dconfig.file and -Dkafka-manager.zkhost .

If I was trying with individual parameter , it was giving same error .
Try with use of both parameter .

vijaygupta358 picture vijaygupta358 on 19 Jan 2017
👍4

I am encountering the same issue Yikes! Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager/prod/kafka-state)]] after [2000 ms]

I am running the kafka-manager on a node where i can confirm that i can telnet into the ZK on 2181.

I am running with
/home/ubuntu/kafka-manager-1.3.3.13/bin/kafka-manager -Dconfig.file=/home/ubuntu/kafka-manager-1.3.3.13/conf/application.conf -Dhttp.port=9292 -Dkafka-manager.zkhosts=kafka1:2181,kafka2:2181,kafka3:2181,kafka4:2181 > /var/log/kafka-manager.log 2>/home/ubuntu/kafka-manager-1.3.3.13/error.log &

When i load the test cluster it's fine. Help needed please.

dinar-dalvi picture dinar-dalvi on 9 Sep 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Jonathan-Wei picture Jonathan-Wei  路  3Comments
tanuj83 picture tanuj83  路  5Comments
dmead picture dmead  路  9Comments
KaimingWan picture KaimingWan  路  9Comments
sanjeevtripurari picture sanjeevtripurari  路  4Comments