Lagom: Consumer interrupted with WakeupException after timeout.

[sarathraj-coder]

sbt runAll
[info] Loading project definition from /Users/sarathraj/lagom/online-auction-scala/project
[info] Set current project to online-auction-scala (in build file:/Users/sarathraj/lagom/online-auction-scala/)
[info] Starting Kafka
[info] Starting Cassandra
.SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
............Kafka Server closed unexpectedly.
..
[info] Cassandra server running at 127.0.0.1:4000
[info] Service locator is running at http://localhost:8000
[info] Service gateway is running at http://localhost:9000
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[info] Service itemImpl listening for HTTP on 0:0:0:0:0:0:0:0:53203
[info] Service webGateway listening for HTTP on 0:0:0:0:0:0:0:0:60496
[info] Service searchImpl listening for HTTP on 0:0:0:0:0:0:0:0:55320
[info] Service userImpl listening for HTTP on 0:0:0:0:0:0:0:0:56181
[info] Service biddingImpl listening for HTTP on 0:0:0:0:0:0:0:0:58427
[info] (Services started, press enter to stop and go back to the console...)
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[error] a.k.KafkaConsumerActor - WakeupException limit exceeded, stopping.
[error] c.l.l.i.b.k.KafkaSubscriberActor - Topic subscription interrupted due to failure WARNING arguments left: 1
[error] a.a.OneForOneStrategy - Consumer actor terminated
java.lang.Exception: Consumer actor terminated
at akka.kafka.internal.SingleSourceLogic$$anonfun$preStart$1.apply(SingleSourceLogic.scala:54) ~[akka-stream-kafka_2.11-0.13.jar:0.13]
at akka.kafka.internal.SingleSourceLogic$$anonfun$preStart$1.apply(SingleSourceLogic.scala:40) ~[akka-stream-kafka_2.11-0.13.jar:0.13]
at akka.stream.stage.GraphStageLogic$StageActor.internalReceive(GraphStage.scala:166) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.stage.GraphStageLogic$StageActor$$anonfun$1.apply(GraphStage.scala:133) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.stage.GraphStageLogic$StageActor$$anonfun$1.apply(GraphStage.scala:133) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:691) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:419) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:497) ~[akka-actor_2.11-2.4.17.jar:na]
[error] a.k.KafkaConsumerActor - WakeupException limit exceeded, stopping.
[error] c.l.l.i.b.k.KafkaSubscriberActor - Topic subscription interrupted due to failure WARNING arguments left: 1
[error] a.a.OneForOneStrategy - Consumer actor terminated
java.lang.Exception: Consumer actor terminated
at akka.kafka.internal.SingleSourceLogic$$anonfun$preStart$1.apply(SingleSourceLogic.scala:54) ~[akka-stream-kafka_2.11-0.13.jar:0.13]
at akka.kafka.internal.SingleSourceLogic$$anonfun$preStart$1.apply(SingleSourceLogic.scala:40) ~[akka-stream-kafka_2.11-0.13.jar:0.13]
at akka.stream.stage.GraphStageLogic$StageActor.internalReceive(GraphStage.scala:166) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.stage.GraphStageLogic$StageActor$$anonfun$1.apply(GraphStage.scala:133) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.stage.GraphStageLogic$StageActor$$anonfun$1.apply(GraphStage.scala:133) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:691) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:419) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618) ~[akka-stream_2.11-2.4.17.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:497) ~[akka-actor_2.11-2.4.17.jar:na]

[edouardKaiser]

............Kafka Server closed unexpectedly.

Your Kafka server didn't start, that's why you experience this behaviour. You can check the logs in./target/lagom-dynamic-projects/lagom-internal-meta-project-kafka/target/log4j_output. Might give you an idea.

[TimMoore]

@sarathraj-coder were you able to find more information in the logs? Feel free to share them here if you'd like someone else to look at them.

Another thing you can try is running sbt clean runAll to remove temporary files. We have seen some instances of Kafka unable to start up because of corrupt data in ZooKeeper. Cleaning these files usually fixes the problem. Note that you'll lose all data you created in development when you do this.

[sarathraj-coder]

issue solved by sbt clean runAll

[franz-see]

@TimMoore what if the error happens in production? How do you clean zookeeper?

[ignasi35]

@franz-see in production, that situation is unlikely to happen since ZooKeeper as it is a key piece of the infrastructure that should never be stopped.

This problem may happen in Lagom's DevMode because it's common to start/stop/start/stop/etc... sbt runAll a few times a week (if not daily). When this error happens in dev mode, there's no production-critical information on the ZooKeeper records so it's not a huge problem to wipe the data folder.

If this error happened in production then some ZooKeeper specific maintenance and recovery may be necessary.