Marathon: Excess threads spawned due to blocking code.
Marathon threads count is O(n) where n is number of applications with health check. This decrease app performance and prevent using marathon at scale.
Why
Marathon combines akka actors and object oriented services and managers. Unfortunately whenever blocking request is called by the actor new thread is spawned. Read more
Where
Every Await, blocking, lock called inside the actor is dangerous and should be re-implemented in asynchronous manner (returns future or callback message).
Dangerous examples that creates new thread:
mesosphere.marathon.core.task.tracker.impl.InstanceTrackerDelegate#instancesBySpecSyncmesosphere.marathon.core.launchqueue.impl.LaunchQueueDelegate#askQueueActormesosphere.marathon.core.launchqueue.impl.LaunchQueueDelegate#askQueueActorFuturemesosphere.util.RWLock#withLock
Above examples were found byt analysing stack traces of akka.dispatch.MonitorableThreadFactory#newThread
Hotfix
- Do NOT use HTTP and TCP health-checks use Mesos command health-check instead
- Do NOT spawn multiple apps at once (use rate limiter or somehow delay concurrent requests)
This will not fix the issue but minimize threads count.
Example:
Change setting to create at most one thread. Run marathon and schedule app or two. Create thread dump and check akka threads count.
Config:
diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf
index 2b4d4bf..d8c2547 100644
--- a/src/main/resources/application.conf
+++ b/src/main/resources/application.conf
@@ -2,4 +2,8 @@ akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
+ actor.default-dispatcher.fork-join-executor {
+ parallelism-min = 1
+ parallelism-max = 1
+ }
}
diff --git a/src/main/scala/mesosphere/marathon/Main.scala b/src/main/scala/mesosphere/marathon/Main.scala
index ab1607f..1d8e95c 100644
--- a/src/main/scala/mesosphere/marathon/Main.scala
+++ b/src/main/scala/mesosphere/marathon/Main.scala
@@ -140,9 +140,9 @@ class MarathonApp(args: Seq[String]) extends AutoCloseable with StrictLogging {
}
}
- setIfNotDefined("scala.concurrent.context.minThreads", "5")
- setIfNotDefined("scala.concurrent.context.numThreads", "x2")
- setIfNotDefined("scala.concurrent.context.maxThreads", "64")
+ setIfNotDefined("scala.concurrent.context.minThreads", "1")
+ setIfNotDefined("scala.concurrent.context.numThreads", "x1")
+ setIfNotDefined("scala.concurrent.context.maxThreads", "1")
}
}
ThreadDump:
2016-11-23 15:31:09
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode):
"ForkJoinPool-2-worker-5" #122 daemon prio=5 os_prio=0 tid=0x00007f1768037800 nid=0x42f2 waiting on condition [0x00007f1757dfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5b6cdb0> (a scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"marathon-akka.actor.default-dispatcher-18" #121 prio=5 os_prio=0 tid=0x00007f17b404d800 nid=0x42f1 waiting on condition [0x00007f17c4b8a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5b6cd10> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"qtp2121400959-89" #89 prio=5 os_prio=0 tid=0x00007f178415a000 nid=0x42a8 runnable [0x00007f17c4887000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005d2f2dd80> (a sun.nio.ch.Util$2)
- locked <0x00000005d2f2dd90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005d2f2dd38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:232)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:180)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:162)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-88" #88 prio=5 os_prio=0 tid=0x00007f1784159800 nid=0x42a7 waiting on condition [0x00007f175d4f9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Scheduler-933290069" #84 prio=5 os_prio=0 tid=0x00007f17a47b4000 nid=0x42a2 waiting on condition [0x00007f175c4f1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005d2d57260> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Timer-0" #79 daemon prio=5 os_prio=0 tid=0x00007f17c004c000 nid=0x4288 in Object.wait() [0x00007f17571f2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000005e5f6ac20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"pool-8-thread-1" #78 prio=5 os_prio=0 tid=0x00007f17d8026000 nid=0x4287 waiting on condition [0x00007f17572f3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5f6ac38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-Framework-0" #76 daemon prio=5 os_prio=0 tid=0x00007f17c001f800 nid=0x4280 waiting on condition [0x00007f17573f4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5f6ac80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:804)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005e5c71d50> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"ForkJoinPool-2-worker-1-EventThread" #75 daemon prio=5 os_prio=0 tid=0x00007f17c001e000 nid=0x427f waiting on condition [0x00007f17574f5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5f6acc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:511)
Locked ownable synchronizers:
- None
"ForkJoinPool-2-worker-1-SendThread(10.10.10.10:2181)" #74 daemon prio=5 os_prio=0 tid=0x00007f17c000e000 nid=0x427e runnable [0x00007f17575f6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005e5f6c0a8> (a sun.nio.ch.Util$2)
- locked <0x00000005e5f6c090> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005e5d25830> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
Locked ownable synchronizers:
- None
"Curator-ConnectionStateManager-0" #73 daemon prio=5 os_prio=0 tid=0x00007f17c000d800 nid=0x427d waiting on condition [0x00007f17576f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4440> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005e5af16a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"ForkJoinPool-2-worker-1" #72 daemon prio=5 os_prio=0 tid=0x00007f1770001800 nid=0x427c runnable [0x00007f17577f7000]
java.lang.Thread.State: RUNNABLE
at org.apache.mesos.MesosSchedulerDriver.join(Native Method)
at org.apache.mesos.MesosSchedulerDriver.run(MesosSchedulerDriver.java:255)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(MarathonSchedulerService.scala:235)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(MarathonSchedulerService.scala:235)
at scala.Option.foreach(Option.scala:257)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(MarathonSchedulerService.scala:235)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1$$anonfun$apply$mcV$sp$1.apply(MarathonSchedulerService.scala:235)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1$$anonfun$apply$mcV$sp$1.apply(MarathonSchedulerService.scala:235)
at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$2$$anon$4.block(ExecutionContextImpl.scala:48)
at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$2.blockOn(ExecutionContextImpl.scala:45)
at scala.concurrent.package$.blocking(package.scala:123)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1.apply$mcV$sp(MarathonSchedulerService.scala:235)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1.apply(MarathonSchedulerService.scala:234)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$startLeadership$1.apply(MarathonSchedulerService.scala:234)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"qtp2121400959-71" #71 prio=5 os_prio=0 tid=0x00007f179c014000 nid=0x427b waiting on condition [0x00007f1757af9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-70" #70 prio=5 os_prio=0 tid=0x00007f179c012800 nid=0x427a runnable [0x00007f1757bfa000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005d3104bf8> (a sun.nio.ch.Util$2)
- locked <0x00000005d3104c08> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005d3104bb0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:232)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:180)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:162)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-69" #69 prio=5 os_prio=0 tid=0x00007f179c010800 nid=0x4279 waiting on condition [0x00007f1757cfb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-67" #67 prio=5 os_prio=0 tid=0x00007f179c00d000 nid=0x4277 waiting on condition [0x00007f1757efd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-66-acceptor-0@4c609d82-ServerConnector@54024e42{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #66 prio=3 os_prio=0 tid=0x00007f179c00b800 nid=0x4276 runnable [0x00007f1757ffe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000005d2f2dec8> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:564)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-65" #65 prio=5 os_prio=0 tid=0x00007f179c00a800 nid=0x4275 waiting on condition [0x00007f175c1ee000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"qtp2121400959-64" #64 prio=5 os_prio=0 tid=0x00007f179c00a000 nid=0x4274 waiting on condition [0x00007f175c2ef000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"MarathonSchedulerService$$EnhancerByGuice$$654b9794" #61 prio=5 os_prio=0 tid=0x00007f17615e0000 nid=0x4271 waiting on condition [0x00007f175c3ef000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fa98d8> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$run$1.apply$mcV$sp(MarathonSchedulerService.scala:170)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$run$1.apply(MarathonSchedulerService.scala:170)
at mesosphere.marathon.MarathonSchedulerService$$anonfun$run$1.apply(MarathonSchedulerService.scala:170)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.package$.blocking(package.scala:123)
at mesosphere.marathon.MarathonSchedulerService.run(MarathonSchedulerService.scala:170)
at mesosphere.marathon.MarathonSchedulerService$$EnhancerByGuice$$654b9794.CGLIB$run$25(<generated>)
at mesosphere.marathon.MarathonSchedulerService$$EnhancerByGuice$$654b9794$$FastClassByGuice$$195a9e4d.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at mesosphere.marathon.DebugModule$MetricsBehavior$$anonfun$invoke$1.apply(DebugConf.scala:87)
at mesosphere.marathon.metrics.Metrics.timed(Metrics.scala:28)
at mesosphere.marathon.DebugModule$MetricsBehavior.invoke(DebugConf.scala:87)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at mesosphere.marathon.MarathonSchedulerService$$EnhancerByGuice$$654b9794.run(<generated>)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
at com.google.common.util.concurrent.Callables$3.run(Callables.java:100)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"marathonSchedulerTimer" #58 prio=5 os_prio=0 tid=0x00007f17615a0800 nid=0x426f in Object.wait() [0x00007f175c7f2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000005e5fb44b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"RxScheduledExecutorPool-2" #57 daemon prio=5 os_prio=0 tid=0x00007f17614d2800 nid=0x426e waiting on condition [0x00007f175caf3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb44e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RxScheduledExecutorPool-1" #56 daemon prio=5 os_prio=0 tid=0x00007f17614d2000 nid=0x426d waiting on condition [0x00007f175cbf4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb45f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-Framework-0" #55 daemon prio=5 os_prio=0 tid=0x00007f176138b000 nid=0x426a waiting on condition [0x00007f175cef5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb4800> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:804)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005e5af1820> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"run-main-0-EventThread" #54 daemon prio=5 os_prio=0 tid=0x00007f1761393000 nid=0x4269 waiting on condition [0x00007f175cff6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb5a60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:511)
Locked ownable synchronizers:
- None
"run-main-0-SendThread(10.10.10.10:2181)" #53 daemon prio=5 os_prio=0 tid=0x00007f1761392800 nid=0x4268 runnable [0x00007f175d0f7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005e5fb5f28> (a sun.nio.ch.Util$2)
- locked <0x00000005e5fb5f10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005e5a79a70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
Locked ownable synchronizers:
- None
"Curator-ConnectionStateManager-0" #52 daemon prio=5 os_prio=0 tid=0x00007f1760446800 nid=0x4267 waiting on condition [0x00007f175d1f8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e5fb54f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005e5af1a30> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"marathon-scheduler-1" #49 prio=5 os_prio=0 tid=0x00007f1761283000 nid=0x4263 sleeping[0x00007f175d7fa000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:266)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"com.google.inject.internal.util.$Finalizer" #48 daemon prio=5 os_prio=0 tid=0x00007f1760f7f800 nid=0x425e in Object.wait() [0x00007f175defb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000005e615bcd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)
Locked ownable synchronizers:
- None
"AsyncAppender-Worker-async" #47 daemon prio=5 os_prio=0 tid=0x00007f1760870800 nid=0x425a waiting on condition [0x00007f175e3fc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e57c5e20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:264)
Locked ownable synchronizers:
- None
"RMI TCP Connection(3)-127.0.0.1" #44 daemon prio=9 os_prio=0 tid=0x00007f1764035800 nid=0x4232 runnable [0x00007f175fffd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000005c456c270> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$10/1694409069.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005de6d9c88> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"JMX server connection timeout 43" #43 daemon prio=9 os_prio=0 tid=0x00007f1758023800 nid=0x422f in Object.wait() [0x00007f1774547000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
- locked <0x00000005de653350> (a [I)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" #42 daemon prio=9 os_prio=0 tid=0x00007f175802a800 nid=0x422e waiting on condition [0x00007f1774648000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005de653380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Connection(idle)" #41 daemon prio=9 os_prio=0 tid=0x00007f176403a000 nid=0x422d waiting on condition [0x00007f1774749000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005e02c38f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" #40 daemon prio=9 os_prio=0 tid=0x00007f176004b000 nid=0x422b runnable [0x00007f177484a000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Attach Listener" #39 daemon prio=9 os_prio=0 tid=0x00007f17b4001000 nid=0x4228 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"pool-4-thread-3" #33 prio=5 os_prio=0 tid=0x00007f18082d1800 nid=0x4207 in Object.wait() [0x00007f17c4987000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1245)
- locked <0x00000005e35818e8> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1319)
at sbt.TrapExit$.sbt$TrapExit$$waitOnThread(TrapExit.scala:89)
at sbt.TrapExit$$anonfun$waitForExit$1.apply(TrapExit.scala:198)
at sbt.TrapExit$$anonfun$waitForExit$1.apply(TrapExit.scala:192)
at sbt.TrapExit$App$$anonfun$processThreads$2.apply(TrapExit.scala:326)
at sbt.TrapExit$App$$anonfun$processThreads$2.apply(TrapExit.scala:321)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.concurrent.TrieMapIterator.foreach(TrieMap.scala:922)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.concurrent.TrieMap.foreach(TrieMap.scala:632)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at sbt.TrapExit$App.processThreads(TrapExit.scala:321)
at sbt.TrapExit.waitForExit(TrapExit.scala:192)
at sbt.TrapExit.finish(TrapExit.scala:182)
at sbt.TrapExit.runManaged0(TrapExit.scala:160)
at sbt.TrapExit.runManaged(TrapExit.scala:150)
at sbt.TrapExit$.apply(TrapExit.scala:41)
at sbt.Run$.executeTrapExit(Run.scala:92)
at sbt.Run.run(Run.scala:55)
at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$46$$anonfun$apply$47.apply(Defaults.scala:769)
at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$46$$anonfun$apply$47.apply(Defaults.scala:767)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005d8ce6538> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"java-sdk-http-connection-reaper" #30 daemon prio=5 os_prio=0 tid=0x00007f17801b7000 nid=0x41b9 waiting on condition [0x00007f17c4a89000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:157)
Locked ownable synchronizers:
- None
"Service Thread" #11 daemon prio=9 os_prio=0 tid=0x00007f18081e7000 nid=0x4198 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread2" #10 daemon prio=9 os_prio=0 tid=0x00007f18081e3800 nid=0x4197 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #9 daemon prio=9 os_prio=0 tid=0x00007f18081e1800 nid=0x4196 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #8 daemon prio=9 os_prio=0 tid=0x00007f18081df800 nid=0x4195 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Event Helper Thread" #7 daemon prio=10 os_prio=0 tid=0x00007f18081dd000 nid=0x4194 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Transport Listener: dt_socket" #6 daemon prio=10 os_prio=0 tid=0x00007f18081d9800 nid=0x4193 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f18081cc800 nid=0x4192 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f18081cb800 nid=0x4191 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1808199000 nid=0x4190 in Object.wait() [0x00007f17c76fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000005d4d48928> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f1808194800 nid=0x418f in Object.wait() [0x00007f17c77fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000005d4d48958> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007f1808010000 nid=0x4188 waiting on condition [0x00007f180f93f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005d8c848e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:188)
at sbt.Execute.next$1(Execute.scala:85)
at sbt.Execute.processAll(Execute.scala:88)
at sbt.Execute.runKeep(Execute.scala:68)
at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:359)
at sbt.EvaluateTask$.run$1(EvaluateTask.scala:358)
at sbt.EvaluateTask$.runTask(EvaluateTask.scala:378)
at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:69)
at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:67)
at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:314)
at sbt.Aggregation$.timedRun(Aggregation.scala:67)
at sbt.Aggregation$.runTasks(Aggregation.scala:76)
at sbt.Aggregation$$anonfun$applyDynamicTasks$1.apply(Aggregation.scala:117)
at sbt.Aggregation$$anonfun$applyDynamicTasks$1.apply(Aggregation.scala:115)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:61)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:61)
at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:259)
at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:256)
at sbt.Command$.process(Command.scala:93)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96)
at sbt.State$$anon$1.process(State.scala:184)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.MainLoop$.next(MainLoop.scala:96)
at sbt.MainLoop$.run(MainLoop.scala:89)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:68)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:63)
at sbt.Using.apply(Using.scala:24)
at sbt.MainLoop$.runWithNewLog(MainLoop.scala:63)
at sbt.MainLoop$.runAndClearLast(MainLoop.scala:46)
at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:30)
at sbt.MainLoop$.runLogged(MainLoop.scala:22)
at sbt.StandardMain$.runManaged(Main.scala:57)
at sbt.xMain.run(Main.scala:29)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
at xsbt.boot.Launch$.run(Launch.scala:109)
at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
at xsbt.boot.Launch$.launch(Launch.scala:117)
at xsbt.boot.Launch$.apply(Launch.scala:18)
at xsbt.boot.Boot$.runImpl(Boot.scala:41)
at xsbt.boot.Boot$.main(Boot.scala:17)
at xsbt.boot.Boot.main(Boot.scala)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f180818d000 nid=0x418e runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f1808021800 nid=0x4189 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f1808023000 nid=0x418a runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f1808025000 nid=0x418b runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f1808026800 nid=0x418c runnable
"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007f1808069800 nid=0x418d runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f18081ea800 nid=0x4199 waiting on condition
JNI global references: 26993
Related
- https://github.com/mesosphere/marathon/issues/3031
- https://github.com/mesosphere/marathon/issues/3030
- https://github.com/mesosphere/marathon/issues/3029
_Special thanks to @msulima for help with tracking this down_
janisz
👍6
All 2 comments
There's actually a bit more to this: when not in an actor, we will still allocate a lot of extra threads and can actually run out of native memory for all the thread stacks very quickly (create a group with 5k apps and marathon instantly dies). You're correct in that we need to remove as many Await.results as we can (and use WorkQueue and stuff to implement async locking when needed).
jasongilanfarr
on 23 Nov 2016
👍6
Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-2208. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.
meichstedt
on 7 Mar 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
timcharper
路
8Comments
akamalov
路
9Comments
timcharper
路
4Comments
robertd
路
5Comments
ozdanborne
路
5Comments
Most helpful comment
There's actually a bit more to this: when not in an actor, we will still allocate a lot of extra threads and can actually run out of native memory for all the thread stacks very quickly (create a group with 5k apps and marathon instantly dies). You're correct in that we need to remove as many Await.results as we can (and use WorkQueue and stuff to implement async locking when needed).