Exist: Unit suite hangs on jetty shutdown

Created on 30 Jul 2016  路  10Comments  路  Source: eXist-db/exist

There trace is attached. My runs do never finish :-(

locktraces.txt

your kit profiler output:

junittestrunner_-_yourkit_java_profiler_2016_02-b40_-_64-bit

@adamretter ? @wolfgangmm ? can you reproduce?

the last logs:

exist_ _tail_-f_test_txt_ _bash_ _174x82

bug high prio
Source
picture dizzzz

All 10 comments

I did not get this hang. Test suite did run through, reporting 2 errors and 1 failure.

wolfgangmm picture wolfgangmm on 30 Jul 2016
😕1

ok, interesting. which java JDK version? I tried 3 times, 3 hangs....

dizzzz picture dizzzz on 30 Jul 2016

in essence the shutdown of jetty is blocked by the repo-scan during the last startup....

jetty shutdown schedule [BLOCKED; waiting to lock org.exist.storage.BrokerPool@59934c50]
 org.exist.storage.BrokerPool.shutdown(BrokerPool.java:1960)
 org.exist.storage.BrokerPool.stopAll(BrokerPool.java:411)
 org.exist.http.servlets.AbstractExistHttpServlet.destroy(AbstractExistHttpServlet.java:90)
.....
 org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:544)
 org.exist.jetty.WebAppContext.doStop(WebAppContext.java:39)
 org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
....
 org.eclipse.jetty.server.Server.doStop(Server.java:482)
 org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
 org.exist.jetty.JettyStart$ShutdownListenerImpl$1.run(JettyStart.java:552)
 java.util.TimerThread.mainLoop(Timer.java:555)
 java.util.TimerThread.run(Timer.java:505)

main [WAITING]
 sun.misc.Unsafe.park(native method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
 java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
 java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
 org.exist.storage.txn.TransactionManager.withLock(TransactionManager.java:344)
 org.exist.storage.txn.TransactionManager.withLock(TransactionManager.java:324)
 org.exist.storage.txn.TransactionManager.commit(TransactionManager.java:139)
 org.exist.repo.Deployment.storeFiles(Deployment.java:779)
 org.exist.repo.Deployment.scanDirectory(Deployment.java:693)
 org.exist.repo.Deployment.lambda$scanDirectory$4(Deployment.java:698)
 org.exist.repo.Deployment$$Lambda$175/1268343946.accept(unknown source)
 java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
....
 java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
 org.exist.repo.Deployment.scanDirectory(Deployment.java:698)
 org.exist.repo.Deployment.lambda$scanDirectory$4(Deployment.java:698)
 org.exist.repo.Deployment$$Lambda$175/1268343946.accept(unknown source)
 java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
....
 java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
 org.exist.repo.Deployment.scanDirectory(Deployment.java:698)
 org.exist.repo.Deployment.deploy(Deployment.java:410)
 org.exist.repo.Deployment.installAndDeploy(Deployment.java:271)
 org.exist.repo.AutoDeploymentTrigger.execute(AutoDeploymentTrigger.java:72)
 org.exist.storage.BrokerPool.callStartupTriggers(BrokerPool.java:1098)
 org.exist.storage.BrokerPool.initialize(BrokerPool.java:1010)
 org.exist.storage.BrokerPool.<init>(BrokerPool.java:716)
 org.exist.storage.BrokerPool.configure(BrokerPool.java:269)
 org.exist.storage.BrokerPool.configure(BrokerPool.java:243)
 org.exist.storage.ModificationTimeTest.startDB(ModificationTimeTest.java:137)
 org.exist.storage.ModificationTimeTest.storeXML(ModificationTimeTest.java:185)
 org.exist.storage.ModificationTimeTest.check_if_modification_time_is_not_updated_on_parse_error(ModificationTimeTest.java:109)
....
 org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
dizzzz picture dizzzz on 31 Jul 2016

Confirmed on my PC and my Notebook (both Macs, latest OS, recent JDKs, SSD)

dizzzz picture dizzzz on 31 Jul 2016
😕1

Just to confirm, you're both on the current develop branch?

joewiz picture joewiz on 31 Jul 2016

good point; yes :-)

dizzzz picture dizzzz on 31 Jul 2016

@dizzzz I am unable to reproduce this from develop on Mac OS X (10.11.6) with Oracle jdk1.8.0_92.jdk. Which OS version / JDK do you have?

However, I did see a shutdown deadlock issue once in a branch from develop that I have, I am trying to reproduce your issue now...

adamretter picture adamretter on 4 Aug 2016

@dizzzz Okay I can reproduce it from develop on Mac OS X (10.11.6) with Azul Zulu 8.15.0.1-macosx (openjdk version "1.8.0_92")

2016-08-04 18:34:36
Full thread dump OpenJDK 64-Bit Server VM (25.92-b15 mixed mode):

"Attach Listener" #709 daemon prio=9 os_prio=31 tid=0x00007f979c09b000 nid=0x810b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Checkpointer" #708 daemon prio=5 os_prio=31 tid=0x00007f97a117c000 nid=0x8503 in Object.wait() [0x0000700002f20000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:174)
    - locked <0x00000007bde502b8> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:745)

"Cleaner-1" #707 daemon prio=5 os_prio=31 tid=0x00007f97a127d800 nid=0x61ff in Object.wait() [0x0000700002d1a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:174)
    - locked <0x00000007bde4fe80> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:745)

"INCompressor" #706 daemon prio=5 os_prio=31 tid=0x00007f979afb7000 nid=0x560f in Object.wait() [0x0000700001ff3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:176)
    - locked <0x00000007bde549c0> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:745)

"Thread-212" #705 prio=5 os_prio=31 tid=0x00007f97a12d1800 nid=0x580f in Object.wait() [0x0000700001ef0000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at org.exist.storage.journal.FileSyncThread.run(FileSyncThread.java:117)
    - locked <0x00000007bdbef798> (a org.exist.storage.journal.FileSyncThread)

"Thread-211" #704 prio=5 os_prio=31 tid=0x00007f97a0819000 nid=0x590f in Object.wait() [0x0000700001ded000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.exist.storage.BrokerPool$StatusReporter.run(BrokerPool.java:2234)
    - locked <0x00000007bdb7a0e8> (a org.exist.storage.BrokerPool$StatusReporter)
    at java.lang.Thread.run(Thread.java:745)

"jetty shutdown schedule" #698 daemon prio=5 os_prio=31 tid=0x00007f97a0877800 nid=0x710b waiting for monitor entry [0x00007000017da000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.exist.storage.BrokerPool.shutdown(BrokerPool.java:1960)
    - waiting to lock <0x00000007bdb79ff8> (a org.exist.storage.BrokerPool)
    at org.exist.storage.BrokerPool.stopAll(BrokerPool.java:411)
    at org.exist.http.servlets.AbstractExistHttpServlet.destroy(AbstractExistHttpServlet.java:91)
    at org.eclipse.jetty.servlet.ServletHolder.destroyInstance(ServletHolder.java:475)
    at org.eclipse.jetty.servlet.ServletHolder.doStop(ServletHolder.java:446)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bed19f10> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:226)
    - locked <0x00000007be8c50c8> (a org.eclipse.jetty.servlet.ServletHandler)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c51c0> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.security.SecurityHandler.doStop(SecurityHandler.java:381)
    at org.eclipse.jetty.security.ConstraintSecurityHandler.doStop(ConstraintSecurityHandler.java:456)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c4dc8> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:127)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c4c50> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:819)
    at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:356)
    at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1410)
    at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1374)
    at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:874)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:272)
    at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:544)
    at org.exist.jetty.WebAppContext.doStop(WebAppContext.java:39)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c6bf0> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bec0fea8> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bec0f710> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bec0f5c8> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.server.Server.doStop(Server.java:482)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be862a30> (a java.lang.Object)
    at org.exist.jetty.JettyStart$ShutdownListenerImpl$1.run(JettyStart.java:552)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

"Scheduler-1543076841" #693 prio=5 os_prio=31 tid=0x00007f97a19cd800 nid=0x6b03 waiting on condition [0x0000700002b14000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007bea2f640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-691" #691 prio=5 os_prio=31 tid=0x00007f979d185800 nid=0x64db waiting on condition [0x000070000280b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-690" #690 prio=5 os_prio=31 tid=0x00007f979d209800 nid=0x5e03 waiting on condition [0x0000700002708000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-689" #689 prio=5 os_prio=31 tid=0x00007f97a1326000 nid=0x6a0f waiting on condition [0x0000700002605000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-688" #688 prio=5 os_prio=31 tid=0x00007f979d1e9000 nid=0x5a1f waiting on condition [0x0000700002502000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-687" #687 prio=5 os_prio=31 tid=0x00007f979b652800 nid=0x7f07 waiting on condition [0x00007000023ff000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-686" #686 prio=5 os_prio=31 tid=0x00007f979b63a800 nid=0x7b07 waiting on condition [0x00007000022fc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-685" #685 prio=5 os_prio=31 tid=0x00007f979adc9000 nid=0x7d07 waiting on condition [0x00007000021f9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-684" #684 prio=5 os_prio=31 tid=0x00007f979c258800 nid=0x8307 waiting on condition [0x00007000020f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-683" #683 prio=5 os_prio=31 tid=0x00007f979c1b5800 nid=0x65f3 waiting on condition [0x0000700001be7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"qtp2040656033-682" #682 prio=5 os_prio=31 tid=0x00007f979c084000 nid=0x6313 waiting on condition [0x00007000016d8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be874ba0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
    at java.lang.Thread.run(Thread.java:745)

"org.eclipse.jetty.util.RolloverFileOutputStream" #663 daemon prio=5 os_prio=31 tid=0x00007f979b413800 nid=0x7903 in Object.wait() [0x000070000290e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    - locked <0x00000007beace2e0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007f979a02e000 nid=0x5203 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007f9799831800 nid=0x5003 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f979a81c000 nid=0x4e03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f979a02d800 nid=0x4c03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f979b068800 nid=0x4a03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f979b032000 nid=0x3c0f runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f979982c000 nid=0x3803 in Object.wait() [0x0000700000d3a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    - locked <0x00000007a007f198> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f979a028800 nid=0x3603 in Object.wait() [0x0000700000c37000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
    - locked <0x00000007a007f368> (a java.lang.ref.Reference$Lock)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=31 tid=0x00007f979b001800 nid=0x1703 waiting on condition [0x0000700000216000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be3d47d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at org.exist.storage.txn.TransactionManager.withLock(TransactionManager.java:344)
    at org.exist.storage.txn.TransactionManager.withLock(TransactionManager.java:324)
    at org.exist.storage.txn.TransactionManager.commit(TransactionManager.java:139)
    at org.exist.repo.Deployment.storeFiles(Deployment.java:779)
    at org.exist.repo.Deployment.scanDirectory(Deployment.java:693)
    at org.exist.repo.Deployment.lambda$scanDirectory$4(Deployment.java:698)
    at org.exist.repo.Deployment$$Lambda$176/410816406.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at org.exist.repo.Deployment.scanDirectory(Deployment.java:698)
    at org.exist.repo.Deployment.lambda$scanDirectory$4(Deployment.java:698)
    at org.exist.repo.Deployment$$Lambda$176/410816406.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at org.exist.repo.Deployment.scanDirectory(Deployment.java:698)
    at org.exist.repo.Deployment.deploy(Deployment.java:410)
    at org.exist.repo.Deployment.installAndDeploy(Deployment.java:271)
    at org.exist.repo.AutoDeploymentTrigger.execute(AutoDeploymentTrigger.java:72)
    at org.exist.storage.BrokerPool.callStartupTriggers(BrokerPool.java:1098)
    at org.exist.storage.BrokerPool.initialize(BrokerPool.java:1010)
    - locked <0x00000007bdb79ff8> (a org.exist.storage.BrokerPool)
    at org.exist.storage.BrokerPool.<init>(BrokerPool.java:716)
    at org.exist.storage.BrokerPool.configure(BrokerPool.java:269)
    at org.exist.storage.BrokerPool.configure(BrokerPool.java:243)
    at org.exist.storage.ModificationTimeTest.startDB(ModificationTimeTest.java:137)
    at org.exist.storage.ModificationTimeTest.storeXML(ModificationTimeTest.java:185)
    at org.exist.storage.ModificationTimeTest.check_if_modification_time_is_not_updated_on_parse_error(ModificationTimeTest.java:109)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)

"VM Thread" os_prio=31 tid=0x00007f979a819800 nid=0x3403 runnable 

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f979b00d800 nid=0x2403 runnable 

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f979b00e000 nid=0x2603 runnable 

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f979c000000 nid=0x2803 runnable 

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f9799825000 nid=0x2a03 runnable 

"GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007f9799825800 nid=0x2c03 runnable 

"GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007f979a002000 nid=0x2e03 runnable 

"GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007f979a003000 nid=0x3003 runnable 

"GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007f979a003800 nid=0x3203 runnable 

"VM Periodic Task Thread" os_prio=31 tid=0x00007f979a02f000 nid=0x5403 waiting on condition 

JNI global references: 753


Found one Java-level deadlock:
=============================
"jetty shutdown schedule":
  waiting to lock monitor 0x00007f979ad34ae8 (object 0x00000007bdb79ff8, a org.exist.storage.BrokerPool),
  which is held by "main"
"main":
  waiting for ownable synchronizer 0x00000007be3d47d0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "jetty shutdown schedule"

Java stack information for the threads listed above:
===================================================
"jetty shutdown schedule":
    at org.exist.storage.BrokerPool.shutdown(BrokerPool.java:1960)
    - waiting to lock <0x00000007bdb79ff8> (a org.exist.storage.BrokerPool)
    at org.exist.storage.BrokerPool.stopAll(BrokerPool.java:411)
    at org.exist.http.servlets.AbstractExistHttpServlet.destroy(AbstractExistHttpServlet.java:91)
    at org.eclipse.jetty.servlet.ServletHolder.destroyInstance(ServletHolder.java:475)
    at org.eclipse.jetty.servlet.ServletHolder.doStop(ServletHolder.java:446)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bed19f10> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:226)
    - locked <0x00000007be8c50c8> (a org.eclipse.jetty.servlet.ServletHandler)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c51c0> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.security.SecurityHandler.doStop(SecurityHandler.java:381)
    at org.eclipse.jetty.security.ConstraintSecurityHandler.doStop(ConstraintSecurityHandler.java:456)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c4dc8> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:127)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c4c50> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:819)
    at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:356)
    at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1410)
    at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1374)
    at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:874)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:272)
    at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:544)
    at org.exist.jetty.WebAppContext.doStop(WebAppContext.java:39)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be8c6bf0> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bec0fea8> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bec0f710> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007bec0f5c8> (a java.lang.Object)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73)
    at org.eclipse.jetty.server.Server.doStop(Server.java:482)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    - locked <0x00000007be862a30> (a java.lang.Object)
    at org.exist.jetty.JettyStart$ShutdownListenerImpl$1.run(JettyStart.java:552)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)
"main":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007be3d47d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    at org.exist.storage.txn.TransactionManager.withLock(TransactionManager.java:344)
    at org.exist.storage.txn.TransactionManager.withLock(TransactionManager.java:324)
    at org.exist.storage.txn.TransactionManager.commit(TransactionManager.java:139)
    at org.exist.repo.Deployment.storeFiles(Deployment.java:779)
    at org.exist.repo.Deployment.scanDirectory(Deployment.java:693)
    at org.exist.repo.Deployment.lambda$scanDirectory$4(Deployment.java:698)
    at org.exist.repo.Deployment$$Lambda$176/410816406.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at org.exist.repo.Deployment.scanDirectory(Deployment.java:698)
    at org.exist.repo.Deployment.lambda$scanDirectory$4(Deployment.java:698)
    at org.exist.repo.Deployment$$Lambda$176/410816406.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    at org.exist.repo.Deployment.scanDirectory(Deployment.java:698)
    at org.exist.repo.Deployment.deploy(Deployment.java:410)
    at org.exist.repo.Deployment.installAndDeploy(Deployment.java:271)
    at org.exist.repo.AutoDeploymentTrigger.execute(AutoDeploymentTrigger.java:72)
    at org.exist.storage.BrokerPool.callStartupTriggers(BrokerPool.java:1098)
    at org.exist.storage.BrokerPool.initialize(BrokerPool.java:1010)
    - locked <0x00000007bdb79ff8> (a org.exist.storage.BrokerPool)
    at org.exist.storage.BrokerPool.<init>(BrokerPool.java:716)
    at org.exist.storage.BrokerPool.configure(BrokerPool.java:269)
    at org.exist.storage.BrokerPool.configure(BrokerPool.java:243)
    at org.exist.storage.ModificationTimeTest.startDB(ModificationTimeTest.java:137)
    at org.exist.storage.ModificationTimeTest.storeXML(ModificationTimeTest.java:185)
    at org.exist.storage.ModificationTimeTest.check_if_modification_time_is_not_updated_on_parse_error(ModificationTimeTest.java:109)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
    at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)

Found 1 deadlock.
adamretter picture adamretter on 4 Aug 2016

@dizzzz @wolfgangmm so I can see why this happens:

jetty shutdown schedule thread

  1. Locks Transaction Manager
  2. Attempts to lock BrokerPool instance

main thread

  1. Locks BrokerPool instance
  2. Attempts to lock Transaction Manager

Unfortunately I have no idea why we are trying to shutdown the database at the same time as we are trying to initialize the database?!?

Every time I look at BrokerPool I just want to bang my head on the desk... I also just realised that it is not thread-safe with respect to the instances and instancesInitializtionException variables, I just tried to fix it with R/W Locking, but the current design makes that almost impossible.

adamretter picture adamretter on 4 Aug 2016

@dizzzz @wolfgangmm Well there is a fix for this here - https://github.com/eXist-db/exist/pull/1081

However, it's not pretty and there may still be other issues; The current design of BrokerPool.java doesn't lend itself to verifying the concurrency semantics. I will most likely attempt to rewrite/replace it in the long run

adamretter picture adamretter on 4 Aug 2016
Was this page helpful?
0 / 5 - 0 ratings

Related issues

adamretter picture adamretter  路  6Comments
dizzzz picture dizzzz  路  4Comments
mathias-goebel picture mathias-goebel  路  4Comments
adamretter picture adamretter  路  4Comments
merenyics picture merenyics  路  3Comments