Hikaricp: Leaked memory with hikariCP when used as a resource inside tomcat

[faroukelabady]

trying to build a new web application with spring hibernate, tomcat and mysql , at first I created the Hikari datasource programatically , then I decided to move the cofigruation into the context xml , to be able to change it dynamically but when I undeploy my web application I get the following error. I tried to change the put the mysql driver and hikariCP into tomcat directory with no use , also tried to to set classesToInitialize="com.mysql.jdbc.NonRegisteringDriver inside tomcat server xml as follows
< Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
classesToInitialize="com.mysql.jdbc.NonRegisteringDriver" /> but it only make the first warning disappear while the Hikari warning still exists

Jun 10, 2016 4:38:25 AM org.apache.catalina.core.StandardContext reload
INFO: Reloading Context with name [/ThymeleafTemplate] has started
Jun 10, 2016 4:38:25 AM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Jun 10, 2016 4:38:25 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class org.springframework.web.context.ContextLoaderListener
java.lang.NoClassDefFoundError: org/apache/logging/log4j/message/SimpleMessage
at org.apache.logging.slf4j.Log4jLogger.log(Log4jLogger.java:368)
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:216)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:581)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1003)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:979)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:931)
at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4859)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5478)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3821)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:291)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5616)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.SimpleMessage
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1308)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1142)
... 23 more

Jun 10, 2016 4:38:25 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ThymeleafTemplate] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
Jun 10, 2016 4:38:25 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ThymeleafTemplate] appears to have started a thread named [Hikari housekeeper (pool farouk)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Jun 10, 2016 4:38:25 AM org.apache.catalina.core.StandardContext reload
SEVERE: Exception starting Context with name [/ThymeleafTemplate]
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ThymeleafTemplate]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3828)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:291)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5616)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot@1de98865]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:153)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4928)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5058)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
... 8 more
Caused by: java.lang.IllegalArgumentException: The main resource set specified [/home/frouk/wtpwebapps/ThymeleafTemplate] is not valid
at org.apache.catalina.webresources.StandardRoot.createMainResourceSet(StandardRoot.java:723)
at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:684)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
... 11 more

Jun 10, 2016 4:38:25 AM org.apache.catalina.core.StandardContext reload
INFO: Reloading Context with name [/ThymeleafTemplate] is completed

Reproduce

configure Hikari as tomocat resource

Expected

when undeploy the web application the connection pool should be closed

Observed

the hikari pool still exists and working in the background which results in a memory leak

[brettwooldridge]

Is the web app configured to destroy the pool at shutdown? It does not look like shutdown is being called.

Also, have you tried using the DataSource rather than the driver?

[faroukelabady]

what kind of configuration ? , I tried lots of methods,and what I managed to shutdown is the AbandonedConnectionCleanupThread , either by using special ServletContextListener implementation , or use the < Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
classesToInitialize="com.mysql.jdbc.NonRegisteringDriver" />, but couldn't manage to make the hikariCP warning to disappear.

[brettwooldridge]

@faroukelabady You probably want something like a ServletContextListener that calls close() on the HikariDataSource in the contextDestroyed() method.

[brettwooldridge]

@faroukelabady Another possiblity with spring is configuring the destroy-method in the bean declaration for the datasource, eg. destroy-method="close".

See the bean declaration at the bottom here for an example of the destroy method.

[faroukelabady]

it will work if i configure hikari inside spring , but if I put the hikari configuration into a resource inside tomcat context , that is when the issue occurs

[brettwooldridge]

I don't really know what that means, but it sounds like a Tomcat configuration issue than a HikariCP. We have thousands of users on Tomcat without issues. I suggest you try asking on stackoverflow or the Tomcat mailing list.

[faroukelabady]

Okay I thought I will get an answer here , thanks for your effort :)

[natechadwick]

I can confirm that for a Spring application, adding the destroy-method="close" attribute to the data source as described above does indeed resolve the memory leak reporting by Tomcat 7.

[janbodnar]

In the Tomcat's context.xml, you have a configuration option closeMethod="close" for your
datasource.

  <Resource name="jdbc/myDs" auth="Container"
                factory="com.zaxxer.hikari.HikariJNDIFactory"
                dataSourceClassName="com.mysql.jdbc.jdbc2.optional.MysqlDataSource"
                dataSource.url="jdbc:mysql://localhost/testdb?useSSL=false"
                type="javax.sql.DataSource"
                minimumIdle="5" 
                maximumPoolSize="10"
                connectionTimeout="300000"
                database="testdb"
                server="localhost"
                dataSource.user="testuser"
                dataSource.password="test623"
                dataSource.cachePrepStmts="true"
                dataSource.prepStmtCacheSize="250"
                dataSource.prepStmtCacheSqlLimit="2048"
                closeMethod="close"/>

The following post gives further explanation: http://stackoverflow.com/a/13631242/2008247