Generator-jhipster: Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSource'

[sudheerp]

Logging the issue as suggested by brettw. I have tried with HikariCP 2.2.4 and 2.2.5 versions without luck
http://stackoverflow.com/questions/27440985/unable-to-register-mbean-hikaridatasource-hikaripool-0-with-key-datasource

issue

I am getting below error in prod mode for (Java8+Oauth2+MySql+Hazelcast+no clustered http sessions) combination. Dev mode has worked fine.

Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSou
rce'; nested exception is javax.management.InstanceAlreadyExistsException: com.z
axxer.hikari:name=dataSource,type=HikariDataSource

In the log, first part of the datasource (hazelcast) creation says [dev] mode. Not sure this is deliberate.

Log

Dec 12, 2014 2:44:11 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive D:\toolsapache-tomcat-7.0.55\webapps\re
tailenergy.war
[INFO] com.fg.re.ApplicationWebXml - Running with Spring profile(s) : prod
2014-12-12 14:44:19.125 WARN 1368 --- [ost-startStop-1] o.s.b.l.LoggingApplicat
ionListener : Logging environment value '-Djava.util.logging.config.file="
D:\toolsapache-tomcat-7.0.55\conf\logging.properties"' cannot be opened and wil
l be ignored (using default location instead)
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more in
fo.
[DEBUG] com.fg.re.config.AsyncConfiguration - Creating Async Task Executor
[DEBUG] com.fg.re.config.MetricsConfiguration - Registering JVM gauges
[INFO] com.fg.re.config.MetricsConfiguration - Initializing Metrics JMX reportin
g
[INFO] com.hazelcast.instance.DefaultAddressPicker - null [dev] [3.2.5] Prefer I
Pv4 stack is true.
[INFO] com.hazelcast.instance.DefaultAddressPicker - null [dev] [3.2.5] Picked A
ddress[192.168.1.9]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localp
ort=5701], bind any local is true
20140814) starting at Address[192.168.1.9]:5701
8-2014 Hazelcast.com
ulticastJoiner
dress[192.168.1.9]:5701 is STARTING
Members [1] {
Member [192.168.1.9]:5701 this
}

dress[192.168.1.9]:5701 is STARTED
[DEBUG] com.fg.re.config.CacheConfiguration - Starting HazelcastCacheManager
y.used=242.6M, memory.free=219.4M, memory.total=462.0M, memory.max=891.0M, memor
y.used/total=52.51%, memory.used/max=27.23%, load.process=-100.00%, load.system=
100.00%, load.systemAverage=-100.00%, thread.count=38, thread.peakCount=38, even
t.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.operat
ion.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.
size=0, executor.q.system.size=0, executor.q.operation.size=0, executor.q.priori
tyOperation.size=0, executor.q.response.size=0, operations.remote.size=0, operat
ions.running.size=0, proxy.count=0, clientEndpoint.count=0, connection.active.co
unt=0, connection.count=0
[DEBUG] com.fg.re.config.DatabaseConfiguration - Configuring Datasource
[DEBUG] com.zaxxer.hikari.HikariConfig - HikariCP pool HikariPool-0 configuratio
n:
[DEBUG] com.zaxxer.hikari.HikariConfig - autoCommit......................true
[DEBUG] com.zaxxer.hikari.HikariConfig - catalog.........................
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionCustomizer............com.zax
xer.hikari.HikariConfig$1@5a3cdbda
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionCustomizerClassName...
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionInitSql...............
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionTestQuery.............
[DEBUG] com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSource......................
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSourceClassName.............com.mys
ql.jdbc.jdbc2.optional.MysqlDataSource
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................
[DEBUG] com.zaxxer.hikari.HikariConfig - dataSourceProperties............{user=r
oot, url=jdbc:mysql://localhost:3306/retailenergy, password=, prepStmtCa
cheSqlLimit=2048, cachePrepStmts=true, useServerPrepStmts=true, prepStmtCacheSiz
e=250}
[DEBUG] com.zaxxer.hikari.HikariConfig - driverClassName.................
[DEBUG] com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
[DEBUG] com.zaxxer.hikari.HikariConfig - initializationFailFast..........true
[DEBUG] com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
[DEBUG] com.zaxxer.hikari.HikariConfig - jdbc4ConnectionTest.............false
[DEBUG] com.zaxxer.hikari.HikariConfig - jdbcUrl.........................
[DEBUG] com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........0
[DEBUG] com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000

[DEBUG] com.zaxxer.hikari.HikariConfig - maximumPoolSize.................10
[DEBUG] com.zaxxer.hikari.HikariConfig - metricRegistry..................com.cod
ahale.metrics.MetricRegistry@4152b59f
[DEBUG] com.zaxxer.hikari.HikariConfig - minimumIdle.....................10
[DEBUG] com.zaxxer.hikari.HikariConfig - password........................

[DEBUG] com.zaxxer.hikari.HikariConfig - poolName........................HikariP
ool-0
[DEBUG] com.zaxxer.hikari.HikariConfig - readOnly........................false
[DEBUG] com.zaxxer.hikari.HikariConfig - registerMbeans..................false
[DEBUG] com.zaxxer.hikari.HikariConfig - threadFactory...................
[DEBUG] com.zaxxer.hikari.HikariConfig - transactionIsolation............
[DEBUG] com.zaxxer.hikari.HikariConfig - username........................
[INFO] com.zaxxer.hikari.HikariDataSource - HikariCP pool HikariPool-0 is starti
ng.
[DEBUG] com.fg.re.config.DatabaseConfiguration - Configuring Liquibase
[INFO] com.fg.re.config.hazelcast.HazelcastCacheRegionFactory - Starting up Haze
lcastCacheRegionFactory
[DEBUG] com.fg.re.config.MailConfiguration - Configuring mail server
[INFO] com.fg.re.config.WebConfigurer - Web application configuration, using pro
files: [prod]
[DEBUG] com.fg.re.config.WebConfigurer - Initializing Metrics registries
[DEBUG] com.fg.re.config.WebConfigurer - Registering Metrics Filter
[DEBUG] com.fg.re.config.WebConfigurer - Registering Metrics Servlet
[DEBUG] com.fg.re.config.WebConfigurer - Registering Caching HTTP Headers Filter

[DEBUG] com.fg.re.config.WebConfigurer - Registering static resources production
Filter
[DEBUG] com.fg.re.config.WebConfigurer - Registering GZip Filter
[INFO] com.fg.re.config.WebConfigurer - Web application fully configured
[INFO] com.fg.re.Application - Running with Spring profile(s) : [prod]
[INFO] com.fg.re.config.ThymeleafConfiguration - loading non-reloadable mail mes
sages resources
[WARN] org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebAppl
icationContext - Exception encountered during context initialization - cancellin
g refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean wit
h name 'mbeanExporter' defined in class path resource [org/springframework/conte
xt/annotation/MBeanExportConfiguration.class]: Invocation of init method failed;
nested exception is org.springframework.jmx.export.UnableToRegisterMBeanExcepti
on: Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSou
rce'; nested exception is javax.management.InstanceAlreadyExistsException: com.z
axxer.hikari:name=dataSource,type=HikariDataSource
at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.initializeBean(AbstractAutowireCapableBeanFactory.java:1554) ~[spring-be
ans-4.0.8.RELEASE.jar:4.0.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[spring-beans
-4.0.8.RELEASE.jar:4.0.8.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBean
Factory.createBean(AbstractAutowireCapableBeanFactory.java:475) ~[spring-beans-4
.0.8.RELEASE.jar:4.0.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getOb
ject(AbstractBeanFactory.java:302) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEAS
E]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistr
y.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[spring-beans-4.0.8.RELEA
SE.jar:4.0.8.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBe
an(AbstractBeanFactory.java:298) ~[spring-beans-4.0.8.RELEASE.jar:4.0.8.RELEASE]

[brettwooldridge]

Maybe this is a known issue, but I asked @sudheerp to open it over here. If he is using HikariCP 2.2.5, he definitely has a codeline that properly un-registers HikariCP's MBean's when the HikariDataSource is closed or shutdown.

The exception:

Unable to register MBean [HikariDataSource (HikariPool-0)] with key 'dataSou
rce'; nested exception is javax.management.InstanceAlreadyExistsException: com.z
axxer.hikari:name=dataSource,type=HikariDataSource

Seems to imply to me a "re-deploy" scenario where HikariCP has already initialized and registered once, but because the close() or shutdown() method was never called on the HikariDataSource, it never un-registered.

[sudheerp]

Not sure these two methods in HazelcastCacheRegionFactory.java generated by jhipster has any relevancy to these errors. Any idea whether this is going to take time? I can limit my choice without hazelcast for immediate needs. Kindly let me know,

public void start(final Settings settings, final Properties properties) throws CacheException {
    // Do nothing the hazelcast hazelcastInstance is injected
    log.info("Starting up {}", getClass().getSimpleName());

    if (hazelcastInstance == null) {
        throw new IllegalArgumentException("Hazelcast hazelcastInstance must not be null");
    }
    cleanupService = new CleanupService(hazelcastInstance.getName());
}

public void stop() {
    // Do nothing the hazelcast instance is managed globally
    log.info("Shutting down {}", getClass().getSimpleName());
    cleanupService.stop();
}

[sudheerp]

Fresh observation. Earlier I had two jhipster webapps running on the same tomcat. Removing the other one has taken off the error. Can I know the reason? In production there can be a high possibility of two jhipster webapps running on the same tomcat.

[brettwooldridge]

@sudheerp in this case, each webapp must be configured with a unique poolName property in HikariCP, otherwise their MBean registration names will collide.

[jmirc]

You just need to add a different poolName to the HikariConfig class.
In the DataSourceConfiguration, you can add new property config and you will be able to set a different poolName by application.

config.addDataSourceProperty("poolName", propertyResolver.getProperty("poolName"))

Add a new spring.datasource.poolName property in the application.yml file.

I am closing this issue because it is not related to JHipster.

[pvlastaridis]

Actually, is not

config.addDataSourceProperty("poolName", propertyResolver.getProperty("poolName"))

but

config.setPoolName(propertyResolver.getProperty("poolName"));

[brettwooldridge]

@pvlastaridis that is correct.

[pvlastaridis]

But finally what worked for me (trying to deploy two jhip apps on the same tomcat) and it is funny is that after fixing poolName I got embarrassing message ............with key 'dataSource'; nested exception is javax.management.InstanceAlreadyExistsException: com.zaxxer.hikari: name=dataSource,type=HikariDataSource. Notice with key 'datasourse'!!!??? So what I tried and it worked is to give Hikari another key for datasource so I changed class name to:

public DataSource dataSource2() {

    HikariConfig config = new HikariConfig();
    config.setPoolName("AARSHikaripool-1");

[pmarrone]

I am also trying to get around this problem, since I need two or more instances of the same application running on the same server. Changing the name of the 'dataSource' bean method is not an option form me, since the idea is that applications can be configured from a configuration file without needing to recompile. I'll let you know if I manage to figure this out.

[iuricmp]

@pmarrone Did you try this?

HikariConfig config = new HikariConfig();
config.setPoolName("AARSHikaripool-" + env.getProperty("server.context-path"));

[pmarrone]

I changed the poolname using a configuration property as @pvlastaridis stated, like this

config.setPoolName(propertyResolver.getProperty("poolname"));

The issue is changing the MBean name either by configuration or by some unique name generated automatically. Changing the 'dataSource' method name in the source code, is not an option, because that would not allow me to have two instances of the same applications running alongside on the same server.

Options that come to my mind are

1. Find out how to change the bean name from externalized configuration somehow

@Bean(name="#{this.wont.work.but.you.get.the.idea})
public DataSource dataSource {

1. Somehow set the JXM MBean name independently from that Spring bean name

config.setMBeanNamePlease(env.getProperty("this.would.be.fine.too"));

Since I am not a Spring expert and this is the first time I even hear about JMX, I am open to suggestions. Thanks @iuricmp for your quick reply.

Update

From https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html

Every JMX MBean must have an object name. The object name is an instance of the JMX class ?ObjectName and must conform to the syntax defined by the JMX specification. Namely, the object name must contain a domain and a list of key-properties.

From http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#boot-features-jmx

1. Monitoring and management over JMX
   Java Management Extensions (JMX) provide a standard mechanism to monitor and manage applications. By default Spring Boot will create an MBeanServer with bean id ‘mbeanServer’ and expose any of your beans that are annotated with Spring JMX annotations (@ManagedResource, @ManagedAttribute, @ManagedOperation).

See the JmxAutoConfiguration class for more details.

Checking the code of JmxAutoConfiguration in https://github.com/spring-projects/spring-boot/blob/master/spring-boot-autoconfigure/src/main/java/org/springframework/boot/autoconfigure/jmx/JmxAutoConfiguration.java
I saw that there is a spring.jmx.default-domain property that is used in the naming strategy.

Setting that property to some value in one of my apps' application.properties solved this issue

spring.jmx.default-domain: test

[brettwooldridge]

@iuricmp @pvlastaridis Someone more clever than me could probably figure-out how to use Sping's RandomValuePropertySource to generate a unique pool name ... something like "mypool-${random.int}".

[pmarrone]

@brettwooldridge: as @pvlastaridis said, changing the pool name alone didn't fix the name clashing since the MBean gets registered as 'dataSource' twice. Since changing the method name to 'dataSource2' was not an option for me, I used spring.jmx.default-domain property to avoid this collisions. Any thoughts about this? Feedback is highly appreciated.

As @sudheerp stated above, if you or your company are using JHipster's project structure, it seems likely that eventually you'll end up with more than one JHipster application on the same production server.

[pmarrone]

@brettwooldridge
The config.setPoolName() function takes a string, so you should be able to generate a simple string in the most basic of ways. Similar to @iuricmp example, you could replace

config.setPoolName("AARSHikaripool-" + env.getProperty("server.context-path"));

by

config.setPoolName("AARSHikaripool-" + Math.random());

or even

config.setPoolName("AARSHikaripool-" + Calendar.getInstance().getTimeInMillis());

but you would still get collisions because of the 'dataSource' bean name

[brettwooldridge]

@pmarrone Is it possible to achieve with a custom JMX domain configuration this is dynamic?

[pmarrone]

@brettwooldridge What worked for me was setting spring.jmx.default-domain property in each instance configuration (in my case, with an externalized application.properties file).
So, in app 1 application.properties:

spring.jmx.default-domain:   app1

And in app 2 application.properties:

spring.jmx.default-domain: app2

I'm pretty sure you could set this automatically on runtime to a generated value if you wanted to. It would just imply setting the environment variable spring.jmx.default-domain to a custom string, just like "AARSHikaripool-" + env.getProperty("server.context-path") before Hikari configuration.

[aquaglia]

@pmarrone What you suggested worked for me, too, many thanks!

[yaswanthbs]

@pmarrone I have done the same, on what you have suggested. But it didn't work. Can anyone help me in this.
Environment:
spring - 4.2.5.RELEASE
spring boot - 1.3.3.RELEASE
hikariCP - 2.4.7

code:

@Bean(name="HikariDataSource",destroyMethod = "shutdown")
    public DataSource dataSource2() {
        HikariConfig config = new HikariConfig();
        config.setDriverClassName("oracle.jdbc.OracleDriver");
        config.setJdbcUrl("");
        config.setUsername("");
        config.setPassword("");
        config.addDataSourceProperty("cachePrepStmts", "true");
        config.addDataSourceProperty("prepStmtCacheSize", "250");
        config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");
        config.setPoolName("Hikaripool-1");

        HikariDataSource ds = new HikariDataSource(config);
        return ds;
    }
        @Bean
    public JdbcTemplate getJdbcTemplate() {

        return new JdbcTemplate(dataSource2());
    }

Exception:

java.lang.IllegalStateException: Failed to load ApplicationContext
Caused by: org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [HikariDataSource (null)] with key 'dataSource'; nested exception is javax.management.InstanceAlreadyExistsException: com.zaxxer.hikari:name=dataSource,type=HikariDataSource
Caused by: javax.management.InstanceAlreadyExistsException: com.zaxxer.hikari:name=dataSource,type=HikariDataSource

Can anyone help me in resolving this issue.

[brettwooldridge]

@yaswanthbs Is this a single app, or two apps running in different containers in the same VM (as the above original issue was)?

First, turn on DEBUG level logging for the package com.zaxxer.hikari. It looks like two pools are somehow being created. The second one, with the same name as the first, will fail to register. If you don't declare a poolName then HikariCP will auto-generate them, which will work, but is likely to just cover-up the underlying issue of why two pools are being created. That is either a Spring Boot or JHipster configuration issue ... must likely Spring Boot.

[pmarrone]

@yaswanthbs , It's been a long time since I used JHipster, but your problem still looks exactly the same. Are you sure you tried this?

What worked for me was setting spring.jmx.default-domain property in each instance configuration (in my case, with an externalized application.properties file).
So, in app 1 application.properties:

spring.jmx.default-domain: app1
And in app 2 application.properties:

spring.jmx.default-domain: app2

[akang]

@pmarrone Your tip worked for me. Thanks sir! I do think it should be added to the .yml at the point when the project gets generated though (using at the minimum the project name)

[xrom888]

works for me:
spring.jmx.default-domain=app1

thanks a lot...

[NarsimhaPardhu]

If the servlet container is TOMCAT then see this...

With spring.jmx.enabled set to true (_the default_), any beans in the context that are _MBeans_ will be automatically registered with the JMX server. This fails because Tomcat has already registered them. You have a few options:

Set spring.jmx.enabled=false

Reference Link: https://github.com/spring-projects/spring-boot/issues/9179

And it's worked for me.

[NarsimhaPardhu]

works for me:
spring.jmx.default-domain=app1

thanks a lot...

spring:
jmx:
default-domain: [application_name]

It only works for deploying, mutliple jhipster/JMX enabled spring contexts in single tomcat container.
But, If we want to give domain name to a particular context thorugh TOMCAT VIRTUAL_HOST, then we get below exception

Caused by: org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [HikariDataSource (HikariPool-2)] with key 'dataSource'; nested exception is javax.management.InstanceAlreadyExistsException: [application_name]:name=dataSource,type=HikariDataSource
at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:628)
at org.springframework.jmx.export.MBeanExporter.registerBeans(MBeanExporter.java:550)
at org.springframework.jmx.export.MBeanExporter.afterSingletonsInstantiated(MBeanExporter.java:432)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:781)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
................................

In this case disabling JMX, we can resolve above issue