Quarkus: [jdbc-mysql extension] MySQL high availability datasource health check not supported in native Quarkus

[SchulteMarkus]

I am using the jdbc-mysql Quarkus extension for connecting my Quarkus application to a MySQL cluster. My datatsource URL is like:

jdbc:mysql:replication://host/db

As you can see, I am using the _Connector/J_ feature replication as described on https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-master-slave-replication-connection.html

While this works fine running Quarkus running in JVM, a Quarkus native fails to do the datasource health check.

When running my Quarkus application as a native, the health endpoint returns:

$ curl localhost:8080/health
{   
    "status": "DOWN",
    "checks": [
        {   
            "name": "Database connections health check",
            "status": "DOWN",
            "data": {
                "default": "Unable to execute the validation check for the default DataSource: !InvalidLoadBalanceExceptionChecker!"
            }
        }
    ]
}

The corresponding exception is

2020-03-18 13:31:17,255 WARN  [io.agr.pool] {} Datasource '<default>': !InvalidLoadBalanceExceptionChecker!
2020-03-18 13:31:17,255 DEBUG [io.agr.pool] {} Cause: : java.sql.SQLException: !InvalidLoadBalanceExceptionChecker!
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:85)
        at com.mysql.cj.jdbc.ha.LoadBalancedConnectionProxy.<init>(LoadBalancedConnectionProxy.java:244)
        at com.mysql.cj.jdbc.ha.LoadBalancedConnectionProxy.createProxyInstance(LoadBalancedConnectionProxy.java:120)
        at com.mysql.cj.jdbc.ha.ReplicationConnectionProxy.initializeSlavesConnection(ReplicationConnectionProxy.java:448)
        at com.mysql.cj.jdbc.ha.ReplicationConnectionProxy.<init>(ReplicationConnectionProxy.java:168)
        at com.mysql.cj.jdbc.ha.ReplicationConnectionProxy.createProxyInstance(ReplicationConnectionProxy.java:88)
        at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:209)
        at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:200)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:390)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:372)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1126)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:834)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:527)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: com.mysql.cj.exceptions.WrongArgumentException: !InvalidLoadBalanceExceptionChecker!
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
        at com.mysql.cj.util.Util.getInstance(Util.java:169)
        at com.mysql.cj.jdbc.ha.LoadBalancedConnectionProxy.<init>(LoadBalancedConnectionProxy.java:239)
        ... 15 more
Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.ha.StandardLoadBalanceExceptionChecker
        at com.oracle.svm.core.hub.ClassForNameSupport.forName(ClassForNameSupport.java:60)
        at java.lang.Class.forName(DynamicHub.java:1211)
        at com.mysql.cj.util.Util.getInstance(Util.java:167)
        ... 16 more

Adding com.mysql.cj.jdbc.ha.StandardLoadBalanceExceptionChecker to the reflection-config.json seems to be only the tip of the iceberg, doing so results in exceptions of kind

2020-03-18 13:18:02,839 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] {} HTTP Request to /health failed, error id: 8865e180-03db-4b52-b335-efa4e0f8223b-1: com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface com.mysql.cj.jdbc.ha.LoadBalancedConnection, interface com.mysql.cj.jdbc.JdbcConnection] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.
        at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:101)
        at com.oracle.svm.reflect.proxy.DynamicProxySupport.getProxyClass(DynamicProxySupport.java:113)
        at java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:66)
        at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:1006)
        at com.mysql.cj.jdbc.ha.LoadBalancedConnectionProxy.createProxyInstance(LoadBalancedConnectionProxy.java:121)
        at com.mysql.cj.jdbc.ha.ReplicationConnectionProxy.initializeSlavesConnection(ReplicationConnectionProxy.java:448)
        at com.mysql.cj.jdbc.ha.ReplicationConnectionProxy.<init>(ReplicationConnectionProxy.java:168)
        at com.mysql.cj.jdbc.ha.ReplicationConnectionProxy.createProxyInstance(ReplicationConnectionProxy.java:88)
        at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:209)
        at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:200)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:390)
        at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:372)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1126)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:834)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:527)

Workaround
Removing replication from the datasource URL "solves" the issue.

Environment

• Java: v11
• Quarkus: 1.3.0.Final
• Build tool: Gradle

[geoand]

cc @Sanne

[Sanne]

thanks! Haven't tried yet but this should be easy to fix.

[SchulteMarkus]

:-)

Well, maybe I can place a very similar topic here? Fresh from development, in the same setup

com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface com.mysql.cj.jdbc.JdbcPreparedStatement, interface com.mysql.cj.jdbc.JdbcStatement] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.,

[machi1990]

@Sanne @SchulteMarkus indeed some proxies registration were missing. I have added them here https://github.com/machi1990/quarkus/commit/4d87443d48121a8ab670d850de982dca5aedbf1f but I am still having issues regarding which I'll look at later on:

Caused by: org.hibernate.HibernateException: Unable to resume previously suspended transaction
    at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:128)
    at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:57)
    at org.hibernate.id.enhanced.TableStructure$1.getNextValue(TableStructure.java:125)
    at org.hibernate.id.enhanced.NoopOptimizer.generate(NoopOptimizer.java:40)
    at org.hibernate.id.enhanced.SequenceStyleGenerator.generate(SequenceStyleGenerator.java:523)
    at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:115)
    at org.hibernate.event.internal.DefaultPersistEventListener.entityIsTransient(DefaultPersistEventListener.java:185)
    at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:128)
    at org.hibernate.event.internal.DefaultPersistEventListener.onPersist(DefaultPersistEventListener.java:55)
    at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:102)
    at org.hibernate.internal.SessionImpl.firePersist(SessionImpl.java:710)
    ... 48 more
Caused by: java.lang.IllegalStateException: BaseTransaction.checkTransactionState - ARJUNA016051: thread is already associated with a transaction!
    at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.checkTransactionState(BaseTransaction.java:264)
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionManagerImple.resume(TransactionManagerImple.java:96)
    at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:121)

@Sanne maybe I am missing something?

[SchulteMarkus]

@machi1990 Hmm, don't know. Has there anything to be added from package com.mysql.cj.jdbc.ha?!

[machi1990]

@SchulteMarkus I have opened #8052 to fix the issue, can you check it locally if it fixes the issue for you? Thanks.

[SchulteMarkus]

@SchulteMarkus I have opened #8052 to fix the issue, can you check it locally if it fixes the issue for you? Thanks.

I will do so tomorrow probably.

[machi1990]

Thanks. You may need to build the PR locally, let me know if you'll need some guidance.

[gsmet]

To build Quarkus, just use mvn clean install -DskipTests -DskipITs then update your project to use the quarkus-bom version 999-SNAPSHOT.

[SchulteMarkus]

To build Quarkus, just use mvn clean install -DskipTests -DskipITs then update your project to use the quarkus-bom version 999-SNAPSHOT.

No success building locally (when ext4 is not enough - w00t?!)

[ERROR] Failed to generate extension doc: java.nio.file.FileSystemException: /home/markus-abcdef/git-repos/quarkus/target/asciidoc/generated/config/quarkus-hibernate-search-elasticsearch-config-group-hibernate-search-elasticsearch-runtime-config-elasticsearch-additional-backends-runtime-config.adoc: Filename is too long

But you can be sure I will re-open here once I get the chance to test the updated version.

Target release is v1.4.0? :-( I would love to have it in 1.3.1. Too mention an argument - in my opinion, this is more likely a bug (I would expect any jdbc-mysql URL to work) than a feature.

[geoand]

-DskipDocs which most likely not run the doc generation process that failed for you.

[Sanne]

Target release is v1.4.0? :-( I would love to have it in 1.3.1. Too mention an argument - in my opinion, this is more likely a bug (I would expect any jdbc-mysql URL to work) than a feature.

yes I agree, we'll consider backporting. But since 1.3.0 was already released, any new issue targets the next version - that's just the process.

[gsmet]

The fix will be part of 1.3.1.Final.

[Sanne]

thanks @gsmet !

[SchulteMarkus]

Hi guys,

I have given building Quarkus another chance. -DskipDocs does not help (-Dmaven.javadoc.skip=true neither), so I excluded Maven module extensions -> "hibernate-search-elasticsearch" (and integration-tests -> "hibernate-search-elasticsearch") from build.

Putting it all together: The latest version of Quarkus (commit https://github.com/quarkusio/quarkus/commit/9db47139f07e451bc60110fcf333b577fe056556) works fine related to this issue :-)

Thanks a lot guys for feedback, and even backporting!

[SchulteMarkus]

Yesterday I presented Quarkus including native image to my company. Would not have worked out without your fix. What a timing guys!

[Sanne]

great to hear, thanks @SchulteMarkus !

[shahmalhar]

@machi1990 Mysql failover is not working with native image with 1.5.0.Final, below is my DB connectivity URL:
quarkus.datasource.jdbc.url=jdbc:mysql://localhost:3306,localhost1:3306,localhost2:3306/cpaps

Can someone help me here?