Quarkus: Kafka not starting with JDK 14

[gsmet]

I'm still trying to get CI on JDK 14:

• [x] REST Assured
• [x] Gradle
• [ ] Now Kafka is not starting properly apparently:

Caused by: java.lang.RuntimeException: Unable to start Quarkus test resource io.quarkus.it.kafka.KafkaTestResource@c11332b
    at io.quarkus.test.common.TestResourceManager.start(TestResourceManager.java:51)
    ... 44 more
Caused by: java.lang.RuntimeException: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at io.quarkus.it.kafka.KafkaTestResource.start(KafkaTestResource.java:30)
    at io.quarkus.test.common.TestResourceManager.start(TestResourceManager.java:46)
    ... 44 more
Caused by: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
    at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:375)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
    at io.debezium.kafka.KafkaServer.startup(KafkaServer.java:220)
    at java.base/java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4780)
    at io.debezium.kafka.KafkaCluster.startup(KafkaCluster.java:250)
    at io.quarkus.it.kafka.KafkaTestResource.start(KafkaTestResource.java:28)
    ... 45 more

[gunnarmorling]

To reproduce as per @gsmet:

mvn clean install -DskipTests -DskipITs

then go into integration-tests/kafka and run

mvn clean install

I'll take a look into this.

[geoand]

Seems to me like we could get around this if Kafka doesn't start with Java 14 but just starting it in docker, no?
It's not great, but if this is the only thing holding us back from Java 14 testing I think it's worth it.

[gsmet]

I will let Gunnar handle this and tell us what's best.

There's no real hurry there.

[gunnarmorling]

Seems to me like we could get around this if Kafka doesn't start with Java 14 but just starting it in docker, no?

That's something I wanted to bring up, too. It's easily done via Testcontainers; here's how that's utilized with Debezium (i.e. Kafka Connect), you'd only need the actual Kafka part of that. So that'd definitely be an option, but in my experience it will increase execution time of these tests.

So give me some time to look into this and see whether we can get KafkaCluster running, and if not, we can fall back to Docker ("manually" or via TC).

[cescoffier]

I just did a manual test and the Kafka support works with Java 14. IT's just a test issue.

I would be keen to keep using Debezium as it is a bit faster and it much easier to configure (SASL...)

[gunnarmorling]

Just tried to run of the Debezium tests using KafkaCluster with JDK
14, and it worked fine. So it must be something with how things are
used here.

[jaikiran]

I had a look at this one. The issue here is that although the Zookeeper embedded server starts up fine (at localhost:2182) during the test startup, the org.apache.zookeeper.Zookeeper client (3.4.x version from this library[1]) ends up using a wrong host name trying to establish a connection with the Zookeeper server that's running. The reason appears to be because of a change in Java 14 introduced as part of https://bugs.openjdk.java.net/browse/JDK-8225499. As part of that change, the InetSocketAddress.toString() now includes a /<unresolved> string literal as part of hostname string (for example, in this specific case, it returns localhost/<unresolved>:2182). It so happens that the org.apache.zookeeper.Zookeeper uses (at least a couple of classes) which rely on the previous behaviour of the InetSocketAddress.toString() and don't expect the /<unresolved> part. A unresolved InetSocketAddress is first created here[2] and is then parsed here[3]. Both these classes are used by the constructor of org.apache.zookeeper.Zookeeper[4] to create a client connection manager.

A quick check doesn't show any open issues in their repo for this issue against Java 14 (I didn't dig too deep though) and I don't really see an easy way out on this one, given how deep this logic resides without a way to override (as far as I can see). No matter which address you will pass to it (even 127.0.0.1 for example), it will still internally create an unresolved InetSocketAddress first and then use the toString() to get the host name and then try resolving that address (which will end up having a /<unresolved> literal suffixed to it).

[1] https://github.com/apache/zookeeper/blob/branch-3.4/
[2] https://github.com/apache/zookeeper/blob/branch-3.4/zookeeper-server/src/main/java/org/apache/zookeeper/client/ConnectStringParser.java#L76
[3] https://github.com/apache/zookeeper/blob/branch-3.4/zookeeper-server/src/main/java/org/apache/zookeeper/client/StaticHostProvider.java#L135
[4] https://github.com/apache/zookeeper/blob/branch-3.4/zookeeper-server/src/main/java/org/apache/zookeeper/ZooKeeper.java#L447

[jaikiran]

Hello @gunnarmorling,

Just tried to run of the Debezium tests using KafkaCluster with JDK 14, and it worked fine. So it must be something with how things are used here.

Which version of Kafka and Zookeeper (embedded) libraries is that using?

[jaikiran]

Which version of Kafka and Zookeeper (embedded) libraries is that using?

I just checked myself. So looks like Debezium is on 3.5.x of zookeeper[1], unlike Quarkus which is on 3.4.x. Looking at the 3.5 branch of Zookeeper upstream, a lot seems to have changed in this area of the code (especially the StaticHostProvider) and that might explain why it passes there. So maybe we should try upgrading to that version here?

[1] https://github.com/debezium/debezium/blob/master/pom.xml#L62

[jaikiran]

I tested a run against my GH actions with the upgraded version of Zookeeper and the tests passed (against 8, 11 and 14 Java version). So I've opened https://github.com/quarkusio/quarkus/pull/8229 to fix this issue.

[geoand]

Thanks a lot for your analysis @jaikiran!