Quarkus: Native build of Kafka Streams fails at runtime with class RocksDBException not found

[andreas-eberle]

Describe the bug
The native build of kafka streams aggregator fails at runtime with the exception

2020-02-07 10:42:15,466 INFO  [org.apa.kaf.str.KafkaStreams] (pool-1-thread-1) stream-client [temperature-aggregator-9a921186-0418-4988-bc9e-e962abcaaeeb] State transition from CREATED to REBALANCING
RocksDBExceptionJni::ThrowNew/class - Error: unexpected exception!
Exception in thread "temperature-aggregator-9a921186-0418-4988-bc9e-e962abcaaeeb-GlobalStreamThread" java.lang.NoClassDefFoundError: Lorg/rocksdb/RocksDBException;
        at com.oracle.svm.jni.functions.JNIFunctions.FindClass(JNIFunctions.java:328)
        at org.rocksdb.RocksDB.open(RocksDB.java)
        at org.rocksdb.RocksDB.open(RocksDB.java:286)
        at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:72)
        at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:180)
        at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:202)
        at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
        at org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
        at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.lambda$init$0(MeteredKeyValueStore.java:107)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.measureLatency(MeteredKeyValueStore.java:260)
        at org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:105)
        at org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:136)
        at org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
        at org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
        at org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
        at org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
2020-02-07 10:42:15,550 INFO  [org.apa.kaf.cli.Metadata] (kafka-producer-network-thread | temperature-aggregator-9a921186-0418-4988-bc9e-e962abcaaeeb-StreamThread-1-producer) [Producer clientId=temperature-aggregator-9a921186-0418-4988-bc9e-e962abcaaeeb-StreamThread-1-producer] Cluster ID: SKpikR8ISFaFesCFNFz8Lg

Expected behavior
Native build works at runtime.

Actual behavior
The aggregator container fails immediately after startup with the exception shown above. The producer container works normally.

This happens with graalvm 19.2.1, 19.3.1-java8 and 19.3.1-java11.

To Reproduce
Steps to reproduce the behavior:

1. git clone https://github.com/quarkusio/quarkus-quickstarts
2. cd quarkus-quickstarts/kafka-streams-quickstart
3. ./mvnw clean package -Pnative -Dnative-image.container-runtime=docker -Dquarkus.native.builder-image="quay.io/quarkus/ubi-quarkus-native-image:19.2.1"
   (change the image of graalvm if you like to, but didn't make a change for me)
4. export QUARKUS_MODE=native
5. docker-compose up -d --build
6. Sometimes you need to do docker-compose up -d again, so the kafka container is actually ready before the producer and aggregator try to connect to it.
7. docker ps
8. docker logs <ID of the aggregator contaier>
9. you should see the exception

NOTE I actually had to change the docker-compose file on Ubuntu to this docker-compose.zip. On MacOsX the original docker-compose file worked fine.

Configuration
Exactly as in the quickstarts.

Environment (please complete the following information):
This failed on Ubuntu18.04 as well as on MacOsX

• Output of uname -a or ver:
  Linux predator 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Output of java -version:

openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1, mixed mode, sharing)

• GraalVM version (if different from Java):
  Used 19.2.1, 19.3.1-java8 and 19.3.1-java11 via docker-build
• Quarkus version or git rev:
  1.2.0.Final

[wfrank2509]

Please see also https://github.com/quarkusio/quarkus/issues/6667 which seems to be related.

[andreas-eberle]

I was able to get the quickstart example to start with this jniconfig.json (package to zip due to github restrictions) put into the resurces directory of the the aggregator.

Additionally, I added

<additionalBuildArgs>
  <additionalBuildArg>-H:JNIConfigurationFiles=jniconfig.json</additionalBuildArg>
</additionalBuildArgs>

in the aggregator pom.xml to look like

        <plugins>
          <plugin>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-maven-plugin</artifactId>
            <version>${quarkus.version}</version>
            <executions>
              <execution>
                <goals>
                  <goal>native-image</goal>
                </goals>
                <configuration>
                  <enableHttpUrlHandler>true</enableHttpUrlHandler>
                  <enableJni>true</enableJni>
                  <additionalBuildArgs>
                    <additionalBuildArg>-H:JNIConfigurationFiles=jniconfig.json</additionalBuildArg>
                  </additionalBuildArgs>
                </configuration>
              </execution>
            </executions>
          </plugin>

[andreas-eberle]

I tried to add the classes in the kafka-streams extension by adding the following two lines to KafkaStreamsProcessor.registerCompulsoryClasses() but it does not seem to take any effect.

        reflectiveClasses.produce(new ReflectiveClassBuildItem(true, false, false, RocksDBException.class));
        reflectiveClasses.produce(new ReflectiveClassBuildItem(true, false, false, Status.class));

@gunnarmorling: I saw you worked on this method the last. Do you have an idea what I'm doing wrong?

[wfrank2509]

Also tried a slightly different thing to register the Reflection Classes in aggregator module of the kafka-streams quickstart:

application.properties:
quarkus.native.additional-build-args=-H:ReflectionConfigurationFiles=reflection-config.json

Add File reflection-config.json to main/resources:

[
  {
    "name" : "org.rocksdb.RocksDBException",
    "allDeclaredConstructors" : true,
    "allPublicConstructors" : true,
    "allDeclaredMethods" : true,
    "allPublicMethods" : true,
    "allDeclaredFields" : true,
    "allPublicFields" : true
  },
  {
    "name" : "org.rocksdb.Status",
    "allDeclaredConstructors" : true,
    "allPublicConstructors" : true,
    "allDeclaredMethods" : true,
    "allPublicMethods" : true,
    "allDeclaredFields" : true,
    "allPublicFields" : true
  }
]

Unfortunately same result when running the native build container:
java.lang.NoClassDefFoundError: Lorg/rocksdb/RocksDBException;

[wfrank2509]

The image classpath configured for the native build contains the rocksdb jar.

docker run -v /Users/wfrank/quarkus-quickstarts/kafka-streams-quickstart/aggregator/target/kafka-streams-quickstart-aggregator-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java8 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -H:ReflectionConfigurationFiles=reflection-config.json --verbose --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -H:+JNI -jar kafka-streams-quickstart-aggregator-1.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http --no-server -H:-UseServiceLoaderFeature -H:+StackTrace kafka-streams-quickstart-aggregator-1.0-SNAPSHOT-runner
Apply jar:file:///project/lib/io.netty.netty-handler-4.1.44.Final.jar!/META-INF/native-image/io.netty/handler/native-image.properties
Apply jar:file:///project/lib/io.netty.netty-common-4.1.44.Final.jar!/META-INF/native-image/io.netty/common/native-image.properties
Apply jar:file:///project/lib/io.netty.netty-buffer-4.1.44.Final.jar!/META-INF/native-image/io.netty/buffer/native-image.properties
Apply jar:file:///project/lib/io.netty.netty-transport-4.1.44.Final.jar!/META-INF/native-image/io.netty/transport/native-image.properties
Apply jar:file:///project/lib/io.netty.netty-codec-http-4.1.44.Final.jar!/META-INF/native-image/io.netty/codec-http/native-image.properties
Apply jar:file:///project/lib/io.netty.netty-codec-http2-4.1.44.Final.jar!/META-INF/native-image/io.netty/codec-http2/native-image.properties
Apply jar:file:///project/lib/org.eclipse.yasson-1.0.6.jar!/META-INF/native-image/org.eclipse/yasson/native-image.properties
Apply jar:file:///project/lib/org.graalvm.sdk.graal-sdk-19.3.1.jar!/META-INF/native-image/org.graalvm.polyglot/native-image.properties
Executing [
/opt/graalvm/jre/bin/java \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-Dtruffle.TrustAllTruffleRuntimeProviders=true \
-Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
-Dgraalvm.ForcePolyglotInvalid=true \
-Dgraalvm.locatorDisabled=true \
-d64 \
-XX:-UseJVMCIClassLoader \
-XX:+UseJVMCINativeLibrary \
-Xss10m \
-Xms1g \
-Xmx10052848840 \
-Duser.country=US \
-Duser.language=en \
-Dorg.graalvm.version=19.3.1 \
-Dorg.graalvm.config=CE \
-Dcom.oracle.graalvm.isaot=true \
-Djvmci.class.path.append=/opt/graalvm/jre/lib/jvmci/graal.jar \
-javaagent:/opt/graalvm/jre/lib/svm/builder/svm.jar \
-Djdk.internal.lambda.disableEagerInitialization=true \
-Djdk.internal.lambda.eagerlyInitialize=false \
-Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \
-Dsun.nio.ch.maxUpdateArraySize=100 \
-Djava.util.logging.manager=org.jboss.logmanager.LogManager \
-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory \
-Dvertx.disableDnsResolver=true \
-Dio.netty.leakDetection.level=DISABLED \
-Dio.netty.allocator.maxOrder=1 \
-Xbootclasspath/a:/opt/graalvm/jre/lib/boot/graaljs-scriptengine.jar:/opt/graalvm/jre/lib/boot/graal-sdk.jar \
-cp \
 <... multiple jars ...> \
com.oracle.svm.hosted.NativeImageGeneratorRunner \
-watchpid \
1 \
-imagecp \
 <... multiple jars ...>:/project/lib/io.quarkus.quarkus-kafka-streams-999-SNAPSHOT.jar:/project/lib/org.rocksdb.rocksdbjni-5.18.3.jar:/project/kafka-streams-quickstart-aggregator-1.0-SNAPSHOT-runner.jar \
-H:Path=/project \
-H:ClassInitialization=:build_time \
-H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime \
-H:+JNI \
-H:Class=io.quarkus.runner.GeneratedMain \
-H:ClassInitialization=io.netty.handler.ssl.util.ThreadLocalInsecureRandom:run_time \
-H:ClassInitialization=io.netty.util.AbstractReferenceCounted:run_time \
-H:ClassInitialization=io.netty.buffer.PooledByteBufAllocator:run_time,io.netty.buffer.ByteBufAllocator:run_time,io.netty.buffer.ByteBufUtil:run_time,io.netty.buffer.AbstractReferenceCountedByteBuf:run_time \
-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/transport/reflection-config.json \
-H:ClassInitialization=io.netty.handler.codec.http.HttpObjectEncoder:run_time,io.netty.handler.codec.http.websocketx.WebSocket00FrameEncoder:run_time,io.netty.handler.codec.http.websocketx.extensions.compression.DeflateDecoder:run_time \
-H:ClassInitialization=io.netty:build_time \
-H:ClassInitialization=io.netty.handler.codec.http2.Http2CodecUtil:run_time,io.netty.handler.codec.http2.Http2ClientUpgradeCodec:run_time,io.netty.handler.codec.http2.Http2ConnectionHandler:run_time,io.netty.handler.codec.http2.DefaultHttp2FrameWriter:run_time \
-H:IncludeResourceBundles=yasson-messages \
-H:ClassInitialization=org.graalvm.polyglot:build_time \
-H:FallbackThreshold=0 \
-H:+ReportExceptionStackTraces \
-H:-AddAllCharsets \
-H:EnableURLProtocols=http \
-H:-UseServiceLoaderFeature \
-H:+StackTrace \
-H:CLibraryPath=/opt/graalvm/jre/lib/svm/clibraries/linux-amd64 \
-H:ReflectionConfigurationFiles=/project/reflection-config.json \
-H:Name=kafka-streams-quickstart-aggregator-1.0-SNAPSHOT-runner
]

[gunnarmorling]

Do you have a reproducer handy? Will the kafka-streams-quickstart example
expose this issue when used with the latest Quarkus version?

>

[wfrank2509]

Yes ... I am using the plain kafka-streams-quickstart example to reproduce the issue.
And switched to the current quarkus master (SNAPSHOT) where the issue to resolve the missing 'librocksdbjni-linux64.so' was solved (This error hits you first if you try to run the kafka-streams quickstarts with 1.2.0-Final. But luckily there is already a merged PR for this (#6952)).

[andreas-eberle]

Also, I had issues with the provided docker-compose file causing the containers to not find each other due to wrong names. With the compose file I attached in my initial post, the containers can at least find each other and connect to kafka.

[gunnarmorling]

Ok, I'll try and take a look next week. I reckon one of the recent
GraalVM updates is triggering this.

[gunnarmorling]

For the time being, you can add the following class to the _aggregator_ project to resolve the issue:

package org.acme.quarkus.sample.kafkastreams;

import org.graalvm.nativeimage.hosted.Feature;
import org.rocksdb.RocksDBException;
import org.rocksdb.Status;

import com.oracle.svm.core.annotate.AutomaticFeature;
import com.oracle.svm.core.jni.JNIRuntimeAccess;

@AutomaticFeature
class JNIRegistrationFeature implements Feature {

    @Override
    public void beforeAnalysis(BeforeAnalysisAccess access) {
        JNIRuntimeAccess.register(RocksDBException.class);
        JNIRuntimeAccess.register(RocksDBException.class.getConstructors());
        JNIRuntimeAccess.register(Status.class);
        JNIRuntimeAccess.register(Status.class.getDeclaredConstructors());
    }
}

I'll send a PR for taking care of this in the Quarkus Kafka Streams extension shortly.

[wfrank2509]

Thanks @gunnarmorling for the workaround!

Just wondering why the reflection config files and additions of the missing classes in the extension did not have any effect?

[gunnarmorling]

@wfrank2509, sorry, had missed your reply. There's a difference between reflective access and JNIRuntimeAccess (which is what I'm using above).

[gunnarmorling]

PR provided at https://github.com/quarkusio/quarkus/pull/7329.