Xgboost: [jvm-packages] java.lang.NoClassDefFoundError: Could not initialize class ml.dmlc.xgboost4j.java.XGBoostJNI

[edumucelli]

Environment info

Operating System: Debian 8 "Jessie"
Compiler: gcc (Debian 6.3.0-6) 6.3.0 20170205
xgboost version used: master commit 03e213c7cdb9841d3bd482179400d6a184633147

Hi,

I am using XGBoost4j for prediction only and everything went OK on my local machine. For instance, you can find attached result of the mvn package within local.mvn.package.txt. However, it throws the following error on the test environment, which are containers running Debian Jessie:

java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: Could not initialize class ml.dmlc.xgboost4j.java.XGBoostJNI
java.lang.NoClassDefFoundError: Could not initialize class ml.dmlc.xgboost4j.java.XGBoostJNI
    at ml.dmlc.xgboost4j.java.Booster.init(Booster.java:455)
    at ml.dmlc.xgboost4j.java.Booster.<init>(Booster.java:47)
    at ml.dmlc.xgboost4j.java.Booster.loadModel(Booster.java:85)
    at ml.dmlc.xgboost4j.java.XGBoost.loadModel(XGBoost.java:57)
    at com.company.ml.api.PredictorFactory.createXGBoost(PredictorFactory.java:45)
    at com.company.ml.api.PredictorFactory.createPredictor(PredictorFactory.java:31)
    at com.company.ml.api.MachineLearning.newXGB(MachineLearning.java:12)
    at com.company.ml.api.MachineLearningITest.machineLearning_TestFlowWithMultiplicationFunctionScoreIsOK(MachineLearningITest.java:55)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I have read the issue #2497, but unfortunately the commit 2cb51f7 mentioned by @superbobry did not helped. I am wondering if something is missing on the container. During its boot it installs certain packages, e.g., cmake, etc. You can find a log of its boot within the agent.build.txt attached.

And that is what I have locally:

ldd libxgboost4j.so
    linux-vdso.so.1 (0x00007fff923ad000)
    libjvm.so => /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so (0x00007f4b51891000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4b514ee000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4b511ea000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f4b50fbd000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4b50da6000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4b50b87000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4b507e9000)
    /lib64/ld-linux-x86-64.so.2 (0x000055e5fdb84000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4b505e5000)

Does something else should be installed on the container? I guess everything needed is already on the generated package but I may be missing something.

Edit: Looks like JNI installation is OK on the container, I have enabled verbosity with JAVA_OPTS="-verbose:jni" and the at least it shows no errors. It is bizarre it cannot find XGBoostJNI file yet.

Thank you in advance.

[edumucelli]

I have finally solved this problem. First I have spotted the following error among the tests I have wrote:

java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: /tmp/libxgboost4j7161479872188549441.so: libgomp.so.1: cannot open shared object file: No such file or directory
java.lang.UnsatisfiedLinkError: /tmp/libxgboost4j7161479872188549441.so: libgomp.so.1: cannot open shared object file: No such file or directory
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:66)
    at ml.dmlc.xgboost4j.java.NativeLibLoader.smartLoad(NativeLibLoader.java:152)
    at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:40)
    at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
    at ml.dmlc.xgboost4j.java.Booster.init(Booster.java:455)
    at ml.dmlc.xgboost4j.java.Booster.<init>(Booster.java:47)
    at ml.dmlc.xgboost4j.java.Booster.loadModel(Booster.java:85)
    at ml.dmlc.xgboost4j.java.XGBoost.loadModel(XGBoost.java:57)
    at com.blablacar.ml.api.PredictorFactory.createXGBoost(PredictorFactory.java:45)
    at com.blablacar.ml.api.PredictorFactory.createPredictor(PredictorFactory.java:31)
    at com.blablacar.ml.api.MachineLearning.newXGB(MachineLearning.java:12)

I have then installed libgomp1, i.e., apt-get install -y libgomp1

Then, on the next run another error was thrown

java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: /tmp/libxgboost4j7490659266172391676.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by /tmp/libxgboost4j7490659266172391676.so)
java.lang.UnsatisfiedLinkError: /tmp/libxgboost4j7490659266172391676.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by /tmp/libxgboost4j7490659266172391676.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:66)
    at ml.dmlc.xgboost4j.java.NativeLibLoader.smartLoad(NativeLibLoader.java:152)
    at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:40)
    at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
    at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:128)
    at com.blablacar.ml.api.XGBoostPredictor.predict(XGBoostPredictor.java:45)

This one was a harder one as I had built libxgboost.so locally with Glibc (libc6) version 2.24. Thus on the container it required at least 2.23 as shown in the error. To finally solve the problem I had to compile the xgboost locally with Glibc that matchs the container's one, i.e., 2.19.

So if you are facing such error, be aware of the missing libraries, such as libgomp and to match the version in which you have compiled xgboost.

[edumucelli]

It would have been nice to have libgomp and other required libraries documented somewhere, e.g., https://xgboost.readthedocs.io/en/latest/jvm/index.html. Maybe I have missed in the docs the list of requirements up to the compiled-level xgboost library.

[CodingCat]

did you disable test when doing mvn package?

[edumucelli]

@CodingCat no, every build I made was using the tests. The problem is, the required libs are easily installable on the local environment. But my goal is to use XGboost in production, which is not always accessible. So the ideal is that the library should be ready to use on production environments where very few people have access, e.g., containers. Building locally is OK, but since other libraries are required by the xgboost4j.so, when the jar goes to the production environment, it breaks on Linux distributions that have only core libraries installed.

[belkacemlahouel]

For your information, you may solve it using this: https://stackoverflow.com/questions/16605623/where-can-i-get-a-copy-of-the-file-libstdc-so-6-0-15

sudo apt-get install libstdc++6
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade