Deeplearning4j: UnsatisfiedLinkError: no jnind4jcpu (Raspberry PI 4 with 1.0.0-beta6)

[ticofab]

Issue Description

Trying to train a model, exception thrown. Maybe the solution is just installing missing packages?
java.lang.UnsatisfiedLinkError: no jnind4jcpu in java.library.path

Version Information

Please indicate relevant versions, including, if relevant:

• Deeplearning4j 1.0.0-beta6
• OpenJDK 11.0.6 2020-01-14
• Raspbian GNU/Linux 10 (buster)

Additional Information

20:25:01 Loaded [CpuBackend] backend
Uncaught error from thread [default-akka.actor.default-dispatcher-4]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[default]
java.lang.ExceptionInInitializerError
    at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.<init>(NativeOpExecutioner.java:91)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at java.base/java.lang.Class.newInstance(Class.java:584)
    at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5215)
    at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5129)
    at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:226)
    at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:579)
    at io.ticofab.piai.learning.Settings$.getNetworkConfiguration(Settings.scala:56)
    at io.ticofab.piai.learning.ModelEvaluator$.$anonfun$train$1(ModelEvaluator.scala:54)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.util.Try$.apply(Try.scala:213)
    at io.ticofab.piai.learning.ModelEvaluator$.train(ModelEvaluator.scala:21)
    at io.ticofab.piai.learning.ManualTrainer.<init>(ManualTrainer.scala:11)
    at io.ticofab.piai.Supervisor.$anonfun$new$1(Supervisor.scala:23)
    at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:90)
    at akka.actor.Props.newActor(Props.scala:226)
    at akka.actor.ActorCell.newActor(ActorCell.scala:609)
    at akka.actor.ActorCell.create(ActorCell.scala:635)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:509)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:531)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:294)
    at akka.dispatch.Mailbox.run(Mailbox.scala:229)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:242)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: http://nd4j.org/getstarted.html
    at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:112)
    at org.nd4j.nativeblas.NativeOpsHolder.<clinit>(NativeOpsHolder.java:35)
    ... 31 more
Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcpu in java.library.path: [/usr/java/packages/lib, /usr/lib/arm-linux-gnueabihf/jni, /lib/arm-linux-gnueabihf, /usr/lib/arm-linux-gnueabihf, /usr/lib/jni, /lib, /usr/lib]
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2660)
    at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:829)
    at java.base/java.lang.System.loadLibrary(System.java:1870)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1543)
    at org.bytedeco.javacpp.Loader.load(Loader.java:1192)
    at org.bytedeco.javacpp.Loader.load(Loader.java:1042)
    at org.nd4j.nativeblas.Nd4jCpu.<clinit>(Nd4jCpu.java:13)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:315)
    at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:85)
    ... 32 more
Caused by: java.lang.UnsatisfiedLinkError: /home/pi/.javacpp/cache/nd4j-native-1.0.0-beta6-linux-armhf.jar/org/nd4j/nativeblas/linux-armhf/libjnind4jcpu.so: /home/pi/.javacpp/cache/opencv-4.1.2-1.5.2-linux-armhf.jar/org/bytedeco/opencv/linux-armhf/libgomp.so.1: version `GOMP_4.0' not found (required by /home/pi/.javacpp/cache/nd4j-native-1.0.0-beta6-linux-armhf.jar/org/nd4j/nativeblas/linux-armhf/libnd4jcpu.so)
    at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
    at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2430)
    at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2487)
    at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2684)
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2617)
    at java.base/java.lang.Runtime.load0(Runtime.java:767)
    at java.base/java.lang.System.load(System.java:1834)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1493)
    ... 38 more

[saudet]

@sshepel It looks like you're using a too recent version of GCC here:
https://github.com/KonduitAI/docker/blob/master/manifests/jenkins-agent/centos/7/armhf/Dockerfile#L93
This should be raspberrypi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/arm-linux-gnueabihf instead.

[sshepel]

k, will update it ASAP

[sshepel]

Docker image has been updated, but we have failing build on master... Will let you guys know, when you can try snapshots.

[saudet]

Probably due to OpenBLAS 0.3.8 <- @AlexDBlack

[AlexDBlack]

@sshepel Got a link to the CI logs for this? Or is it one of these known issues?
https://github.com/eclipse/deeplearning4j/issues/8748

[saudet]

@AlexDBlack I think this is just about eclipse/deeplearning4j not being up-to-date and failing to build.

[saudet]

@vb216 Do you think that the toolchain for GCC 4.9 in https://github.com/raspberrypi/tools/blob/master/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/ is also good?

[saudet]

Actually, that one looks like the recommended toolchain:
https://github.com/raspberrypi/tools/blob/master/arm-bcm2708/arm-linux-gnueabihf

@sshepel I'll try to update the presets to use that toolchain, and if that works, we should be able to leave libnd4j as is.

[vb216]

@saudet not sure, unfortunately Pi change their distro quite fundamentally fairly often too, can be a pain

[saudet]

@ticofab The presets now get built with GCC 4.9 as well, so it should work now. Please give it a try with 1.0.0-SNAPSHOT: https://deeplearning4j.konduit.ai/config/config-snapshots

[ticofab]

I am really sorry I haven't been able to setup the system to reproduce this. I would close it for now. Thank you for you attention and efforts.