Deeplearning4j: libopenblas_nolapack.so.0: cannot open shared object file: No such file or directory

[sergmain]

Hello,

I've just tried to run my application on beta2 and I've got the follow exception:
Caused by: java.lang.UnsatisfiedLinkError: /app/.javacpp/cache/openblas-0.3.0-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libjniopenblas_nolapack.so: libopenblas_nolapack.so.0: cannot open shared object file: No such file or directory

You can find full stacktrace here - https://gist.github.com/sergmain/0685cda1456721595637def8ca347662

Few days ago, I opened an issue https://github.com/deeplearning4j/deeplearning4j/issues/6083
Since then the issue was fixed and beta2 was released.

I rolled back to beta and my application started to work.

there is stub project for reproducing this problem on heroku
https://github.com/sergmain/dl4j-uber-jar
It doesn't contain actual keras model in this repo but you can use any.

Summary:
beta - working
beta2 - not working
target OS - Heroku's PaaS
target pratform for DL4J is specified in /.mvn/jvm.config

[saudet]

Could you set the "org.bytedeco.javacpp.logger" system property to "slf4j", with a logger set to debug, and post the log again? Thanks!

[sergmain]

Here - https://gist.github.com/sergmain/d42dd786b3d9325f94b53dee376f8d19

Btw, developers from bytedeco should rewrite its logging system. Using system property to configure application is a bad practice.

[agibsonccc]

@sergmain We maintain bytdececo too. We also use system properties to configure nd4j as well. I'm not sure where you got that from, but you just dissed a good number of the top java libraries used by more than half the ecosystem. When you've built your successful library, you can show us how to do it :).

We're trying to help you here. The reply is completely unrelated to helping you with your problem.

[sergmain]

There are such things as JEE aplication servers, servlet containers, portals, OSGi container (not sure about it, actualy) which use single JVM to run numbers af application. In such environment using system properties is a bad practice.

About relating to topic. @saudet asked me to make some sets and rerun test. It took time to realize what does it mean "set the "org.bytedeco.javacpp.logger" system property to "slf4j" and even I had to check out completely repo of javacpp and look in the code to understand how to do that.

[agibsonccc]

@sergmain yes we run things there too.Most libraries do not make the assumptions about those things anymore. There are also desktop application with plugins. Generally, system properties are for once per startup global state. In practice this isn't a problem and has more benefits than problems (and frankly as a library author myself, there is just no general purpose way to do this, the alternative is a bunch of random 1 off configuration files. Again, we are not changing it and your suggestions are not relevant.

Beyond that, instead of complaining, file an issue on the docs and I'm sure he'd be happy to update it if you have a concrete suggestion on making things better. You're not helping us help you by acting the way you are. Please ease up and just let us do what we need to do to help you.

[saudet]

@sergmain Yes, this is a global setting, it's not meant to be changed per application. In any case, JNI does not support loading from multiple class loaders, so JavaCPP and ND4J need to be loaded by a shared class loader anyway.

It seems there is something strange with the URL class. These 2 lines:

Extracting jar:file:/app/target/dl4j-uber-jar.jar!/BOOT-INF/lib/mkl-dnn-0.15-1.4.2-linux-x86_64.jar!/org/bytedeco/javacpp/linux-x86_64/libmklml_intel.so
Loading /app/.javacpp/cache/mkl-dnn-0.15-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libmklml_intel.so

Should actually be:

Extracting jar:file:/app/target/dl4j-uber-jar.jar!/BOOT-INF/lib/mkl-dnn-0.15-1.4.2-linux-x86_64.jar!/org/bytedeco/javacpp/linux-x86_64/libmklml_intel.so#libopenblas_nolapack.so.0
Loading /app/.javacpp/cache/mkl-dnn-0.15-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libopenblas_nolapack.so.0

What's the exact version of the JDK as returned by java -version?

[sergmain]

@saudet , properties is here - https://gist.github.com/sergmain/9263e5f56254e19434878714515a74bb

heroku project on github as well, so you can request the help directly form heroku team
https://github.com/heroku

[saudet]

There doesn't seem to be anything funny with this environment, not sure it's an issue with Heroku. What happens if you use your uber JAR locally with a simple main() method? Does it succeed?

[sergmain]

Yes, it's working, but my dev enviroment is windows. I'd like to point out to the fact that 1.0.0-beta is working in heroku.

[saudet]

1.0.0-beta2 should also work the same with OpenBLAS. That will happen with you exclude the MKL-DNN dependency.

[sergmain]

idk what does that mean - "That will happen with you exclude the MKL-DNN dependency."
project which I use for testing is here https://github.com/sergmain/dl4j-uber-jar

[saudet]

org.nd4j.linalg.factory.Nd4j.create(1) works for me on Linux with the libs it bundles. Might be related to Spring then. Could you try to load JavaCPP and ND4J as shared libraries?

[agibsonccc]

@sergmain could you try the uber jar with shade plugin instead maybe? I know what we're showing you here for troubleshooting is new. If you don't want to worry about some of the things sam is telling you, try doing it the way we do it in the standalone examples project: https://github.com/deeplearning4j/dl4j-examples/blob/master/standalone-sample-project/pom.xml

[saudet]

In any case, if MKL-DNN isn't a concern, we can exclude it from your project like this in the pom.xml file:

        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.backend}</artifactId>
            <version>${nd4j.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.bytedeco.javacpp-presets</groupId>
                    <artifactId>mkl-dnn</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

And it will work just like 1.0.0-beta.

[sergmain]

@saudet with excluded mkl, the result is the same
https://gist.github.com/sergmain/2f1e719fc74833a056577c650f4f1314

[saudet]

@sergmain Could you also post the log from JavaCPP?

[saudet]

Also, does this happen only with Spring? Or does it happen with other containers or just raw Java on Heroku's PaaS?

[sergmain]

@saudet the last gist contains log from javacpp. Or you need other log info?

I don't use dl4j with any other containers nor PaaS, so idk. btw, on windows it's working.
"Also, does this happen only with Spring?" - my application is Spring based only.

[saudet]

So it works with Spring on Windows?

[herofe]

I'm having the same problem, running with spring framework and beta2 on a docker instance based on amazon linux 2018.03 with java8.

Not solved yet, but workaround for a testing instances for me was to spin up a normal ec2 instance with same OS as my docker instance and run the application, look in the directory .javacpp and from there copy the missing library files from the ec2 instance to /usr/lib on the docker instance before starting the app.

For me the missing libs where;
libopenblas_nolapack.so.0
libmklml_intel.so
libiomp5.so

All found in .javacpp/cache/lib, must be some packages not included on my docker instance that are installed on the ec2 instance?

[saudet]

@borkox observed the same issue with Spring on Windows, so it looks like this issue is related to Spring only. Is anyone able to reproduce outside Spring?

[herofe]

I managed to corner the problem a bit more.
I'm running same code with spring on two different amazon instances, and its failing on one and working on one.

The difference is installed cuda and intel mkl-dnn on the working instance.

For beta2 I also had to exclude mkl-dnn in the pom as indicated.

Working amazon ami: Deep Learning Base AMI (Amazon Linux) Version 9.0 (ami-8abca260)
Not working amazon ami: amzn-ami-hvm-2018.03.0.20180811-x86_64-gp2 (ami-047bb4163c506cd98)

[saudet]

There are other differences like glibc 2.17... Is there a Docker image out there for which it fails to work properly? That would be the easiest way to debug this.

[herofe]

Yes. This Docker is not working for me.

FROM amazonlinux:2018.03

RUN \
yum -y install java-1.8.0 && \
yum -y remove java-1.7.0

[saudet]

@herofe Thanks! Would you also have a small project that I can execute in it?

[herofe]

Nothing that I can share out of the box. Would need to pull out the pom and one or two java file in a new project and test if that fails. Can help with that next week if you need it.

[saudet]

There's https://github.com/sergmain/dl4j-uber-jar, I wonder what I'd have to run exactly to get it to fail...

[borkox]

I think I found the reason.
I started a debug session like this:
"c:\Program Files\Java\jdk1.8.0_60\bin\java.exe" -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -jar target\rltrader-1.0-SNAPSHOT.jar

Then looking at class org.bytedeco.javacpp.Loader.class method static String load(Class cls, Properties properties, boolean pathsFirst) I noticed that 2 two DLL libraries are loaded ok, but the third isnt loaded because the paths of the extracted libraries are not correct. I will share images.

[saudet]

Yes, it's not extracted correctly, but the cause of why that happens is unclear. It sounds like the container is hijacking the java.net.URL class and replacing it with a buggy version that doesn't support fragments.

[sergmain]

@saudet Maybe this is important info about heroku environment:
"Why are my file uploads missing/deleted?"
https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted

That's why I create a temp model file in ./target in my dl4j-user-jar sample.
https://github.com/sergmain/dl4j-uber-jar/blob/a903ce7abffeb5dd1549a4a579fd5f544cd6425d/src/main/java/com/xxx/ModelService.java#L26

[borkox]

Samuel, I was not sure if this is correct, see my picture:

[borkox]

Well, at least the next picture sounds strange to me

How comes that obenblas lib is resolved to mklml.dll
The long line urls which is not visible on the screenshot is:
file:/D:/work/rltrader-copy/rltrader/target/rltrader-1.0-SNAPSHOT.jar!/BOOT-INF/lib/openblas-0.3.0-1.4.2-windows-x86_64.jar!/org/bytedeco/javacpp/windows-x86_64/libopenblas.dll

[sergmain]

with -Dorg.bytedeco.javacpp.cachedir=./target/.javacpp the error is the same

[sergmain]

list of actual libraries:
https://gist.github.com/sergmain/5c4eb435af43e2dabc89e4113ccbb4b3

[borkox]

So I think I found the problem, but let Samuel confirm that. Actually I found 2 problems.

Whet it wants to load the libopenblas it sees that lib is already loaded from some other place.

So it depends how libs are loaded because libopenblas_nolapack is present into two folders:
C:\Users\borkox.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64
C:\Users\borkox.javacpp\cacheopenblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64

So if I change that in runtime and the file is resolved properly there is another problem, that the dependent libraries cannot be loaded:

See the three questionmarks at the bottom.

[sergmain]

libs with ldd info
https://gist.github.com/sergmain/18cd2138e837220839e416025e4601cd

./target/.javacpp/hdf5-1.10.2-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libjnihdf5.so
libhdf5_hl_cpp.so.100 => /app/./target/.javacpp/hdf5-1.10.2-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libhdf5_hl_cpp.so.100 (0x00007f11d418d000)
libhdf5_hl.so.100 => /app/./target/.javacpp/hdf5-1.10.2-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libhdf5_hl.so.100 (0x00007f11d3f69000)
libhdf5_cpp.so.102 => /app/./target/.javacpp/hdf5-1.10.2-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libhdf5_cpp.so.102 (0x00007f11d3cfd000)
libhdf5.so.101 => /app/./target/.javacpp/hdf5-1.10.2-1.4.2-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libhdf5.so.101 (0x00007f11d3737000)

Is that ok? - /app/./target/.javacpp/...

[saudet]

@sergmain @borkox When you run it outside Spring, it runs correctly, right? So that's how it should work inside Spring as well, but it doesn't, why? What's the difference?

[sergmain]

"When you run it outside Spring, it runs correctly, right?:, Nop
ldd(1) - Linux man page
In the usual case, ldd invokes the standard dynamic linker (see ld.so(8)) with the LD_TRACE_LOADED_OBJECTS environment variable set to 1, which causes the linker to display the library dependencies.
https://linux.die.net/man/1/ldd

[borkox]

Non deterministic behaviour can happen when you iterate values from a map or when you iterate keys from a map.

[borkox]

It runs well outside spring, but we wouldn't spend this time to debug if we liked to void spring.

[saudet]

@sergmain Ok, even better, do you have a Docker image and a small project that I could use to debug this here?

@borkox Map, which map?

[sergmain]

@saudet Nop, I don't have any

[borkox]

Samuel, I was talking in principle that non deterministic behaviour can come from several different places, like iterating a map, an order in a packed ZIP file, not a concrete thing for our case. If you see my screenshots, I pointed out that "libopenblas it sees that lib is already loaded from some other place". Isn't that the track we should follow?

[sergmain]

@saudet I think, bug is here:
https://github.com/bytedeco/javacpp/blob/c50d69f896332e6817cb371fdfc43b86fbbedc85/src/main/java/org/bytedeco/javacpp/Loader.java#L1141

You should do it in 2-way:

• exatract all assets, create symbolic link
• load all assets

[saudet]

@sergmain That's not related to this issue, but if you would like to change this, feel free to send a pull request and let's evaluate it.

[borkox]

example.zip

I created a small project which is doing nothing but is failing with unsatisfied linkage error. Samuel can use this to debug in windows environment. I deleted the target folder, because it was 800 MB. Samuel, if you need the built jar I will attach it here. So there can be unnecessary libs, I am sorry , but I just deleted everything from my original failing project and just put static initializer to trigger the error.

I hope will help to debug the issue.

The main class is extremely simple:

package com.example;


import org.nd4j.linalg.api.buffer.DataBuffer;
import org.nd4j.linalg.factory.Nd4j;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class App implements CommandLineRunner {

    static{
        Nd4j.EPS_THRESHOLD = 1e-7;
        //Double precision for the gradient checks.
        // See also http://nd4j.org/userguide.html#miscdatatype
        Nd4j.setDataType(DataBuffer.Type.DOUBLE);
    }

    @Override
    public void run(String... args) throws Exception {

    }
    public static void main(String[] args) {
        SpringApplication.run(App.class, args);
    }

}

So first you need to build it :
mvn clean install

And then you run it with run.bat

For easines I put run_debug.bat which will suspend the VM until you connect with debugger.

[saudet]

@borkox Thanks! I can reproduce the issue with that here. So, it is related to Spring...

[saudet]

I found source of the issue. The issue is with Spring overriding the protocol handler for JAR files with org.springframework.boot.loader.jar.Handler, which doesn't support fragments. When running, for example, System.out.println(new java.net.URL("jar:file:/myjar.jar!/file#foo")); with Spring we get jar:file:/myjar.jar!/file, which is incorrect, but without Spring we get the correct output of jar:file:/myjar.jar!/file#foo.

Is there a version of Spring you guys can use that doesn't have this bug? Or is this a "feature" of Spring and we'll need to work around this?

[borkox]

Hi Samuel. If that's the case, then how would you explain that the same version of this is working on linux ? With Spring again. For me problem is with Windows only.

[saudet]

It fails in exactly the same way for me on Linux.

[herofe]

Googled and found this from spring-boot project, looks like the problem saudet found with the spring Handler stripping fragments?

https://github.com/spring-projects/spring-boot/issues/14011

Looks like its included in spring-boot 1.5.16, which is the latest snapshot. I will check later this week to see if its fixing the issue.

Also in the bug report, it was indicated that the problem only materializes if the application is started with java -jar, and not if you start it with maven using mvn spring-boot:run
Could be a workaround.

[sergmain]

Checked this issue with Spring boot v2.1.0.M2 and it was fixed. Fixed in all snapshot versions too.

[saudet]

I'll close this for now, but let me know if there is more to this issue, thanks!

[lutogniew]

I seem to have the same problem, so I'll post here. I'll begin by saying that I'm indeed using Spring Boot, but I've tried the 2.1.0.M2 version, to no avail. However, it seems to me the problem is elsewhere.

My app still fails with java.lang.UnsatisfiedLinkError: C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\jniopenblas_nolapack.dll: Can't find dependent libraries.

I've naturally analysed the jniopenblas_nolapack.dll with Dependency Walker, and can see that it's missing its dependency LIBOPENBLAS_NOLAPACK.DLL.

I've verified that there is no LIBOPENBLAS_NOLAPACK.DLL in openblas-0.3.0-1.4.2-windows-x86_64.jar file downloaded by Gradle, only jniopenblas_nolapack.dll, jniopenblas.dll and libopenblas.dll.

However, jniopenblas_nolapack.dll clearly references LIBOPENBLAS_NOLAPACK.DLL, which is nowhere to be found in the jar.

No other DLLs from the jar (jniopenblas.dll, libopenblas.dll) have any missing dependencies.

I might be completely missing the point here, but shouldn't there be a libopenblas_nolapack.dll in the openblas-0.3.0-1.4.2-windows-x86_64.jar file?

There is also no LIBOPENBLAS_NOLAPACK.DLL anywhere on my system:

C:\>where /r c:\ LIBOPENBLAS_NOLAPACK.DLL
INFO: Could not find files for the given pattern(s).

I've also tried finding binary packages for OpenBLAS for Windows to see if they contain the missing lib, but e.g. OpenBLAS.v0.3.0.x86_64-w64-mingw32.tar.gz from https://github.com/JuliaLinearAlgebra/OpenBLASBuilder/releases contain no such file.

In fact, even Google knows nothing about libopenblas_nolapack.dll.

I've also noticed an older version in .javacpp/cache, openblas-0.2.19-1.3-windows-x86_64.jar - indeed, dl4j 0.9.1 worked on the same environment without a hitch, so it's probably worked with this older OpenBLAS. I've opened jniopenblas.dll from that version, and it has no missing dependencies. Unfortunately, forcing downgrade of OpenBLAS predictably results in java.lang.NoClassDefFoundError: org/bytedeco/javacpp/openblas_nolapack with dl4j 1.0.0-beta2. And since I'd like to use the new dl4j API, I'm reluctant to downgrade dl4j too.

Any idea where the problem may lie? Any help would be appreciated.

> systeminfo
OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.17134 N/A Build 17134
System Type:               x64-based PC

> java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

> java -Dorg.bytedeco.javacpp.logger=slf4j -Dlogging.level.org.bytedeco=DEBUG -jar ...

2018-09-16 17:54:14.606  INFO 17636 --- [           main] org.nd4j.linalg.factory.Nd4jBackend      : Loaded [CpuBackend] backend
2018-09-16 17:54:15.386 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading class org.nd4j.nativeblas.Nd4jCpu
2018-09-16 17:54:15.387 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading class org.nd4j.nativeblas.Nd4jCpu
2018-09-16 17:54:15.393 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\nd4j-native-1.0.0-beta2-windows-x86_64.jar\org\nd4j\nativeblas\windows-x86_64\libnd4jcpu.dll
2018-09-16 17:54:15.401 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading library nd4jcpu
2018-09-16 17:54:15.403 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load for nd4jcpu: java.lang.UnsatisfiedLinkError: no nd4jcpu in java.library.path
2018-09-16 17:54:15.409 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\nd4j-native-1.0.0-beta2-windows-x86_64.jar\org\nd4j\nativeblas\windows-x86_64\jnind4jcpu.dll
2018-09-16 17:54:15.429 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading library nd4jcpu
2018-09-16 17:54:15.431 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load for nd4jcpu: java.lang.UnsatisfiedLinkError: no nd4jcpu in java.library.path
2018-09-16 17:54:15.434 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Pointer             : Registering org.nd4j.nativeblas.Nd4jCpu$NativeOps[address=0x2abf43d0,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x2abf43d0,deallocatorAddress=0x668ae4e0]]
2018-09-16 17:54:15.437  INFO 17636 --- [           main] org.nd4j.nativeblas.NativeOpsHolder      : Number of threads used for NativeOps: 4
2018-09-16 17:54:15.445 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading class org.bytedeco.javacpp.openblas_nolapack
2018-09-16 17:54:15.446 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading class org.bytedeco.javacpp.openblas_nolapack
2018-09-16 17:54:15.450 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading library iomp5
2018-09-16 17:54:15.452 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load for iomp5#iomp5: java.lang.UnsatisfiedLinkError: no iomp5 in java.library.path
2018-09-16 17:54:15.460 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\libiomp5md.dll
2018-09-16 17:54:15.467 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\msvcr120.dll
2018-09-16 17:54:15.473 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading library mklml_intel
2018-09-16 17:54:15.479 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load for mklml_intel#mklml_intel: java.lang.UnsatisfiedLinkError: no mklml_intel in java.library.path
2018-09-16 17:54:15.484 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\mklml.dll
2018-09-16 17:54:15.490 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\mklml.dll
2018-09-16 17:54:15.495 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Locking C:\Users\Lutogniew\.javacpp\cache to create symbolic link
2018-09-16 17:54:15.497 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Creating symbolic link C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\libopenblas.dll to C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\mklml.dll
2018-09-16 17:54:15.501 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading library openblas_nolapack
2018-09-16 17:54:15.503 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load for [email protected]: java.lang.UnsatisfiedLinkError: no openblas_nolapack in java.library.path
2018-09-16 17:54:15.505 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Locking C:\Users\Lutogniew\.javacpp\cache before extracting
2018-09-16 17:54:15.506 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Extracting jar:file:/E:/Repos/coins/preprocessor/build/libs/preprocessor-1.0-SNAPSHOT.jar!/BOOT-INF/lib/openblas-0.3.0-1.4.2-windows-x86_64.jar!/org/bytedeco/javacpp/windows-x86_64/jniopenblas_nolapack.dll
2018-09-16 17:54:15.528 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Creating symbolic link C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\libiomp5md.dll to C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\libiomp5md.dll
2018-09-16 17:54:15.530 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Creating symbolic link C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\msvcr120.dll to C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\msvcr120.dll
2018-09-16 17:54:15.531 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Creating symbolic link C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\mklml.dll to C:\Users\Lutogniew\.javacpp\cache\mkl-dnn-0.15-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\mklml.dll
2018-09-16 17:54:15.533 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\jniopenblas_nolapack.dll
2018-09-16 17:54:15.536 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\jniopenblas_nolapack.dll: java.lang.UnsatisfiedLinkError: C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\jniopenblas_nolapack.dll: Can't find dependent libraries
2018-09-16 17:54:15.537 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Loading library jniopenblas_nolapack
2018-09-16 17:54:15.539 DEBUG 17636 --- [           main] org.bytedeco.javacpp.Loader              : Failed to load for jniopenblas_nolapack: java.lang.UnsatisfiedLinkError: no jniopenblas_nolapack in java.library.path
2018-09-16 17:54:15.541  INFO 17636 --- [           main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2018-09-16 17:54:15.548 ERROR 17636 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.UnsatisfiedLinkError: no jniopenblas_nolapack in java.library.path
        at java.lang.ClassLoader.loadLibrary(Unknown Source) ~[na:1.8.0_171]
        at java.lang.Runtime.loadLibrary0(Unknown Source) ~[na:1.8.0_171]
        at java.lang.System.loadLibrary(Unknown Source) ~[na:1.8.0_171]
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1225) ~[javacpp-1.4.2.jar!/:1.4.2]
        at org.bytedeco.javacpp.Loader.load(Loader.java:983) ~[javacpp-1.4.2.jar!/:1.4.2]
        at org.bytedeco.javacpp.Loader.load(Loader.java:882) ~[javacpp-1.4.2.jar!/:1.4.2]
        at org.bytedeco.javacpp.openblas_nolapack.<clinit>(openblas_nolapack.java:10) ~[openblas-0.3.0-1.4.2.jar!/:1.4.2]
        at java.lang.Class.forName0(Native Method) ~[na:1.8.0_171]
        at java.lang.Class.forName(Unknown Source) ~[na:1.8.0_171]
        at org.bytedeco.javacpp.Loader.load(Loader.java:941) ~[javacpp-1.4.2.jar!/:1.4.2]
        at org.bytedeco.javacpp.Loader.load(Loader.java:898) ~[javacpp-1.4.2.jar!/:1.4.2]
        at org.bytedeco.javacpp.presets.openblas_nolapack.blas_set_num_threads(openblas_nolapack.java:189) ~[openblas-0.3.0-1.4.2.jar!/:0.3.0-1.4.2]
        at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.setMaxThreads(CpuBlas.java:136) ~[nd4j-native-1.0.0-beta2.jar!/:na]
        at org.nd4j.nativeblas.Nd4jBlas.<init>(Nd4jBlas.java:52) ~[nd4j-native-api-1.0.0-beta2.jar!/:na]
        at org.nd4j.linalg.cpu.nativecpu.blas.CpuBlas.<init>(CpuBlas.java:31) ~[nd4j-native-1.0.0-beta2.jar!/:na]
        at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createBlas(CpuNDArrayFactory.java:91) ~[nd4j-native-1.0.0-beta2.jar!/:na]
        at org.nd4j.linalg.factory.BaseNDArrayFactory.blas(BaseNDArrayFactory.java:72) ~[nd4j-api-1.0.0-beta2.jar!/:1.0.0-beta2]
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.getEnvironmentInformation(NativeOpExecutioner.java:1236) ~[nd4j-native-1.0.0-beta2.jar!/:na]
        at org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.printEnvironmentInformation(DefaultOpExecutioner.java:649) ~[nd4j-api-1.0.0-beta2.jar!/:1.0.0-beta2]
        at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5575) ~[nd4j-api-1.0.0-beta2.jar!/:1.0.0-beta2]
        at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5447) ~[nd4j-api-1.0.0-beta2.jar!/:1.0.0-beta2]
        at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:213) ~[nd4j-api-1.0.0-beta2.jar!/:1.0.0-beta2]
        at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadStaticModel(WordVectorSerializer.java:2488) ~[deeplearning4j-nlp-1.0.0-beta2.jar!/:na]
        at org.test.preprocessor.PreprocessorApplication.run(PreprocessorApplication.java:52) [classes!/:na]
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:823) ~[spring-boot-2.1.0.M2.jar!/:2.1.0.M2]
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:807) ~[spring-boot-2.1.0.M2.jar!/:2.1.0.M2]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:345) ~[spring-boot-2.1.0.M2.jar!/:2.1.0.M2]
        at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:137) [spring-boot-2.1.0.M2.jar!/:2.1.0.M2]
        at org.test.preprocessor.PreprocessorApplication.main(PreprocessorApplication.java:26) [classes!/:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_171]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_171]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [preprocessor-1.0-SNAPSHOT.jar:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [preprocessor-1.0-SNAPSHOT.jar:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [preprocessor-1.0-SNAPSHOT.jar:na]
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [preprocessor-1.0-SNAPSHOT.jar:na]
Caused by: java.lang.UnsatisfiedLinkError: C:\Users\Lutogniew\.javacpp\cache\openblas-0.3.0-1.4.2-windows-x86_64.jar\org\bytedeco\javacpp\windows-x86_64\jniopenblas_nolapack.dll: Can't find dependent libraries
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_171]
        at java.lang.ClassLoader.loadLibrary0(Unknown Source) ~[na:1.8.0_171]
        at java.lang.ClassLoader.loadLibrary(Unknown Source) ~[na:1.8.0_171]
        at java.lang.Runtime.load0(Unknown Source) ~[na:1.8.0_171]
        at java.lang.System.load(Unknown Source) ~[na:1.8.0_171]
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1205) ~[javacpp-1.4.2.jar!/:1.4.2]
        ... 33 common frames omitted

2018-09-16 17:54:15.552  INFO 17636 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

[lutogniew]

So I've decided to build OpenBLAS myself to see if this would fix the issue. I followed the instructions here:

https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio

The only difference being in the configure line - we want a "nolapack" DLL, so I guess the line should be:

cmake .. -G "Ninja" -DCMAKE_CXX_COMPILER=clang-cl -DCMAKE_C_COMPILER=clang-cl -DCMAKE_Fortran_COMPILER=flang -DBUILD_WITHOUT_LAPACK=yes -DNOFORTRAN=0 -DDYNAMIC_ARCH=ON -DCMAKE_BUILD_TYPE=Release

Note all the commands should be executed from the Anaconda prompt. If you get failures with clang-cl.exe being unable to compile the test program, make sure you have no whitespace in the absolute path to OpenBLAS project directory - clang / the build process seem not to support those.

Now, putting the resulting lib as libopenblas_nolapack.dll in .javacpp/cache did nothing - even though Dependency Walker now showed that everything's alright with jniopenblas_nolapack.dll, Loader still complained. Running the app with -Djava.library.path="" didn't help either.

However, putting the DLL as libopenblas_nolapack.dll in one of the directories listed in java.library.path when calling java -XshowSettings:properties -version suddenly made the Loader happy. I succesfully loaded a Word2Vec file that previously failed because of missing dependencies.

I've also naively tried making symlinks pointing at libopenblas.dll, mklml.dll on the off chance that it works - of course it didn't.

So there's a workaround. I've no idea if the DLL is built properly, but it seems to work. Perhaps it'll help someone.

I'd still love to know what's the official way of making it work though.

[saudet]

If the problem doesn't occur outside Spring, then the issue is with Spring.

[sergmain]

@lutogniew could you try spring-boot-2.1.0-M3 ?
Also try to add jvm.config like this -
https://github.com/sergmain/dl4j-uber-jar/blob/master/.mvn/jvm.config

for windows - javacpp.platform=windows-x86_64

[lutogniew]

Okay, found the issue. It was the org.springframework.boot Gradle plugin. Version 2.0.4.RELEASE breaks the app, version 2.0.5.RELEASE makes it work - even with Spring Boot 2.1.0.M2. Boot 2.1.0.M3 works too (but doesn't work with plugin 2.0.4.RELEASE either).

So:

• with Boot Gradle plugin 2.0.4.RELEASE none of the versions 2.0.4.RELEASE, 2.0.5.RELEASE, 2.1.0.M2 nor 2.1.0.M3 of Spring Boot work.
• with Boot Gradle plugin 2.0.5.RELEASE all of the versions 2.0.4.RELEASE (the one that I've used and that failed originally), 2.0.5.RELEASE, 2.1.0.M2 and 2.1.0.M3 work for me.

Checked different configurations a few times.

@sergmain Thank you for pointing me in the right direction!

Thanks guys.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.