Deeplearning4j: Error: lazy symbol binding failed: Symbol not found: ___emutls_get_address

[arshtepe]

Issue Description

Please describe our issue, along with:
I have error when run my project in 1.0.0-beta5, it is working in 1.0.0-beta4

dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
  Referenced from: ~/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

Version Information

Please indicate relevant versions, including, if relevant:

• Deeplearning4j: 1.0.0-beta5
• Platform information (OS, etc): macOs Mojave 10.14.4
• CUDA version, if used
• NVIDIA driver version, if in use

Additional Information

Where applicable, please also provide:

• Full log or exception stack trace (ideally in a Gist: gist.github.com):

dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
  Referenced from: ~/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

• https://gist.github.com/arshtepe/e244dc9ebfdbe21ca7fd12465b9977d5

[saudet]

@sshepel Which versions of GCC and Xcode are we building with these days?

[sshepel]

@saudet for release and generic builds we have:

-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0

[raver119]

@arshtepe can you share the code you're running?

I've been running examples before release, they were running fine. Just ran examples right now - still running fine. Tried lenet example, and some basic lstm/dense examples. No issues as well.

[raver119]

[arshtepe]

@saudet
Xcode 10.3
Build version 10G8

cmake version 3.15.2
gcc -dumpversion -> 4.2.1
@raver119 sorry I can't share the code :( but I can provide any details which you want

[saudet]

@sshepel And the version of Xcode?

[sshepel]

@saudet Xcode version: 8.3.3

[saudet]

GCC 8.x was released beginning of 2018: https://www.gnu.org/software/gcc/releases.html
So let's try to use a version of Xcode that was released around the same time frame. That would be close to Xcode 9.2, which is the last to run on Mac OS X 10.12.6: https://en.wikipedia.org/wiki/Xcode

[sshepel]

@saudet sure, will do that ASAP.

[sshepel]

Ok, Xcode has been updated to version 9.2.0. Snapshots are on the way!
@arshtepe could you please try to use snapshots tomorrow to check if the issue has been resolved for you...

[arshtepe]

@sshepel okey, I will let you know, thanks :)

[zhaihao]

I have the same problem.

code:

import org.nd4s.Implicits._
val arr = (1 to 9).toNDArray(3,3)
println(arr)

output:

19:58:02.671 [info ] o.n.linalg.factory.Nd4jBackend(196)      - Loaded [CpuBackend] backend
19:58:03.135 [info ] o.n.nativeblas.NativeOpsHolder(104)      - Number of threads used for OpenMP: 4
dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
19:58:03.602 [info ] org.nd4j.nativeblas.Nd4jBlas(55)         - Number of threads used for OpenMP BLAS: 4
  Referenced from: /Users/zhaihao/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ___emutls_get_address
  Referenced from: /Users/zhaihao/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

system env:

Mac OSX 10.14.6

$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: 10.3.0.0.1.1562985497
volume: /
location: /
install-time: 1563950915
groups: com.apple.FindSystemFiles.pkg-group

$ cmake --version
cmake version 3.14.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ gcc -v
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

I hope this would help.

[raver119]

at this moment I'm not sure why it's reproducible for you, but we suppose that's an issue related to thread_local storage specifier, so I've disabled it for macos now. once snapshots rebuilt please try again

[arshtepe]

@raver119 I can't update to snapshot, it is loading infinitely
my pom

    <repositories>
        <repository>
            <id>snapshots-repo</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
            </snapshots>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>

this way doesn't work too

    <repositories>
        <repository>
            <id>snapshots-repo</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
            </snapshots>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native-platform</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-modelimport</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui_2.12</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-api</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>

[zhaihao]

@arshtepe see https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-core

[raver119]

@raver119 I can't update to snapshot, it is loading infinitely

It looks like sonatype has some problems atm, we've noticed similar problems here as well

[arshtepe]

now I have new error ,
btw code https://gitlab.com/Eliza_H/cnn_network/blob/master/src/main/java/NetworkModel.java

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.<init>(NativeOpExecutioner.java:91)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5246)
    at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5153)
    at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:182)
    at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:579)
    at NetworkCNNModel.execute(NetworkCNNModel.java:115)
    at NetworkCNNModel.main(NetworkCNNModel.java:276)
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: http://nd4j.org/getstarted.html
    at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:106)
    at org.nd4j.nativeblas.NativeOpsHolder.<clinit>(NativeOpsHolder.java:34)
    ... 12 more
Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcpu in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1487)
    at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
    at org.bytedeco.javacpp.Loader.load(Loader.java:1040)
    at org.nd4j.nativeblas.Nd4jCpu.<clinit>(Nd4jCpu.java:10)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:84)
    ... 13 more
Caused by: java.lang.UnsatisfiedLinkError: ..../.javacpp/cache/nd4j-native-1.0.0-20190911.011548-14906-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libjnind4jcpu.dylib: dlopen(/Users/artemshtepenko/.javacpp/cache/nd4j-native-1.0.0-20190911.011548-14906-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libjnind4jcpu.dylib, 1): Library not loaded: @rpath/libmkldnn.0.dylib
  Referenced from: ..../.javacpp/cache/nd4j-native-1.0.0-20190911.011548-14906-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/./libnd4jcpu.dylib
  Reason: image not found
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1461)
    ... 19 more

[raver119]

@saudet any ideas why it still tries to load libmkldnn?
1) for x86_64 binary libnd4j doesn't use mkldnn atm.
2) all libmkldnn references from java were removed.

[raver119]

p.s. @AlexDBlack tried to run latest snapshot on windows and it runs fine for him.

[saudet]

The artifacts are probably just not up-to-date..

[streambuf]

i have same problem when try run org.deeplearning4j.examples.convolution.AnimalsClassification
commit 391ec1bcfa69b13804b78103a645f425c1dd5ac0

[saudet]

This problem is probably caused by the upgrade to GCC 8.3.
These patches needed to support rpath haven't been upgraded on the CI server:
https://github.com/bytedeco/javacpp-presets/blob/master/ci/install-travis.sh#L218
The workaround for now is to install GCC 8.3 with brew install gcc@8.

[streambuf]

gcc -v
gcc version 9.2.0 (Homebrew GCC 9.2.0)
not helped

[streambuf]

This problem is probably caused by the upgrade to GCC 8.3.
These patches needed support rpath haven't been upgraded on the CI server:
https://github.com/bytedeco/javacpp-presets/blob/master/ci/install-travis.sh#L218
The workaround for now is to install GCC 8.3 with brew install gcc@8.

It's help me. Thanks!

[timothyjward]

In case it is helpful I have created a simple test repo which shows this issue.

After running brew install gcc@8 I see both beta4 and beta5 working, but the latest snapshots still fail with the UnsatisfiedLinkError

[saudet]

Yeah, the snapshots are currently broken, unfortunately. It may take a couple of weeks to get things running well again.

[timothyjward]

Yeah, the snapshots are currently broken, unfortunately. It may take a couple of weeks to get things running well again.

That's understandable, multi-platform native code can be a real pain!

[rafmontano]

Hi All

I am new on this, so please forget me for anything that should I know or it should be given for granted.

Look, I have been using a model for quite a while and since i upgraded to Catalina, I have been issues with all the models including the test demo. I am using now 1.0.0-beta5.

I upgraded from Java 8 to Java 12, and avoid core dump problems... however now I am having the following error Message just compiling, any of the demo cases and my model.

Please any work around / suggestion is really appreciated.

__"/Library/Java/JavaVirtualMachines/jdk-12.0.2.jdk
Picked up _JAVA_OPTIONS: -Xmn40G -Xms50G -Xmx50G -XX:+UseG1GC
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for OpenMP: 24
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 24
dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
Referenced from: /Users/Rafa/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ___emutls_get_address
Referenced from: /Users/Rafa/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
Expected in: /usr/lib/libSystem.B.dylib

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
"__

[eraly]

@rafmontano Try after running brew install gcc@8. Thank you for your patience! Let us know if you are still running into trouble.

[rafmontano]

Hi Susan,

Thanks for come back to me. Running Brew Install gcc8 didn't work for me in MacOS Catalina and Java 8. Gave me this error:

/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=63118:/Applications/IntelliJ IDEA CE.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath
Picked up _JAVA_OPTIONS: -Xmn8G -Xms8G -Xmx8G -XX:+UseG1GC
#

A fatal error has been detected by the Java Runtime Environment:

#

SIGSEGV (0xb) at pc=0x00007fff674d20b4, pid=38904, tid=0x0000000000000f03

#

JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode bsd-amd64 compressed oops)

Problematic frame:

C [libc++abi.dylib+0x30b4] __cxxabiv1::__si_class_type_info::has_unambiguous_public_base(__cxxabiv1::__dynamic_cast_info, void, int) const+0x4

#

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

An error report file with more information is saved as:

/Users/Rafa/Dropbox/DeepLearning4j/NewProjects/1.0.0-beta4/dl4j-examples-predict/hs_err_pid38904.log

#

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

#

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

---

I have 2 machines, I managed to roll back 1 to MacOS Mojave (it was not that easy), and now i have the model operational in that one.

Please I have a questions, Which Version of Java do you recommend? I upgraded in the rush of the moment to java 12 (from 8), to see if that solve the problem and solve it that one, however that caused the lazy symbol issue on this post, which in MacOS Catalina I have not managed to pass.

[sabarishs]

@rafmontano Try after running brew install gcc@8. Thank you for your patience! Let us know if you are still running into trouble.

This helped solve my problem! Thanks

[rafmontano]

Thanks Sabarishs,I have managed to run my model. Moving to Java 11 and running brew install gcc@8, solved the problem.

[raver119]

AFAIK, this issue is resolved now, and updated binaries are available via snapshots. If, for some reason, issue is still reproducible for you - feel free to reopen this issue, or file a new one.

[hinsbergen]

Hi, I'm running into the same problems. The command "brew install gcc@8" indeed solved it for my Mac, but not for Windows where I run another instance.

I have downloaded the latest version (beta5) from Maven Central; it is dated September 6. Where can I get new jars where this error has been fixed? Or do I have to wait for beta6? Thanks!

[AlexDBlack]

@hinsbergen The fix is only available via snapshots at present: https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

It will be in the next release of course (1.0.0-beta6), which should be out within 1-2 weeks.