Graal: Native-image build of application fails to successfully link on macOS graalvm-ce-java8-19.3.1

[retrogradeorbit]

Describe GraalVM and your environment :

• GraalVM version or commit id if built from source: 19.3.1
• CE or EE: CE
• Build Time or run time failure: build time
• JDK version: JDK8
• Native compiler information:
  Run the following to capture compiler version

  
  

  
  
  • In windows: cl.exe
  
  
  • In macOS : cc -v
  
  
  • In Linux: gcc --version
    
    

Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

• Native linker information:
  Run the following to capture linker version

  
  

  
  
  • In windows: cl.exe
  
  
  • In macOS : cc -Wl,-v
  
  
  • In Linux: gcc -Wl,--version
    
    

@(#)PROGRAM:ld  PROJECT:ld64-302.3
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
Library search paths:
    /usr/lib
    /usr/local/lib
Framework search paths:
    /Library/Frameworks/
    /System/Library/Frameworks/
Undefined symbols for architecture x86_64:
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

• OS and OS Version: Mac OS X 10.12.6 16G29 (Circle CI mac build node)
• Architecture: AMD64
• The output of java -Xinternalversion:

Java HotSpot(TM) 64-Bit Server VM (25.152-b16) for bsd-amd64 JRE (1.8.0_152-b16), built on Sep 14 2017 02:31:13 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)

Have you verified this issue still happens when using the latest snapshot?
Yes. Build fails.
This exact issue doesn't happen with graalvm-ce-java8-darwin-amd64-20.1.0-dev, but another issue does. On dev version the native-image fails before it even tries to link

Describe the issue
A clojure uberjar (that uses JNI to access a shared library) in trying to be built into a native-image cannot be successfully linked on a mac by graal ce versions 19.3.0 and 19.3.1
Builds ok to a binary on linux/amd64 on 19.3.1

Describe the full native-image command

Capture full native-image command by running with the --verbose flag e.g.:

~/graalvm-ce-java8-19.3.1/bin/native-image \
        -jar target/uberjar/spire-$(VERSION)-standalone.jar \
        -H:Name=build/spire \
        -H:+ReportExceptionStackTraces \
        -J-Dclojure.spec.skip-macros=true \
        -J-Dclojure.compiler.direct-linking=true \
        -H:ConfigurationFileDirectories=graal-configs/ \
        --initialize-at-build-time \
        --initialize-at-run-time=com.jcraft.jsch.PortWatcher \
        -H:Log=registerResource: \
        -H:EnableURLProtocols=http,https \
        --verbose \
        --allow-incomplete-classpath \
        --no-fallback \
        --no-server \
        "-J-Xmx6g"

/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/bin/native-image \
        -jar target/uberjar/spire-0.1.0-SNAPSHOT-standalone.jar \
        -H:Name=build/spire \
        -H:+ReportExceptionStackTraces \
        -J-Dclojure.spec.skip-macros=true \
        -J-Dclojure.compiler.direct-linking=true \
        -H:ConfigurationFileDirectories=graal-configs/ \
        --initialize-at-build-time \
        --initialize-at-run-time=com.jcraft.jsch.PortWatcher \
        -H:Log=registerResource: \
        -H:EnableURLProtocols=http,https \
        --verbose \
        --allow-incomplete-classpath \
        --no-fallback \
        --no-server \
        "-J-Xmx6g"
Executing [
/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/bin/java \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-Dtruffle.TrustAllTruffleRuntimeProviders=true \
-Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
-Dgraalvm.ForcePolyglotInvalid=true \
-Dgraalvm.locatorDisabled=true \
-d64 \
-XX:-UseJVMCIClassLoader \
-XX:+UseJVMCINativeLibrary \
-Xss10m \
-Xms1g \
-Xmx6871947672 \
-Duser.country=US \
-Duser.language=en \
-Dorg.graalvm.version=19.3.1 \
-Dorg.graalvm.config=CE \
-Dcom.oracle.graalvm.isaot=true \
-Djvmci.class.path.append=/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/graal.jar \
-javaagent:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/svm.jar \
-Djdk.internal.lambda.disableEagerInitialization=true \
-Djdk.internal.lambda.eagerlyInitialize=false \
-Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \
-Dclojure.spec.skip-macros=true \
-Dclojure.compiler.direct-linking=true \
-Xmx6g \
-Xbootclasspath/a:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/boot/graal-sdk.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/boot/graaljs-scriptengine.jar \
-cp \
/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/graal-llvm.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/javacpp-shadowed.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/llvm-platform-specific-shadowed.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/llvm-wrapper-shadowed.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/objectfile.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/pointsto.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/svm-llvm.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/svm.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/graal-management.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/graal.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/jvmci-api.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/jvmci-hotspot.jar \
com.oracle.svm.hosted.NativeImageGeneratorRunner \
-imagecp \
/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/boot/graal-sdk.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/boot/graaljs-scriptengine.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/graal-llvm.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/javacpp-shadowed.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/llvm-platform-specific-shadowed.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/llvm-wrapper-shadowed.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/objectfile.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/pointsto.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/svm-llvm.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/builder/svm.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/graal-management.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/graal.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/jvmci-api.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/jvmci/jvmci-hotspot.jar:/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/library-support.jar:/Users/distiller/project/target/uberjar/spire-0.1.0-SNAPSHOT-standalone.jar \
-H:Path=/Users/distiller/project \
-H:Class=spire.core \
-H:+ReportExceptionStackTraces \
-H:ConfigurationFileDirectories=graal-configs/ \
-H:ClassInitialization=:build_time \
-H:ClassInitialization=com.jcraft.jsch.PortWatcher:run_time \
-H:Log=registerResource: \

-H:EnableURLProtocols=http,https \
-H:+AllowIncompleteClasspath \
-H:FallbackThreshold=0 \
-H:CLibraryPath=/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64 \
-H:Name=build/spire
]
[build/spire:3095]    classlist:   6,213.45 ms
[build/spire:3095]        (cap):   1,948.64 ms
[build/spire:3095]        setup:   3,632.68 ms
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
[thread:17] scope: ForkJoinPool-4-worker-1
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  ServiceLoaderFeature: registerResource: META-INF/services/java.nio.file.spi.FileSystemProvider
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  ServiceLoaderFeature: registerResource: META-INF/services/com.fasterxml.jackson.core.JsonFactory
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  Resources have been added by ServiceLoaderFeature. Automatic registration can be disabled with -H:-UseServiceLoaderFeature
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  ResourcesFeature: registerResource: libspire.dylib
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  ResourcesFeature: registerResource: libsunec.dylib
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  ServiceLoaderFeature: registerResource: META-INF/services/sun.net.spi.nameservice.NameServiceDescriptor
  [thread:17] scope: ForkJoinPool-4-worker-1.registerResource
  Resources have been added by ServiceLoaderFeature. Automatic registration can be disabled with -H:-UseServiceLoaderFeature
[build/spire:3095]   (typeflow):  42,905.22 ms
[build/spire:3095]    (objects):  38,361.85 ms
[build/spire:3095]   (features):   1,870.45 ms
[build/spire:3095]     analysis:  84,671.96 ms
[build/spire:3095]     (clinit):     691.39 ms
[build/spire:3095]     universe:   1,885.24 ms
[build/spire:3095]      (parse):   5,905.49 ms
[build/spire:3095]     (inline):   8,226.59 ms
[build/spire:3095]    (compile):  34,101.61 ms
[build/spire:3095]      compile:  50,460.88 ms
[build/spire:3095]        image:   4,901.90 ms
[build/spire:3095]        write:     999.76 ms
Fatal error: java.lang.RuntimeException: java.lang.RuntimeException: host C compiler or linker does not seem to work: java.lang.RuntimeException: returned 1

Running command: cc -v -o /Users/distiller/project/build/spire -Wl,-no_compact_unwind -Wl,-exported_symbols_list -Wl,/var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/exported_symbols.list -Wl,-x -arch x86_64 -L/var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476 -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64 /var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/build/spire.o /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libzip.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnet.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnio.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libffi.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/liblibchelper.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libjvm.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libstrictmath.a -lstdc++ -lm -Wl,-framework,CoreFoundation -lpthread -ldl -lz

Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch x86_64 -macosx_version_min 10.12.0 -o /Users/distiller/project/build/spire -L/var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476 -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64 -no_compact_unwind -exported_symbols_list /var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/exported_symbols.list -x /var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/build/spire.o /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libzip.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnet.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnio.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libffi.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/liblibchelper.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libjvm.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libstrictmath.a -lc++ -lm -framework CoreFoundation -lpthread -ldl -lz -lSystem /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture x86_64:
  "_VerifyClassname", referenced from:
      _Java_sun_misc_URLClassPath_knownToNotExist0 in libjava.a(URLClassPath.o)
      _Java_java_lang_ClassLoader_findBootstrapClass in libjava.a(ClassLoader.o)
  "_VerifyFixClassname", referenced from:
      _Java_sun_misc_URLClassPath_knownToNotExist0 in libjava.a(URLClassPath.o)
      _Java_java_lang_ClassLoader_findBootstrapClass in libjava.a(ClassLoader.o)
      _Java_java_lang_ClassLoader_defineClass2 in libjava.a(ClassLoader.o)
      _Java_java_lang_ClassLoader_defineClass1 in libjava.a(ClassLoader.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
    at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
    at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:462)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:315)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:454)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:115)
Caused by: java.lang.RuntimeException: host C compiler or linker does not seem to work: java.lang.RuntimeException: returned 1

Running command: cc -v -o /Users/distiller/project/build/spire -Wl,-no_compact_unwind -Wl,-exported_symbols_list -Wl,/var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/exported_symbols.list -Wl,-x -arch x86_64 -L/var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476 -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64 /var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/build/spire.o /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libzip.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnet.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnio.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libffi.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/liblibchelper.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libjvm.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libstrictmath.a -lstdc++ -lm -Wl,-framework,CoreFoundation -lpthread -ldl -lz

Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch x86_64 -macosx_version_min 10.12.0 -o /Users/distiller/project/build/spire -L/var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476 -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib -L/Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64 -no_compact_unwind -exported_symbols_list /var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/exported_symbols.list -x /var/folders/ms/xg67k5sn16xc7sdr_w3q45840000gn/T/SVM-8490479744702547476/build/spire.o /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libzip.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnet.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libnio.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libsunec.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libffi.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/liblibchelper.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libjvm.a /Users/distiller/graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/svm/clibraries/darwin-amd64/libstrictmath.a -lc++ -lm -framework CoreFoundation -lpthread -ldl -lz -lSystem /Applications/Xcode-9.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture x86_64:
  "_VerifyClassname", referenced from:
      _Java_sun_misc_URLClassPath_knownToNotExist0 in libjava.a(URLClassPath.o)
      _Java_java_lang_ClassLoader_findBootstrapClass in libjava.a(ClassLoader.o)
  "_VerifyFixClassname", referenced from:
      _Java_sun_misc_URLClassPath_knownToNotExist0 in libjava.a(URLClassPath.o)
      _Java_java_lang_ClassLoader_findBootstrapClass in libjava.a(ClassLoader.o)
      _Java_java_lang_ClassLoader_defineClass2 in libjava.a(ClassLoader.o)
      _Java_java_lang_ClassLoader_defineClass1 in libjava.a(ClassLoader.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

    at com.oracle.svm.hosted.image.NativeBootImageViaCC.write(NativeBootImageViaCC.java:388)
    at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:652)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:445)
    at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Image build request failed with exit status 1
com.oracle.svm.driver.NativeImage$NativeImageError: Image build request failed with exit status 1
    at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1482)
    at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1260)
    at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1222)
    at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1181)
make: *** [build/spire] Error 1

Exited with code exit status 2

Code snippet or code repository that reproduces the issue

https://github.com/epiccastle/spire

Steps to reproduce the issue
Please include both build steps as well as run steps

1. Install Leiningen (clojure build tool to make the uberjar https://leiningen.org/ )
2. git clone [email protected]:epiccastle/spire.git
3. cd spire
4. make clean all
   or
5. make clean all GRAALVM=/path/to/graalvm/home/to/use

Expected behavior
A native binary is produced

Additional context
Add any other context about the problem here. Specially important are stack traces or log output. Feel free to link to gists or to screenshots if necesary

• This project uses JNI. I am unsure if that is related somehow.
• Also FYI: the build log for trying to build on 20.1.0-dev graal is here:
  https://circleci.com/gh/epiccastle/spire/506
  Note this produces a different error, but may help narrow it down?

[retrogradeorbit]

spire-0.1.0-SNAPSHOT-standalone.tar.gz

In case you don't want to install leiningen, here's an up to date uberjar (though it's built on linux.) Try native-image on that.

[retrogradeorbit]

looks to be the same issue as this: https://github.com/oracle/graal/issues/2117

[olpaw]

The reported linker output:

  "_VerifyFixClassname", referenced from:
      _Java_sun_misc_URLClassPath_knownToNotExist0 in libjava.a(URLClassPath.o)
      _Java_java_lang_ClassLoader_findBootstrapClass in libjava.a(ClassLoader.o)
      _Java_java_lang_ClassLoader_defineClass2 in libjava.a(ClassLoader.o)
      _Java_java_lang_ClassLoader_defineClass1 in libjava.a(ClassLoader.o)

indicates that you likely use dynamic class loading somewhere in the application you are trying to build. For native-images we do not support classloading at image runtime. See https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md#dynamic-class-loading--unloading

You can build with -H:+PrintAnalysisCallTree to help you with finding out which part of your application makes use of dynamic class loading (i.e. drags in dependencies to VerifyClassname / VerifyFixClassname)

[retrogradeorbit]

@olpaw This code is built by native-image correctly on linux ce java8 19.3.1. Does the linux native-image 19.3.1 support dynamic class loading while the mac does not? In addition, the native-image in 19.2.1 on mac works on this code.

That could be confusing. let me summarize. For this jar:

19.2.1 linux: works
19.2.1 mac: works
19.3.1 linux: works
19.3.1 mac: fails with this link error.

[retrogradeorbit]

With -H:+PrintAnalysisCallTree no VerifyClassname or VerifyFixClassname are found anywhere:

...
[build/spire:8635]     analysis:  95,971.98 ms
Printing call tree to /home/crispin/dev/epiccastle/spire/reports/call_tree_spire_20200211_222243.txt
Printing list of used methods to /home/crispin/dev/epiccastle/spire/reports/used_methods_spire_20200211_222338.txt
Printing list of used classes to /home/crispin/dev/epiccastle/spire/reports/used_classes_spire_20200211_222338.txt
Printing list of used packages to /home/crispin/dev/epiccastle/spire/reports/used_packages_spire_20200211_222338.txt
[build/spire:8635]     (clinit):     890.65 ms
...
[build/spire:8635]      [total]: 478,431.58 ms
cp build/spire spire
$ rgrep VerifyClassname reports/
$ rgrep VerifyFixClassname reports/

[olpaw]

This code is built by native-image correctly on linux ce java8 19.3.1. Does the linux native-image 19.3.1 support dynamic class loading while the mac does not?

We generally do not support dynamic class loading (a limitation of our approach to AOT compilation).

With -H:+PrintAnalysisCallTree no VerifyClassname or VerifyFixClassname are found anywhere:

Are you using -H:+PrintAnalysisCallTree on the platform where you are getting the linker errors (OSX)?

...
[build/spire:8635]      [total]: 478,431.58 ms

indicates that you are not.

[retrogradeorbit]

OSX is still running. Will have it soon...

[retrogradeorbit]

We generally do not support dynamic class loading (a limitation of our approach to AOT compilation).

So could OSX be doing dynamic class loading where linux would not?

[olpaw]

So could OSX be doing dynamic class loading where linux would not?

I wouldn't be surprised (likely not intentionally but as a side effect of some implementation detail that differs on OSX).

[retrogradeorbit]

OSX Analysis call tree is in. No luck

~/reports# ls -alF
total 884032
drwxr-xr-x 2 root root      4096 Feb 11 15:10 ./
drwx------ 7 root root      4096 Feb 11 15:10 ../
-rw-r--r-- 1 root root 902242202 Feb 11 15:10 call_tree_spire_20200211_070430.txt
-rw-r--r-- 1 root root    347129 Feb 11 15:10 used_classes_spire_20200211_070507.txt
-rw-r--r-- 1 root root   2634931 Feb 11 15:10 used_methods_spire_20200211_070507.txt
-rw-r--r-- 1 root root      5462 Feb 11 15:10 used_packages_spire_20200211_070508.txt
~/reports# rgrep VerifyClassname .
~/reports# rgrep VerifyFixClassname .

Build log and reports available here: https://circleci.com/gh/epiccastle/spire/518#artifacts/containers/0

[dougxc]

Try adding -J-XX:TraceClassLoadingCause=Verify to your native-image command line. This is a JVMCI JDK 8 only flag that will print a Java stack trace when loading a class whose fully qualified name contains the given string.

[retrogradeorbit]

Just FYI found the symbol in these libraries.

~/macos-graal# rgrep VerifyClassname .
Binary file ./graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libverify.dylib matches
Binary file ./graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.dylib matches
Binary file ./graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.a matches
~/macos-graal# rgrep VerifyFixClassname .
Binary file ./graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libverify.dylib matches
Binary file ./graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.dylib matches
Binary file ./graalvm-ce-java8-19.3.1/Contents/Home/jre/lib/libjava.a matches

Does it need a -lverify in the link command? I can't find a libverify.a, though.

(just running TraceClassLoadingCause flag now)

[retrogradeorbit]

@dougxc TraceClassLoadingCause output here: https://circleci.com/gh/epiccastle/spire/521

cant see any VerifyClassname or VerifyFixClassname in the output

[olpaw]

I do find

> grep -i defineClass '/home/pwoegere/Downloads/call_tree_spire_20200211_070430.txt' 
│   │       │           └── directly calls sun.reflect.ClassDefiner.defineClass(java.lang.String, byte[], int, int, java.lang.ClassLoader):java.lang.Class id=8293 @bci=24 
│   │       │               ├── directly calls sun.misc.Unsafe.defineClass(java.lang.String, byte[], int, int, java.lang.ClassLoader, java.security.ProtectionDomain):java.lang.Class id=14296 @bci=30 

though.

[olpaw]

though.

... but that one ends in a com.oracle.svm.core.util.VMError.unsupportedFeature. So it should be fine (not causing linker errors).

I will dig further.

[olpaw]

@retrogradeorbit thanks for the bug report. I will look into this. Can you check if this is reproducible on OSX with Java 11?

[retrogradeorbit]

Will try with Java 11 and report back

[retrogradeorbit]

@olpaw https://github.com/oracle/graal/issues/2144#issuecomment-584510969

[retrogradeorbit]

Java11

graal-ce-java11-9.3.1 on macos builds and links ok. https://circleci.com/gh/epiccastle/spire/527

Running it however gives me a com.oracle.svm.core.jdk.UnsupportedFeatureError at runtime when clojure.lang.Reflector tries to perform reflection. More work is probably needed to port it.

Java11 native image run time error is:

$ ./spire -e '0'
Exception in thread "main" com.oracle.svm.core.jdk.UnsupportedFeatureError: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
    at com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:101)
    at clojure.lang.Reflector.canAccess(Reflector.java:49)
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:156)
    at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
    at spire.config$path_join.invokeStatic(config.clj:22)
    at spire.config$path_join.doInvoke(config.clj:18)
    at clojure.lang.RestFn.invoke(RestFn.java:421)
    at spire.config$init_BANG_.invokeStatic(config.clj:46)
    at spire.config$init_BANG_.invoke(config.clj:41)
    at spire.core$initialise.invokeStatic(core.clj:32)
    at spire.core$initialise.invoke(core.clj:24)
    at spire.core$_main.invokeStatic(core.clj:60)
    at spire.core$_main.doInvoke(core.clj:58)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at spire.core.main(Unknown Source)

[olpaw]

Java11 native image run time error

Thanks for checking. Just to be clear the images you mentioned here:

19.2.1 linux: works
19.2.1 mac: works
19.3.1 linux: works

do work, right? (i.e. no only build correctly but also behave as expected when running them.

[olpaw]

@retrogradeorbit I disassembled all the static libraries the we ship as part of 19.3.1 on Darwin and I was not able to find any transitive symbol dependency that could lead to referencing _VerifyClassname or _VerifyFixClassname.
(I followed all callees transitively and none of the call chains ends up in a JNI Method that I see as leaf method somewhere in call_tree_spire_20200211_070430.txt

Building spire-0.1.0-SNAPSHOT-standalone.jar (taken from https://github.com/oracle/graal/issues/2144#issuecomment-584510969) on OSX locally works as expected:

> bin/native-image -jar spire-0.1.0-SNAPSHOT-standalone.jar -H:Name=spire -H:+ReportExceptionStackTraces -J-Dclojure.spec.skip-macros=true -J-Dclojure.compiler.direct-linking=true -H:ConfigurationFileDirectories=graal-configs/ --initialize-at-build-time --initialize-at-run-time=com.jcraft.jsch.PortWatcher -H:Log=registerResource: -H:EnableURLProtocols=http,https --allow-incomplete-classpath --no-fallback --no-server "-J-Xmx6g"
[spire:1938]    classlist:   6,192.55 ms
[spire:1938]        (cap):   1,118.41 ms
[spire:1938]        setup:   2,141.44 ms
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
[thread:26] scope: ForkJoinPool-4-worker-1
  [thread:26] scope: ForkJoinPool-4-worker-1.registerResource
  ServiceLoaderFeature: registerResource: META-INF/services/java.nio.file.spi.FileSystemProvider
  [thread:26] scope: ForkJoinPool-4-worker-1.registerResource
  ServiceLoaderFeature: registerResource: META-INF/services/com.fasterxml.jackson.core.JsonFactory
  [thread:26] scope: ForkJoinPool-4-worker-1.registerResource
  Resources have been added by ServiceLoaderFeature. Automatic registration can be disabled with -H:-UseServiceLoaderFeature
  [thread:26] scope: ForkJoinPool-4-worker-1.registerResource
  ServiceLoaderFeature: registerResource: META-INF/services/sun.net.spi.nameservice.NameServiceDescriptor
  [thread:26] scope: ForkJoinPool-4-worker-1.registerResource
  Resources have been added by ServiceLoaderFeature. Automatic registration can be disabled with -H:-UseServiceLoaderFeature
[spire:1938]   (typeflow):  16,429.49 ms
[spire:1938]    (objects):  15,114.52 ms
[spire:1938]   (features):   1,099.81 ms
[spire:1938]     analysis:  33,640.27 ms
[spire:1938]     (clinit):     358.87 ms
[spire:1938]     universe:   1,015.59 ms
[spire:1938]      (parse):   1,639.40 ms
[spire:1938]     (inline):   3,590.66 ms
[spire:1938]    (compile):  11,608.13 ms
[spire:1938]      compile:  17,912.87 ms
[spire:1938]        image:   2,025.07 ms
[spire:1938]        write:     599.07 ms
[spire:1938]      [total]:  63,765.01 ms

I suspect the native-toolchain installed on your OSX machine is to blame. For my successful build I have:

> cc -v
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
> cc -Wl,-v
@(#)PROGRAM:ld  PROJECT:ld64-450.3
BUILD 18:16:53 Apr  5 2019
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
    /usr/lib
    /usr/local/lib
Framework search paths:
    /Library/Frameworks/
    /System/Library/Frameworks/

[retrogradeorbit]

@olpaw Thanks for verifying that. I will try and update the build machine. And if still have problems invoice Circle CI support. Will report back what I discover.

[retrogradeorbit]

@olpaw Tried with LLVM 10.0.0, same result. That other uberjar was built on linux. Can you do me another favour before I report this and try your native-image on your mac on this uberjar:

https://538-221739364-gh.circle-artifacts.com/0/target/uberjar/spire-0.1.0-SNAPSHOT-standalone.jar

This is verifiably an uberjar that is failing on the circle ci build nodes. https://circleci.com/gh/epiccastle/spire/538

If this builds on yours it's something to do with the build environment. If this fails on yours it might be something to do with the java environment that is building the jar in the first place.

[olpaw]

Mystery solved. The reason why I could not reproduce is that i was missing a populated graal-configs directory contributing to the image build via -H:ConfigurationFileDirectories=graal-configs/. Once I go through all the steps explained in https://github.com/oracle/graal/issues/2144#issue-563019464 I can reproduce the issue.

[retrogradeorbit]

That narrows it down. It's reflection related!

[olpaw]

It's reflection related!

Yup. The moment sun.reflect.ClassDefiner can be found in the results of call_tree_spire_*.txt (gatherd by -H:+PrintAnalysisCallTree) you will get the linker error.

[olpaw]

    {
        "name": "java.lang.Class",
        "allDeclaredConstructors": true,
        "allPublicConstructors": true,
        "allDeclaredMethods": true,
        "allPublicMethods": true
    },

Do you really need all of that from java.lang.Class? That looks like trouble.

[olpaw]

For the sake of image size alone you should be more selective what you allow for reflective access. I recommend using our java agent to discover what you actually need reflectivity accessible. See https://medium.com/graalvm/introducing-the-tracing-agent-simplifying-graalvm-native-image-configuration-c3b56c486271

[olpaw]

For the sake of image size alone you should be more selective what you allow for reflective access

But I do not want to distract from the fact that dying with a linker error here is something that needs to be fixed on our side.

[retrogradeorbit]

I will check whether I do need all of java.lang.Class. Thanks for the point out. Maybe not. But I cant use tracing alone to find the requirements and I do need a fair bit because it embeds a clojure interpreter, and other people's code they write to run inside it needs access to all manner of things.

[olpaw]

The culprit is the following Java to native call from the analysis results:

└── virtually calls sun.misc.URLClassPath.initLookupCache(java.lang.ClassLoader):void @bci=25
    └── is overridden by sun.misc.URLClassPath.initLookupCache(java.lang.ClassLoader):void id=20802
        ├── directly calls sun.misc.URLClassPath.disableAllLookupCaches():void id=23539 @bci=23
        └── directly calls sun.misc.URLClassPath.getLookupCacheURLs(java.lang.ClassLoader):java.net.URL[] id=23540 @bci=5
            ├── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.boxObjectInLocalHandle(java.lang.Object):com.oracle.svm.jni.nativeapi.JNIObjectHandle id-ref=803 @bci=7
            ├── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.boxObjectInLocalHandle(java.lang.Object):com.oracle.svm.jni.nativeapi.JNIObjectHandle id-ref=803 @bci=5
            ├── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.environment():com.oracle.svm.jni.nativeapi.JNIEnvironment id-ref=1311 @bci=3
            ├── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallEpilogue(int):void id-ref=1312 @bci=12
            ├── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.nativeCallPrologue():int id-ref=1313 @bci=1
            ├── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.rethrowPendingException():void id-ref=1314 @bci=14
            └── directly calls com.oracle.svm.jni.JNIGeneratedMethodSupport.unboxHandle(com.oracle.svm.jni.nativeapi.JNIObjectHandle):java.lang.Object id-ref=806 @bci=10

On Linux this is harmless because libjava.a got built with -ffunction-sections. On OSX that does not work properly an the seemingly harmless dependency to the native code implementation of URLClassPath.getLookupCacheURLs causes the linker to drag-in all native code implementations from libjava.a(URLClassPath.o) (athough _Java_sun_misc_URLClassPath_getLookupCacheURLs alone would be sufficient). Among them there is _Java_sun_misc_URLClassPath_knownToNotExist0 which is the one that depends on _VerifyClassname and _VerifyFixClassname.

On Linux while we have the same dependency to URLClassPath.getLookupCacheURLs this is not a problem because there the linker drags-in only _Java_sun_misc_URLClassPath_getLookupCacheURLs.

You don't see this on master because fa8df5ec6e41 introduced @Delete annotations for those URLClassPath methods. This causes the analysis to cut off the link to native code implementations thus not dragging in anything from URLClassPath native code implementations.

Although already fixed on master, while figuring out what exactly is going on here I found some other places where we still miss defensive @Delete annotations. I will add them and also provide fallback implementations for VerifyClassname and VerifyFixClassname as a second line of defense against such tricky issues.

[olpaw]

Although already fixed on master, while figuring out what exactly is going on here I found some other places where we still miss defensive @Delete annotations. I will add them and also provide fallback implementations for VerifyClassname and VerifyFixClassname as a second line of defense against such tricky issues.

eab7e93b585febf7c2c68948af70157732767963
9e38150d052a467ae7e522b56497d53fb95064be

[retrogradeorbit]

This is a very clear explanation @olpaw thankyou very much for finding this. Good news that it is fixed on master. I will test again when those patches hit the dev builds.