Graal: Undefined reference to JVM_GetSystemPackage, JVM_FillInStackTrace and JVM_LatestUserDefinedLoader

[sdeleuze]

Our spring-graal-native build is broken when compiled against GraalVM master (and green with 20.1 branch) with following errors:

/opt/java/jre/lib/libjava.a(Package.o): In function `Java_java_lang_Package_getSystemPackage0':
/opt/jprt/T/P1/164053.buildslave/s/jdk/src/share/native/java/lang/Package.c:34: undefined reference to `JVM_GetSystemPackage'
/opt/java/jre/lib/libjava.a(Throwable.o): In function `Java_java_lang_Throwable_fillInStackTrace':
/opt/jprt/T/P1/164053.buildslave/s/jdk/src/share/native/java/lang/Throwable.c:49: undefined reference to `JVM_FillInStackTrace'
/opt/java/jre/lib/libjava.a(VM.o): In function `Java_sun_misc_VM_latestUserDefinedLoader0':
/opt/jprt/T/P1/164053.buildslave/s/jdk/src/share/native/sun/misc/VM.c:116: undefined reference to `JVM_LatestUserDefinedLoader'

collect2: error: ld returned 1 exit status
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
    at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
    at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:463)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:346)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:505)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:114)
Caused by: java.lang.RuntimeException: There was an error linking the native image: Linker command exited with 1

Full error message is available at https://ci.spring.io/teams/spring-graal-native/pipelines/spring-graal-native/jobs/master-java8-samples-build/builds/10 (click on "build" tab to see the logs).

To reproduce, follow those instructions, use docker/run-master-java8.sh to run the Docker container and run build.sh then build-samples.sh.

Could be linked to https://github.com/oracle/graal/issues/2448.

Please tag this issue with spring label.

[olpaw]

Looks like SubstrateJvmFuncsFallbacksBuilder is not working correctly.

[eginez]

I am not able to reproduce this locally (as with #2448 ). macOS seems to build properly

[sdeleuze]

Could you try with the Docker environment instructions provided above?

[olpaw]

Looks like SubstrateJvmFuncsFallbacksBuilder is not working correctly.

Verified that SubstrateJvmFuncsFallbacksBuilder does build graal/substratevm/src/com.oracle.svm.native.jvm.posix/src/src_gen/JvmFuncsFallbacks.c properly. Must be something else.

[olpaw]

Can reproduce with Docker container. Investigating...

[olpaw]

The graalvm in /opt/java/ of the spring-graal-native-dev:master-java8 docker container is broken.
nm /opt/java/jre/lib/svm/clibraries/linux-amd64/libjvm.a shows that the archive only contains JvmFuncs.o and is missing JvmFuncsFallbacks.o. Looking into the latest graalvm-ce-dev-builds https://github.com/graalvm/graalvm-ce-dev-builds/releases/download/20.2.0-dev-20200513_0200/graalvm-ce-java8-linux-amd64-20.2.0-dev.tar.gz shows that our snapshot builds are correct (graalvm-ce-java8-20.2.0-dev/jre/lib/svm/clibraries/linux-amd64/libjvm.a contains JvmFuncs.o and JvmFuncsFallbacks.o).

[olpaw]

@sdeleuze where can I download the Dockerfile that builds spring-graal-native-dev:master-java8?

[sdeleuze]

Docker images are available from https://hub.docker.com/r/springci/graalvm-ce/tags (just with GraalVM) or https://hub.docker.com/r/springci/spring-graal-native/tags (GraalVM with some dependencies required for our build).

We are using https://github.com/graalvm/graal-jvmci-8/releases/download/jvmci-20.1-b02/openjdk-8u252+09-jvmci-20.1-b02-linux-amd64.tar.gz and https://github.com/graalvm/labs-openjdk-11/releases/download/jvmci-20.1-b02/labsjdk-ce-11.0.7+10-jvmci-20.1-b02-linux-amd64.tar.gz since those seems the latest available. Is it possible that your are using other ones for your dev builds?

[sdeleuze]

Dockerfiles are availble from https://github.com/spring-projects-experimental/spring-graal-native/blob/master/docker/Dockerfile.graalvm-ce and https://github.com/spring-projects-experimental/spring-graal-native/blob/master/docker/Dockerfile.spring-graal-native with those parameters https://github.com/spring-projects-experimental/spring-graal-native/blob/master/ci/pipeline.yml#L139-L171.

Notice we are using the same for 20.1 (which is green) and master (broken) so I suspect JVMCI being the cause of the breakage.

[olpaw]

Notice we are using the same for 20.1 (which is green) and master (broken) so I suspect JVMCI being the cause of the breakage

The fact that JVMCI_URL for GRAALVM_BRANCH: release/graal-vm/20.1 and GRAALVM_BRANCH: master is the same looks fishy. But it is the latest release. So it must be correct.

[sdeleuze]

Indeed, but I did not found JVMCI releases for 20.2 so I took the latest 20.1 ones. I will be happy to update asap 20.2 builds are available on https://github.com/graalvm/graal-jvmci-8/releases/ and https://github.com/graalvm/labs-openjdk-11/releases/.

[olpaw]

Is it possible that your are using other ones for your dev builds?

I have checked. We don't have any newer ones internally. So they must be the right ones. I will build a docker image equivalent to spring-graal-native-dev:master-java8 locally with the Dockerfile you pointed me to. This should allow me to see what causes libjvm.a to get built incorrectly.

[olpaw]

After some debugging I'm convinced that this is a bug in mx.

In the docker case we always start with an empty ~/.mx/cache. This causes downloading of the dependencies NINJA, NINJA_SYNTAX. See https://github.com/graalvm/mx/blob/25d0517d3cd4d8493cbc867c7b019431ca53a4c0/mx.mx/suite.py#L365..L418
As a consequence when the task for com.oracle.svm.native.jvm.posix_amd64 gets executed we see: Building com.oracle.svm.native.jvm.posix_amd64 with Ninja... [dependencies updated: NINJA, NINJA_SYNTAX]. This corresponds to the following code in mx: https://github.com/graalvm/mx/blob/25d0517d3cd4d8493cbc867c7b019431ca53a4c0/mx.py#L4697. In this code path we get buildNeeded = True. This unfortunately prevents the execution of https://github.com/graalvm/mx/blob/25d0517d3cd4d8493cbc867c7b019431ca53a4c0/mx.py#L4704..L4708 for the other dependencies of com.oracle.svm.native.jvm.posix_amd64. Specifically the dependency to svm-jvmfuncs-fallback-builder which is an mx.ArchivableProject.

In other words why should the fact that some dependency of a project got updated prevent remaind dependencies to to get a chance to contribute to the current project (by not fetching their newestOutput())

cc @gilles-duboscq

[olpaw]

Talked to @gilles-duboscq - turns out my assumptions about newestOutput() are false. The whole concept of mx.BuildTask is not based on data-flow based on the outputs of BuildTasks. It's strictly the dependencies of the structure defined in suite.py that dictate execution. Since mx.ArchivableBuildTask.needsBuild() returns (False, 'Files are already on disk') there is no obligation to ever run mx.ArchivableBuildTask.newestOutput(). I will provide a custom implementation of needsBuild/build. This will ensure svm-jvmfuncs-fallback-builder works as expected.