Quarkus: Big file fails native image generation with java.lang.NegativeArraySizeException

Created on 21 Jun 2019 · 3Comments · Source: quarkusio/quarkus

Big file in META-INF/resources/ fails native image generation with java.lang.NegativeArraySizeException

Project code: https://github.com/quarkus-qe/beefy-scenarios/tree/master/scenario-201-static
Big file generated in process-test-classes phase using https://github.com/quarkus-qe/beefy-scenarios/blob/master/scenario-201-static/src/test/java/io/quarkus/qe/hello/GenerateLargeFile.java
Run it using mvn -pl scenario-201-static clean verify -Dnative command.

Size of the file is 1GB - f.setLength(1024 * 1024 * 1024).
Native build works well if the size is changed to f.setLength(1024 * 1024 * 1023).
Native image command doesn't fail when java.lang.NegativeArraySizeException is thrown.

It's definitely SubstrateVM issue, I would like to ask @bobmcwhirter and @dmlloyd for their view on this. JVM mode handles 1GB file well.

[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] Running Quarkus native-image plugin on OpenJDK GraalVM CE 1.0.0-rc16
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] /Users/rsvoboda/.sdkman/candidates/java/1.0.0-rc-16-grl/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar scenario-201-static-1.0.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+PrintAnalysisCallTree -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-SpawnIsolates -H:-JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]    classlist:   2,902.88 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]        (cap):   1,406.96 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]        setup:   2,716.58 ms
java.lang.RuntimeException: Failed to load resource META-INF/resources/big-file
    at io.quarkus.runtime.ResourceHelper.registerResources(ResourceHelper.java:26)
    at io.quarkus.runner.AutoFeature.beforeAnalysis(Unknown Source)
    at com.oracle.svm.hosted.FeatureHandler$WrappedDeprecatedFeature.beforeAnalysis(FeatureHandler.java:170)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$7(NativeImageGenerator.java:668)
    at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:65)
    at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:668)
    at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:524)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:442)
    at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at io.quarkus.runtime.ResourceHelper.registerResources(ResourceHelper.java:22)
    ... 12 more
Caused by: java.lang.NegativeArraySizeException
    at com.oracle.svm.core.jdk.Resources.registerResource(Resources.java:81)
    ... 17 more
12:00:36,380 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Beta4
12:00:36,897 INFO  [org.xnio] XNIO version 3.7.2.Final
12:00:36,944 INFO  [org.xni.nio] XNIO NIO Implementation Version 3.7.2.Final
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of io.quarkus.runtime.CleanableExecutor$Resetter. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of io.quarkus.runtime.CleanableExecutor$Resetter. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of org.jboss.threads.ThreadLocalResettingRunnable$Resetter. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
Warning: RecomputeFieldValue.FieldOffset automatic substitution failed. The automatic substitution registration was attempted because a call to sun.misc.Unsafe.objectFieldOffset(Field) was detected in the static initializer of org.jboss.threads.ThreadLocalResettingRunnable$Resetter. Detailed failure reason(s): The argument of Unsafe.objectFieldOffset(Field) is not a constant field., Could not determine the field where the value produced by the call to sun.misc.Unsafe.objectFieldOffset(Field) for the field offset computation is stored. The call is not directly followed by a field store or by a sign extend node followed directly by a field store.
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]   (typeflow):  11,008.75 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]    (objects):  10,303.66 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]   (features):     192.71 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]     analysis:  21,948.65 ms
Printing call tree to /Users/rsvoboda/git/beefy-scenarios/scenario-201-static/target/reports/call_tree_scenario-201-static-1.0.0-SNAPSHOT-runner_20190621_120058.txt
Printing list of used classes to /Users/rsvoboda/git/beefy-scenarios/scenario-201-static/target/reports/used_classes_scenario-201-static-1.0.0-SNAPSHOT-runner_20190621_120059.txt
Printing list of used packages to /Users/rsvoboda/git/beefy-scenarios/scenario-201-static/target/reports/used_packages_scenario-201-static-1.0.0-SNAPSHOT-runner_20190621_120059.txt
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]     universe:     532.79 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]      (parse):   2,077.70 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]     (inline):   6,797.90 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]    (compile):  19,625.98 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]      compile:  30,077.06 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]        image:   3,931.59 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]        write:   1,802.31 ms
[scenario-201-static-1.0.0-SNAPSHOT-runner:79606]      [total]:  69,502.54 ms
[INFO]
[INFO] --- maven-failsafe-plugin:2.22.0:integration-test (default) @ scenario-201-static ---

kinbug triagwontfix

Source

rsvoboda

All 3 comments

This is not likely to be realistically fixable; SubstrateVM seems to store resources as byte arrays on the serialized heap, and byte arrays are bounded in size. We probably should have a friendlier error message for this case, but that's about all we can do. I think having a 2GB resource in a native image is pretty solidly an edge case though.