Graal: [native-image] NullPointerException during security services registration

[gwenneg]

Hi GraalVM team!

I'm working on the GraalVM 19.3.0 integration into Quarkus and a NullPointerException is being thrown during three of the Quarkus extensions integration tests.

Here's an example of NPE (this one is from a local build but we have the same failures in the Quarkus CI):

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] /Library/Java/JavaVirtualMachines/graalvm-unknown-java8-19.3.0/Contents/Home/jre/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar quarkus-integration-test-elytron-security-oauth2-999-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+TraceSecurityServices -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services -H:+JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace quarkus-integration-test-elytron-security-oauth2-999-SNAPSHOT-runner
[quarkus-integration-test-elytron-security-oauth2-999-SNAPSHOT-runner:47933]    classlist:  19,354.31 ms
[quarkus-integration-test-elytron-security-oauth2-999-SNAPSHOT-runner:47933]        (cap):   6,483.14 ms
[quarkus-integration-test-elytron-security-oauth2-999-SNAPSHOT-runner:47933]        setup:  10,970.08 ms
13:43:26,876 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Final
[quarkus-integration-test-elytron-security-oauth2-999-SNAPSHOT-runner:47933]     analysis:  59,683.57 ms
Fatal error: java.lang.NullPointerException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
    at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
    at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:462)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:315)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:454)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:115)
Caused by: java.lang.NullPointerException
    at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57)
    at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
    at java.lang.reflect.Field.get(Field.java:393)
    at com.oracle.svm.hosted.SecurityServicesFeature.lambda$getConsParamClassAccessor$0(SecurityServicesFeature.java:279)
    at com.oracle.svm.hosted.SecurityServicesFeature.register(SecurityServicesFeature.java:313)
    at com.oracle.svm.hosted.SecurityServicesFeature.registerServicesForReflection(SecurityServicesFeature.java:191)
    at com.oracle.svm.hosted.ReachabilityHandlerFeature.duringAnalysis(ReachabilityHandlerFeature.java:114)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$8(NativeImageGenerator.java:710)
    at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:63)
    at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:710)
    at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:530)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:445)
    at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Image build request failed with exit status 1

Since this exception didn't give any details about the service that was causing it, I added traces to the GraalVM source code and recompiled it. This helped me determine the services names that are causing the exceptions: org.wildfly.security.password.impl.PasswordFactorySpiImpl and org.bouncycastle.jce.provider.X509StoreCertCollection. There might be other services involved.

I'm still not sure of what's causing this, but there's at least one thing that justifies creating this issue: it would be great if you could display the service name that is causing the NPE throw. Unless this is not supposed to happen at all.

@cstancu: Does this look like a possible GraalVM bug or is there something wrong on our side? If it looks like a bug, I'll try to provide a reproducer.

Here's the Quarkus issue about the GraalVM 19.3.0 integration (this one is limited to JDK8 only): https://github.com/quarkusio/quarkus/pull/5358

[cstancu]

@gwenneg please provide a reproducible. The trace doesn't seem to align properly with the source for the vm-19.3.0 tag. We rely on reading meta-data from the java.security.Provider.knownEngines map for known JDK security services and I think the problem is that we try to look for the EngineDescription metadata for those two third-party engines. Do you call any com.oracle.svm.hosted.SecurityServicesFeature directly, via reflection from Quarkus? Either way, we should not throw a NPE, so I consider this a bug.

[gwenneg]

Thanks for your answer.

Here's the same trace from the Quarkus CI. This one should align with the vm-19.3.0 tag.

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v /home/vsts/work/1/s/integration-tests/jgit/target/quarkus-integration-test-jgit-999-SNAPSHOT-native-image-source-jar:/project:z --user 1001:117 --rm quay.io/quarkus/ubi-quarkus-native-image:19.3.0-java8 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar quarkus-integration-test-jgit-999-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx6g -H:-AddAllCharsets -H:EnableURLProtocols=http,https --enable-all-security-services -H:+JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace quarkus-integration-test-jgit-999-SNAPSHOT-runner
[quarkus-integration-test-jgit-999-SNAPSHOT-runner:19]    classlist:  12,116.62 ms
[quarkus-integration-test-jgit-999-SNAPSHOT-runner:19]        (cap):   1,122.86 ms
[quarkus-integration-test-jgit-999-SNAPSHOT-runner:19]        setup:   3,096.90 ms
08:06:20,064 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Final
[quarkus-integration-test-jgit-999-SNAPSHOT-runner:19]     analysis:  50,465.82 ms
Fatal error: java.lang.NullPointerException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
    at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
    at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:462)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:315)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:454)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:115)
Caused by: java.lang.NullPointerException
    at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57)
    at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
    at java.lang.reflect.Field.get(Field.java:393)
    at com.oracle.svm.hosted.SecurityServicesFeature.lambda$getConsParamClassAccessor$0(SecurityServicesFeature.java:277)
    at com.oracle.svm.hosted.SecurityServicesFeature.register(SecurityServicesFeature.java:311)
    at com.oracle.svm.hosted.SecurityServicesFeature.registerServicesForReflection(SecurityServicesFeature.java:191)
    at com.oracle.svm.hosted.ReachabilityHandlerFeature.duringAnalysis(ReachabilityHandlerFeature.java:114)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$8(NativeImageGenerator.java:710)
    at com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:63)
    at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:710)
    at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:530)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:445)
    at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Image build request failed with exit status 1

[gwenneg]

I'll try to provide a reproducible and check for com.oracle.svm.hosted.SecurityServicesFeature calls ASAP.

[jaikiran]

Hello @gwenneg, I've a reproducible test case within the substratevm project which shows this issue and potential fix. I can submit it in 15-20 minutes for review.

[gwenneg]

Great! Thanks for your help @jaikiran.

[jaikiran]

Here it is https://github.com/oracle/graal/pull/1893

[gwenneg]

As explained in the PR, fix confirmed in Quarkus. Thanks again!

[jaikiran]

@gwenneg, thank you for testing and verifying against Quarkus.

[gwenneg]

@cstancu: I'd like to know if it is possible to introduce this bug fix into 19.3.0 using code substitution on the SecurityServicesFeature class. That would allow us to integrate GraalVM 19.3 into Quarkus without having to wait for 19.3.1.

Here's the code I've been trying to use with 19.3.0 without success so far:

@TargetClass(value = SecurityServicesFeature.class, onlyWith = GraalVersion19_3_0.class)
final class Target_com_oracle_svm_hosted_SecurityServicesFeature {

    @Substitute
    private static Class<?> lambda$getConsParamClassAccessor$0(Map<String, Object> knownEngines, Field consParamClassNameField,
            BeforeAnalysisAccess access, java.lang.String serviceType) {
        try {
            /*
             * Access the Provider.knownEngines map and extract the EngineDescription
             * corresponding to the serviceType. Note that the map holds EngineDescription(s) of
             * only those service types that are shipped in the JDK. From the EngineDescription
             * object extract the value of the constructorParameterClassName field then, if the
             * class name is not null, get the corresponding Class<?> object and return it.
             */
            /* EngineDescription */Object engineDescription = knownEngines.get(serviceType);
            /*
             * This isn't an engine known to the Provider (which actually means that it isn't
             * one that's shipped in the JDK), so we don't have the predetermined knowledge of
             * the constructor param class.
             */
            if (engineDescription == null) {
                return null;
            }
            String constrParamClassName = (String) consParamClassNameField.get(engineDescription);
            if (constrParamClassName != null) {
                return access.findClassByName(constrParamClassName);
            }
        } catch (IllegalAccessException e) {
            VMError.shouldNotReachHere(e);
        }
        return null;

    }
}

final class GraalVersion19_3_0 implements BooleanSupplier {
    public boolean getAsBoolean() {
        final String version = System.getProperty("org.graalvm.version");
        return version.startsWith("19.3.0");
    }
}

[cstancu]

Substitutions provide a mechanism to patch run time code (at build time). You cannot use it to patch build time code, e.g., SecurityServicesFeature.

[gwenneg]

That's good to now, thanks for your reply @cstancu.