Graal: Micronaut app fails with 20.3.0 Release but works with same version compiled from source code

[ilopmar]

Describe the issue

We've been continuously testing the release/graal-vm/20.3 branch for the last couple of weeks to make sure everything would work with Micronaut. All our test applications passed the build this morning at 7am using latest Micronaut snapshot and that mentioned release branch.

This morning I've switched to use the 20.3.0 version released yesterday and our Elasticsearch test application has started to fail to build the native image. It fails for both JDK 8 and 11.

I've been doing some tests locally and I've discovered that we were using and old version to build GraalVM: build 25.252-b09-jvmci-20.2-b01 vs build 25.272-b10-jvmci-20.3-b06. The latter is the one from 20.3.0 release.

I though that was the problem so I've compiled again GraalVM branch release/graal-vm/20.3 using that new java version, but I'm able to build the native image. Switching the jvm to 20.3.0 Release make it fail again.

So, I have a few questions:

• Can you please confirm that the 20.3.0 release was created from this commit c5ff5a5a091bfd974640aaf6fbe51c81bd080438.

• This is what we do on CI. I've disabled pretty much everything because I'm only interested in native-image and I want the build to be as fast as possible.

mx --disable-polyglot --disable-libpolyglot --dynamicimports /substratevm --skip-libraries=true build

For local sometimes I use the following, which creates the agent and the diagnostics-agent:

NATIVE_IMAGES=lib:native-image-diagnostics-agent,lib:native-image-agent LIBGRAAL=true mx --disable-polyglot --disable-libpolyglot --dynamicimports /substratevm build

Can you please share which options do you use to compile the release? I would like to use the same to have repeateble builds. Please keep in mind that I would like to disable all the things I don't need.

One final thing. I'm downloading the java releases to build GraalVM from https://github.com/graalvm/graal-jvmci-8/releases/ for Java 8 and https://github.com/graalvm/labs-openjdk-11/releases/ for Java 11. Are those the right versions I need to download?

Steps to reproduce the issue

• git clone https://github.com/micronaut-graal-tests/micronaut-elasticsearch-graal
• cd micronaut-elasticsearch-graal
• git checkout 2.1.x
• Use GraalVM 20.3.0 release for Java 8
• ./build-native-image.sh
• The build fails. See the error below.

Now Instead of using GraalVM 20.3.0 release, compile GraalVM from source code using release/graal-vm/20.3 branch and this java version.

• The build works and the native-image is created.

Describe GraalVM and your environment:

• GraalVM version: 20.3.0 release
• JDK major version: 8 and 11
• OS: Linux
• Architecture: AMD64

More details

Error using 20.3.0 Release.

[elasticsearch:54453]    classlist:  16,058.37 ms,  1.83 GB
[elasticsearch:54453]        (cap):     977.72 ms,  1.83 GB
[elasticsearch:54453]        setup:   3,389.09 ms,  1.83 GB
[elasticsearch:54453]     analysis:  83,635.69 ms,  4.26 GB
Fatal error:com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing org.graalvm.home.HomeFinder.getInstance() 
Parsing context:
    parsing org.graalvm.polyglot.Engine.getVersion(Engine.java:203)
    parsing com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.getEngineVersion(GraalJSEngineFactory.java:132)
    parsing com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.getParameter(GraalJSEngineFactory.java:168)
    parsing org.apache.logging.log4j.core.script.ScriptManager.<init>(ScriptManager.java:84)
    parsing org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:219)
    parsing org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    parsing org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
    parsing org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
    parsing org.apache.logging.log4j.core.LoggerContext.setConfigLocation(LoggerContext.java:676)
    parsing org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:219)
    parsing org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:131)
    parsing org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:228)
    parsing org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
    parsing org.apache.logging.log4j.LogManager.getContext(LogManager.java:174)
    parsing org.apache.logging.log4j.LogManager.getLogger(LogManager.java:669)
    parsing org.elasticsearch.rest.BytesRestResponse.<clinit>(BytesRestResponse.java:120)

    at com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:138)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:331)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:302)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:103)
    at com.oracle.graal.pointsto.flow.StaticInvokeTypeFlow.update(InvokeTypeFlow.java:434)
    at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:547)
    at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
    at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: com.oracle.svm.core.util.UserError$UserException: ImageSingletons do not contain key org.graalvm.home.HomeFinder
    at com.oracle.svm.core.util.UserError.abort(UserError.java:68)
    at com.oracle.svm.hosted.ImageSingletonsSupportImpl$HostedManagement.doLookup(ImageSingletonsSupportImpl.java:119)
    at com.oracle.svm.hosted.ImageSingletonsSupportImpl.lookup(ImageSingletonsSupportImpl.java:44)
    at org.graalvm.nativeimage.ImageSingletons.lookup(ImageSingletons.java:86)
    at com.oracle.svm.hosted.snippets.SubstrateGraphBuilderPlugins$43.apply(SubstrateGraphBuilderPlugins.java:1011)
    at org.graalvm.compiler.nodes.graphbuilderconf.InvocationPlugin.execute(InvocationPlugin.java:189)
    at org.graalvm.compiler.java.BytecodeParser.applyInvocationPlugin(BytecodeParser.java:2185)
    at org.graalvm.compiler.java.BytecodeParser.tryInvocationPlugin(BytecodeParser.java:2171)
    at com.oracle.svm.hosted.phases.AnalysisGraphBuilderPhase$AnalysisBytecodeParser.tryInvocationPlugin(AnalysisGraphBuilderPhase.java:67)
    at org.graalvm.compiler.java.BytecodeParser.appendInvoke(BytecodeParser.java:1876)
    at org.graalvm.compiler.java.BytecodeParser.genInvokeStatic(BytecodeParser.java:1637)
    at org.graalvm.compiler.java.BytecodeParser.genInvokeStatic(BytecodeParser.java:1617)
    at org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5360)
    at org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3399)
    at org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3206)
    at org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1092)
    at org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:986)
    at org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:84)
    at com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:70)
    at org.graalvm.compiler.phases.Phase.run(Phase.java:49)
    at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:212)
    at org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
    at org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
    at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:223)
    at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:357)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:313)
    ... 10 more
Error: Image build request failed with exit status 1

[eregon]

Regarding building GraalVM:

• LIBGRAAL has no effect, it's just a component like another nowadays
• Use --components=.../COMPONENTS=... to control exactly what gets built, see https://github.com/oracle/graal/blob/c78128b20a9139207aa7374440cc66003c242386/vm/README.md#example-build-only-the-truffleruby-launcher
• NATIVE_IMAGES/FORCE_BASH_LAUNCHERS can be used to control what gets built as native images or Bash launchers, but most likely in your case you want native images always, except maybe for native-image itself.
• Use graalvm-show to see which components you build and native/bash.

You can use mx fetch-jdk to automatically get the correct JVMCI JDK to build GraalVM.

[eregon]

From the stacktrace:

Fatal error:com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing org.graalvm.home.HomeFinder.getInstance() 
Parsing context:
    parsing org.graalvm.polyglot.Engine.getVersion(Engine.java:203)
    parsing com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.getEngineVersion(GraalJSEngineFactory.java:132)
    parsing com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.getParameter(GraalJSEngineFactory.java:168)
    parsing org.apache.logging.log4j.core.script.ScriptManager.<init>(ScriptManager.java:84)
    parsing org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:219)
    parsing org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    parsing org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
    parsing org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)

So that ends up initializing Graal.js through the org.apache.logging.log4j.core.script.ScriptManager here. That's probably unexpected that the native image generation process would end up loading JavaScript that way.

[graemerocher]

@ilopmar why does this app have log4j on the classpath and not logback?

[ilopmar]

@eregon Thanks for the information regarding to build GraalVM. I'll take a closer look at do some tests.

Can you please explain why the output of the build is different?

@graemerocher I added this to make Elasticsearch logging work on the JVM and this to make it work as native-image.

[graemerocher]

@ilopmar but the question is why? Why not use logback?

[ilopmar]

@graemerocher Because it didn't work when I created the test app sometime ago. Maybe I can try this approach I did for Liquibase https://micronaut-projects.github.io/micronaut-liquibase/latest/guide/index.html#_logging

In any case, none of this explains why the Graal.js is triggered and why the builds are not the same, which I think is the really important part because I would like to create the builds as close as possible as the GraalVM team does.

[ilopmar]

@eregon I think the issue migth be because I'm not building graal-js locally so maybe that javascript thing is not triggered when using my build. I've tried the following to build it locally (according to the docs you linked) but it fails:

mx --dynamicimports /graal-js build
Traceback (most recent call last):
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 17180, in <module>
    main()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 17065, in main
    primary = _discover_suites(primarySuiteMxDir, load=should_load_suites)
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3506, in _discover_suites
    _register_visit(primary)
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3502, in _register_visit
    _register_visit(discovered[_suite_import.name])
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3504, in _register_visit
    s._load()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 1604, in _load
    self._load_extensions()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 2026, in _load_extensions
    mod = __import__(extensionsName)
  File "/home/ivan/workspaces/misc/graal/graaljs/graal-js/mx.graal-js/mx_graal_js.py", line 360, in <module>

Adding /graal-js with or without any other dynamic imports fails.

Regarding the rest of the options for building I think I'm already doing what I want with:

mx --disable-polyglot --disable-libpolyglot --dynamicimports /substratevm

or

NATIVE_IMAGES=lib:native-image-diagnostics-agent,lib:native-image-agent mx --disable-polyglot --disable-libpolyglot --dynamicimports /substratevm build

for building the extra agents.

To be honest there is a lot of libs, modules and names and I'm a little bit lost with this. I'd appreaciate any help on this to confirm that those build commands are the right ones.

NATIVE_IMAGES/FORCE_BASH_LAUNCHERS can be used to control what gets built as native images or Bash launchers, but most likely in your case you want native images always, except maybe for native-image itself.

What do you mean with this? Why I don't want a native image for native-image?

Can you please also explain what are "bash launchers"?

[ilopmar]

@graemerocher I've tried adding implementation("org.slf4j:log4j-over-slf4j:1.7.30") but unless I add log4j in implementation scope, the native image fails at runtime when sending a request with:

Caused by: org.apache.commons.logging.LogConfigurationException: No suitable Log implementation
    at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:848)
    at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
    at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
    at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
    at org.elasticsearch.client.RestClient.<clinit>(RestClient.java:99)
    at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:351)
    at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:271)
    ... 101 common frames omitted

[gilles-duboscq]

Can you please confirm that the 20.3.0 release was created from this commit c5ff5a5.

Yes

Can you please share which options do you use to compile the release?

The build job is actually this one:
https://github.com/oracle/graal/blob/c5ff5a5a091bfd974640aaf6fbe51c81bd080438/vm/ci_common/common.hocon#L325-L338

After all the hocon expansion this is the result: https://gist.github.com/gilles-duboscq/7828de2147bc3e9a436b08980ea700b0
I think mostly it's:

mx --strip-jars --sources=sdk:GRAAL_SDK,truffle:TRUFFLE_API,compiler:GRAAL,substratevm:SVM --with-debuginfo --env ce  --force-bash-launchers=env.FORCE_BASH_LAUNCHERS,RMain build

One final thing. I'm downloading the java releases to build GraalVM from https://github.com/graalvm/graal-jvmci-8/releases/ for Java 8 and https://github.com/graalvm/labs-openjdk-11/releases/ for Java 11. Are those the right versions I need to download?

Yes, you can use the command mentioned by Benoit to automatically download the right one. common.json at the root of the repo have the reference to the exact versions used for building.

Regarding your issue with mx --dynamicimports /graal-js build, the stack trace you cited seems to be truncated, it's missing the actual error (python stack traces are printed in the opposite direction to the way it's done for Java).

"bash launchers" are the launcher script that get generated instead of a native-image if you prevent some launchers from being built as native-images using NATIVE_IMAGES or FORCE_BASH_LAUNCHERS.

[eregon]

My guess is org.apache.logging.log4j.core.script.ScriptManager iterates all ScriptEngine's. Since GraalVM releases always include Graal.js, this happens, but the commands above do not include Graal.js so it does not.

Now Engine.getVersion() should probably work there (it needs the HomeFinderFeature, but that should automatically be picked up if graal-sdk.jar is on the classpath, maybe an issue given that's on the bootclasspath?). OTOH, if any of that code is reachable at runtime it wouldn't work because the JavaScript language is not included in the native image (would need --language:js but seems unlikely to be what you want here).

It seems a bit surprising that log4j reaches into ScriptEngine's like this, maybe there is some configuration to disable that behavior?

[ilopmar]

Thank you @gilles-duboscq @eregon and @loicottet for the detailed explanations. I really appreciate it!

I think I'm missing some config or environment variable defined because the instructions you have provided don't work for me:

export PATH=/home/ivan/workspaces/misc/graal/mx:$PATH
export JAVA_HOME=/home/ivan/workspaces/misc/graal/openjdk1.8.0_272-jvmci-20.3-b06

ivan@nobita:~/workspaces/misc/graal/graal/vm$ mx --strip-jars --sources=sdk:GRAAL_SDK,truffle:TRUFFLE_API,compiler:GRAAL,substratevm:SVM --with-debuginfo --env ce  --force-bash-launchers=env.FORCE_BASH_LAUNCHERS,RMain build
Traceback (most recent call last):
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 17180, in <module>
    main()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 17065, in main
    primary = _discover_suites(primarySuiteMxDir, load=should_load_suites)
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3506, in _discover_suites
    _register_visit(primary)
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3502, in _register_visit
    _register_visit(discovered[_suite_import.name])
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3502, in _register_visit
    _register_visit(discovered[_suite_import.name])
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3504, in _register_visit
    s._load()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 1604, in _load
    self._load_extensions()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 2026, in _load_extensions
    mod = __import__(extensionsName)
  File "/home/ivan/workspaces/misc/graal/graaljs/graal-js/mx.graal-js/mx_graal_js.py", line 360, in <module>
    build_args=['--language:js']
TypeError: __init__() takes at least 6 arguments (5 given)

Then I've realized that maybe it's because of python, so I've defined this:

export DEFAULT_VM=server
export MX_PYTHON=python3

But now it fails with other error message:

ivan@nobita:~/workspaces/misc/graal/graal/vm$ mx --strip-jars --sources=sdk:GRAAL_SDK,truffle:TRUFFLE_API,compiler:GRAAL,substratevm:SVM --with-debuginfo --env ce  --force-bash-launchers=env.FORCE_BASH_LAUNCHERS,RMain build
Traceback (most recent call last):
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 17180, in <module>
    main()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 17065, in main
    primary = _discover_suites(primarySuiteMxDir, load=should_load_suites)
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3506, in _discover_suites
    _register_visit(primary)
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3502, in _register_visit
    _register_visit(discovered[_suite_import.name])
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3502, in _register_visit
    _register_visit(discovered[_suite_import.name])
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 3504, in _register_visit
    s._load()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 1604, in _load
    self._load_extensions()
  File "/home/ivan/workspaces/misc/graal/mx/mx.py", line 2026, in _load_extensions
    mod = __import__(extensionsName)
  File "/home/ivan/workspaces/misc/graal/graaljs/graal-js/mx.graal-js/mx_graal_js.py", line 356, in <module>
    mx_sdk.LanguageLauncherConfig(
TypeError: __init__() missing 1 required positional argument: 'language'

Any hints on how to fix this?

@loicottet the workaround of adding HomeFinderFeature works perfect, thanks!

[ilopmar]

I'm sorry, I've realized that graal-js is in another repository. I've cloned that repo, followed the instructions, and I was able to build Graalvm locally with graal-js and reproduce the error.

Again, thank you very much!

[gilles-duboscq]

Given

  File "/home/ivan/workspaces/misc/graal/graaljs/graal-js/mx.graal-js/mx_graal_js.py", line 360, in <module>
    build_args=['--language:js']
TypeError: __init__() takes at least 6 arguments (5 given)

It looks like you already had JS cloned at /home/ivan/workspaces/misc/graal/graaljs but in a version that was incompatible.
Maybe check your /home/ivan/workspaces/misc/graal directory to check that you have only one copy of the JS repo in there.

[ilopmar]

Yeah, I already fixed it. It looks like I had a really old clone of graaljs. I deleted it and cloned again and everything worked!

[ilopmar]

I've found another issue. With the workaround provided I can build the native image, but then some javascript code is still there when running the app.

 in 20ms. Server Running: http://localhost:8080
Failed to clear engine names [Oracle Nashorn]
java.lang.NoSuchFieldException: names
    at java.lang.Class.getDeclaredField(DynamicHub.java:2070)
    at com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.clearEngineFactory(GraalJSEngineFactory.java:235)
    at com.oracle.truffle.js.scriptengine.GraalJSEngineFactory.<clinit>(GraalJSEngineFactory.java:85)
    at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:351)
    at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:271)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(DynamicHub.java:920)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at javax.script.ScriptEngineManager.initEngines(ScriptEngineManager.java:122)
    at javax.script.ScriptEngineManager.init(ScriptEngineManager.java:84)
    at javax.script.ScriptEngineManager.<init>(ScriptEngineManager.java:61)
    at org.apache.logging.log4j.core.script.ScriptManager.<init>(ScriptManager.java:70)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:219)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
    at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:602)
    at org.elasticsearch.search.builder.SearchSourceBuilder.<clinit>(SearchSourceBuilder.java:84)
    at com.oracle.svm.core.classinitialization.ClassInitializationInfo.invokeClassInitializer(ClassInitializationInfo.java:351)
    at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:271)
    at org.elasticsearch.action.search.SearchRequest.<init>(SearchRequest.java:122)
    at micronaut.example.service.MovieServiceImpl.searchMovies(MovieServiceImpl.java:63)
    at micronaut.example.controller.MovieController.searchMovies(MovieController.java:42)
    at micronaut.example.controller.$MovieControllerDefinition$$exec2.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:474)
...

To reproduce this new problem:

• git clone https://github.com/micronaut-graal-tests/micronaut-elasticsearch-graal
• cd micronaut-elasticsearch-graal
• git checkout 2.2.x
• Use GraalVM 20.3.0 release for Java 8
• ./build-native-image.sh
• The native image is built because I've added the workaround.
• Start elasticsearch using Docker (see README for more info): docker run -it --rm -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.9.3
• ./elasticsearch
• curl "localhost:8080/api/movies?title=matrix"

[loicottet]

The whole situation is caused by the fact that services declared by modules are automatically included in the reflection configuration since 20.3 (see https://github.com/oracle/graal/commit/678354c0c31adad5cae9d348e085153c4134f603). In particular, GraalJSEngineFactory is registered as a ScriptEngineFactory by org.graalvm.js.scriptengine, which is present in the base Graal distribution. This new problem makes me think that this service should not be registered in the first place when JS is not needed, I'll ask and confirm it.

[loicottet]

-H:ServiceLoaderFeatureExcludeServiceProviders=com.oracle.truffle.js.scriptengine.GraalJSEngineFactory should be a sturdier workaround until we fix this.

[sdeleuze]

@loicottet -H:ServiceLoaderFeatureExcludeServiceProviders=com.oracle.truffle.js.scriptengine.GraalJSEngineFactory does not seems to fix the issue based on my use case (using Log4j2 in Spring Boot) and the workaround used on Micronaut example side (--features=org.graalvm.home.HomeFinderFeature) is not usable for us since it will disable other required parts. How could we move forward on fixing this?

[loicottet]

@d-kozak can you help with this? This looks like further service loader shenanigans.