Graal: Automatically register classes in META-INF/services for reflection.

[cstancu]

Currently all service providers declared in META-INF/services resources need to be manually registered for reflection. To improve usability this should be automated.

Registration for reflection is needed because the service providers are loaded with Class.forName() and are instantiated with Class.newInstance(). A typical reflection configuration for service providers looks like this:

[
  {
    "name" : "service.ServiceImplementation0",
    "methods": [
      { "name": "<init>", "parameterTypes": [] }
    ]
]

,i.e., it specifies the class and the no-parameter constructor. A complete example of registering service providers can be found in this demo project.

The automatic registration can be achieved by parsing META-INF/services at native image build time and using the SubstrateVM API to register the classes.

Automatic registration should be limited to the META-INF/services resources explicitely registered using the -H:IncludeResources=<resource-path-regexp> option to avoid polution of the image with all resources on the classpath.

SubstrateVM provides the Feature mechanism to extend its internals with plugable components. For reflection registration the RuntimeReflection API can be invoked from a Feature. The reflection documentation provides details about the reflection API. An example of registering elements for reflection from a Feature can be found in GsonFeature.

(Make sure to read the contributing guidelines if you would like to take on this issue.)

[dmlloyd]

The classes should probably only be registered if the service type is actually used, to avoid extra classes being pulled in.

[dmlloyd]

This is somewhat complicated because it does not appear that the Feature API allows dependencies to be added to classes. Might it be better to rewrite calls to ServiceLoader.load(Class) to instantiate customized iterator classes so that dependency analysis can work better?

[cstancu]

I agree that registering only the service providers that are a subtype of the Class reachable from ServiceLoader.load(Class) would be even better. The Class could be detected with an invocation plugin. The invocation plugins can be used to intercept calls during bytecode parsing and as long as the Class is a constant we can unequivocally determine it. (See for example SubstrateGraphBuilderPlugins for some invocation plugin examples). One option would be to expose the invocation plugins API to the Feature API. Or we could just implement this feature in the SubstrateGraphBuilderPlugins, see for example registerAtomicUpdaterPlugins which registers some elements for reflection, instead of exposing too many internals to the Feature API.

[vhiairrassary]

@cstancu I am trying to create a feature for that and here is my attempt (that's a rapid POC and absolutely not the final code as I do not know too much the stack yet).

It allows to compile & execute your example, output is:

> mx native-image -H:IncludeResources=META-INF/services/service.ServiceBase -jar target/ServiceLoaderTest-1.0-SNAPSHOT.jar
   classlist:   1,284.54 ms
       (cap):   1,524.77 ms
FOUND: service.ServiceImplementation0
FOUND: service.ServiceImplementation1
       setup:   2,440.79 ms
  (typeflow):   4,752.80 ms
   (objects):   2,121.46 ms
  (features):      49.31 ms
    analysis:   7,025.13 ms
    universe:     270.66 ms
     (parse):   1,007.09 ms
    (inline):     858.98 ms
   (compile):   5,099.69 ms
     compile:   7,432.31 ms
       image:   1,377.32 ms
       write:     750.23 ms
     [total]:  20,639.55 ms

> ./ServiceLoaderTest-1.0-SNAPSHOT
services.iterator().hasNext() = true
service implementation = class service.ServiceImplementation0
service implementation = class service.ServiceImplementation1

I would love to have your opinion on that. Every feedback is welcome. I can improve it to register only the service type if it is actually used.
BTW using -H:IncludeResources=META-INF/services/*.* gave me some headaches as com.oracle.svm.truffle.api.SubstrateTruffleRuntimeAccess service throws java.lang.NoClassDefFoundError.

diff --git a/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/Resources.java b/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/Resources.java
index e54994cc113..6d53d91eb0b 100644
--- a/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/Resources.java
+++ b/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/Resources.java
@@ -52,8 +52,9 @@ import com.oracle.svm.core.util.VMError;
  */
 public final class Resources {

-    static class ResourcesSupport {
-        final Map<String, List<byte[]>> resources = new HashMap<>();
+    // TODO: find a better way to do that
+    public static class ResourcesSupport {
+        public final Map<String, List<byte[]>> resources = new HashMap<>();
     }

     private Resources() {
diff --git a/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ServiceFeature.java b/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ServiceFeature.java
new file mode 100644
index 00000000000..f902c1991c3
--- /dev/null
+++ b/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ServiceFeature.java
@@ -0,0 +1,67 @@
+package com.oracle.svm.hosted;
+
+import com.oracle.svm.core.annotate.AutomaticFeature;
+import com.oracle.svm.core.jdk.Resources;
+import com.oracle.svm.core.jdk.ResourcesFeature;
+import org.graalvm.nativeimage.Feature;
+import org.graalvm.nativeimage.ImageSingletons;
+import org.graalvm.nativeimage.RuntimeReflection;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+@AutomaticFeature
+public final class ServiceFeature implements Feature {
+    @Override
+    public List<Class<? extends Feature>> getRequiredFeatures() {
+        return Collections.singletonList(ResourcesFeature.class);
+    }
+
+    @Override
+    public void beforeAnalysis(BeforeAnalysisAccess access) {
+        FeatureImpl.BeforeAnalysisAccessImpl config = (FeatureImpl.BeforeAnalysisAccessImpl) access;
+
+        ImageClassLoader classLoader = config.getImageClassLoader();
+
+        for (Map.Entry<String, List<byte[]>> entry : ImageSingletons.lookup(Resources.ResourcesSupport.class).resources.entrySet()) {
+            String fileName = entry.getKey();
+
+            if (fileName.startsWith("META-INF/services/")) {
+                // TODO: `.get(0)`
+                parseProviderConfigurationFile(classLoader, entry.getValue().get(0));
+            }
+        }
+    }
+
+    private void parseProviderConfigurationFile(ImageClassLoader classLoader, byte[] content) {
+        // TODO: remove doc
+        // From https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#The_META-INF_directory
+        // A service provider identifies itself by placing a provider-configuration file in the resource directory META-INF/services.
+        // The file's name should consist of the fully-qualified name of the abstract service class.
+        // The file should contain a newline-separated list of unique concrete provider-class names.
+        // Space and tab characters, as well as blank lines, are ignored.
+        // The comment character is '#' (0x23); on each line all characters following the first comment character
+        // are ignored. The file must be encoded in UTF-8.
+
+        // TODO: better regex
+        String[] lines = new String(content, StandardCharsets.UTF_8).split("\n");
+
+        for (String line : lines) {
+            // TODO: combine regexes
+            String className = line
+                    .replaceAll("[ \t]", "")
+                    .replaceAll("#.*", "");
+
+
+            Class<?> clazz = classLoader.findClassByName(className, false);
+
+            // TODO: remove me
+            System.out.println("FOUND: " + className);
+
+            RuntimeReflection.register(clazz);
+            RuntimeReflection.register(clazz.getDeclaredConstructors());
+        }
+    }
+}

[vhiairrassary]

With automatic detection: https://github.com/oracle/graal/pull/566

[cstancu]

This feature, i.e., automatically registering services for run-time lookup using ServiceLoader, is now enabled by default and can be disabled using -H:-UseServiceLoaderFeature. To enable tracing for this feature use -H:+TraceServiceLoaderFeature.