Quarkus: not a subtype on java 11 with fault-tolerance - when running on custom ForkJoinPool

[danowrw]

Describe the bug
When executing streams on custom ForkJoinPool with fault-tolerance annotation and java 11 Adopt you get below. This works on java 8 Adopt.

not a subtype
at java.base/java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
at org.acme.ExampleResource.notWorkingOnJava11Adopt(ExampleResource.java:36)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:621)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:487)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:437)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:362)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:439)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:400)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:374)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:67)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:488)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:259)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:160)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:362)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:163)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:245)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:123)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:36)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:87)
at io.quarkus.runtime.CleanableExecutor$CleaningRunnable.run(CleanableExecutor.java:231)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: java.util.ServiceConfigurationError: io.smallrye.config.SmallRyeConfigFactory: io.quarkus.runtime.configuration.QuarkusConfigFactory not a subtype
at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:588)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1236)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1264)
at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1299)
at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1384)
at io.smallrye.config.SmallRyeConfigProviderResolver.getFactoryFor(SmallRyeConfigProviderResolver.java:110)
at io.smallrye.config.SmallRyeConfigProviderResolver.getConfig(SmallRyeConfigProviderResolver.java:86)
at org.eclipse.microprofile.config.ConfigProvider.getConfig(ConfigProvider.java:105)
at io.smallrye.config.inject.ConfigProducer.getConfig(ConfigProducer.java:44)
at io.smallrye.config.inject.ConfigProducer.produceBooleanConfigProperty(ConfigProducer.java:86)
at io.smallrye.config.inject.ConfigProducer_ProducerMethod_produceBooleanConfigProperty_4052c0cbfaba0eb3bbfc4a1d91d5a3d235c1dbf5_Bean.create(ConfigProducer_ProducerMethod_produceBooleanConfigProperty_4052c0cbfaba0eb3bbfc4a1d91d5a3d235c1dbf5_Bean.zig:101)
at io.smallrye.config.inject.ConfigProducer_ProducerMethod_produceBooleanConfigProperty_4052c0cbfaba0eb3bbfc4a1d91d5a3d235c1dbf5_Bean.get(ConfigProducer_ProducerMethod_produceBooleanConfigProperty_4052c0cbfaba0eb3bbfc4a1d91d5a3d235c1dbf5_Bean.zig:344)
at io.smallrye.config.inject.ConfigProducer_ProducerMethod_produceBooleanConfigProperty_4052c0cbfaba0eb3bbfc4a1d91d5a3d235c1dbf5_Bean.get(ConfigProducer_ProducerMethod_produceBooleanConfigProperty_4052c0cbfaba0eb3bbfc4a1d91d5a3d235c1dbf5_Bean.zig:44)
at io.quarkus.arc.impl.CurrentInjectionPointProvider.get(CurrentInjectionPointProvider.java:53)
at io.smallrye.faulttolerance.metrics.MetricsCollectorFactory_Bean.create(MetricsCollectorFactory_Bean.zig:326)
at io.smallrye.faulttolerance.metrics.MetricsCollectorFactory_Bean.create(MetricsCollectorFactory_Bean.zig:32)
at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:79)
at io.quarkus.arc.impl.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:99)
at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
at io.quarkus.arc.impl.ComputingCache.getValue(ComputingCache.java:41)
at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:25)
at io.smallrye.faulttolerance.metrics.MetricsCollectorFactory_ClientProxy.arc$delegate(MetricsCollectorFactory_ClientProxy.zig:57)
at io.smallrye.faulttolerance.metrics.MetricsCollectorFactory_ClientProxy.createCollector(MetricsCollectorFactory_ClientProxy.zig:103)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$getMetricsCollector$10(FaultToleranceInterceptor.java:509)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at io.smallrye.faulttolerance.internal.StrategyCache.getMetrics(StrategyCache.java:27)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.getMetricsCollector(FaultToleranceInterceptor.java:509)
at io.smallrye.faulttolerance.FaultToleranceInterceptor.interceptCommand(FaultToleranceInterceptor.java:145)
at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(FaultToleranceInterceptor_Bean.zig:471)
at org.jboss.resteasy.microprofile.client.InvocationContextImpl$InterceptorInvocation.invoke(InvocationContextImpl.java:154)
at org.jboss.resteasy.microprofile.client.InvocationContextImpl.invokeNext(InvocationContextImpl.java:53)
at org.jboss.resteasy.microprofile.client.InvocationContextImpl.proceed(InvocationContextImpl.java:89)
at org.jboss.resteasy.microprofile.client.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:146)
at com.sun.proxy.$Proxy57.getOrgs(Unknown Source)
at org.acme.ExampleResource.getGithubOrgs(ExampleResource.java:63)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.acme.ExampleResource.lambda$notWorkingOnJava11Adopt$0(ExampleResource.java:35)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

Expected behavior
this should work on java 11 too

To Reproduce

1. Checkout https://github.com/danowrw/not-a-subtype
2. Run this on java 11 Adpot
3. call http://localhost:8080/not_working - see this do not work
4. call http://localhost:8080/working -see it works
5. switch to java 8 and see http://localhost:8080/not_working - is working now

[geoand]

cc @michalszynkiewicz @Ladicek @dmlloyd

[Ladicek]

Caused by: java.util.ServiceConfigurationError: io.smallrye.config.SmallRyeConfigFactory: io.quarkus.runtime.configuration.QuarkusConfigFactory not a subtype

Wonder how that could be caused by FT...

[Ladicek]

The reproducer has this comment:

    //below annotation cause execution on custom ForkJoinPool fails
    @Retry
    //This also fails for
    //@Timed
    //@Counted

So definitely sounds like not caused by FT.

[Ladicek]

I'll try the reproducer anyway, and perhaps it's fixed by using ConProp -- I'll report back.

[Ladicek]

Hmm, if I let the task run on an injected ManagedExecutor, it works fine, but if I run the task ThreadContext.contextualRunnable-ized on the custom FJ pool, then it fails :open_mouth:

[geoand]

So sounds like an MP - CP problem, no?

[Ladicek]

Actually, no. Turns out that the FJ pool sets the context classloader for all threads in the pool to the system classloader. This can be worked around by using a custom ForkJoinWorkerThreadFactory that returns a subclass of ForkJoinWorkerThread, which allows using a constructor that doesn't set the context classloader. Like this:

customThreadPool = new ForkJoinPool(4, new ForkJoinPool.ForkJoinWorkerThreadFactory() {
    @Override
    public ForkJoinWorkerThread newThread(ForkJoinPool pool) {
        return new ForkJoinWorkerThread(pool) {};
    }
}, null, false);

[jaikiran]

This looks like an issue with classloading. The ForkJoinPool uses the system classloader (which won't have access to some the application jars). Quarkus "fixes" this issue by hacking the "common fork join" pool to set the right classloader for it https://github.com/quarkusio/quarkus/blob/master/core/deployment/src/main/java/io/quarkus/runner/bootstrap/ForkJoinClassLoading.java#L13. However since the application here is creating a custom ForkJoinPool it won't get this "fix" by default. The application will have to use a custom ForkJoinWorkerThreadFactory to use the proper classloader. Looks like Java 8 and Java 11 APIs have changed in this area for ForkJoinPool, so I'm not sure (I haven't looked in detail) which API is applicable to get this working.

[Ladicek]

But, if you want to use thread pools manually, I'd suggest using MP CP ManagedExecutor. You can configure max number of running tasks and queue size by creating your own instance using ManagedExecutor.builder().

In any case, this is not a bug in Quarkus, I'd say.

[jaikiran]

Looks like Java 8 and Java 11 APIs have changed in this area for ForkJoinPool, so I'm not sure (I haven't looked in detail) which API is applicable to get this working.

@Ladicek's reply a few minutes before mine has the sample code which I think should fix this issue for the application
https://github.com/quarkusio/quarkus/issues/9397#issuecomment-630037633

[geoand]

But at the very least, we should have some piece in the documentation telling people about this?

[Ladicek]

I can't think of any immediately obvious place in the docs where we can put this and expect people to actually read it :-)

What I think we should do is make context classloader a propagatable item in MP CP. Because if people want to propagate the request scope, propagate current transaction, propagate security context, etc., to their thread pool tasks and use custom thread pools (such as arbitrary FJ pools), they have to ThreadContext.contextualRunnable-ize their Runnables anyway. If context classloader was also propagated, it would naturally work.

[Ladicek]

Hmm, but that seems to be there already: https://github.com/eclipse/microprofile-context-propagation/blob/master/api/src/main/java/org/eclipse/microprofile/context/ThreadContext.java#L235-L247

@FroMage @manovotn Do I understand it correctly that the Application context in MP ConProp also includes the context classloader? Because it looks like in Quarkus, the context classloader doesn't propagate, and I can't find any ThreadContextProvider in the Quarkus codebase that would capture the context classloader and later restore it.

[jaikiran]

Because it looks like in Quarkus, the context classloader doesn't propagate,

Hello @Ladicek, it's not that the context classloader doesn't propagate in Quarkus. The issue here is that the application is using an API (the ForkJoinPool constructor which takes just the 1 int param) which (as per its API construct) is instructed to use a defaultForkJoinWorkerThreadFactory which doesn't honour the current thread's context classloader. That's why switching to the multi-arg constructor (which allows you to specify which classloader to use) does the trick.

[Ladicek]

@jaikiran I'm specifically talking about MicroProfile Context Propagation at this point. If that included propagating the context classloader (at this point, I believe it should, but in Quarkus, it doesn't), then I could easily run ThreadContext.contextualRunnable(() -> { ... }) tasks on arbitrary thread pools, including custom FJ pool.

[FroMage]

It doesn't propagate the TCCL indeed. We've never had a use-case for it. I guess we could, but I would have to ask @stuartwdouglas his opinion about this because it may be harmful.

[manovotn]

Application context is underspecified in MP CP. The API mentions TCCL so in theory it could/should be there, but to my knowledge there are no TCKs testing this. SR doesn't implement this because, as Stephane said, we haven't had a use case yet.

[Ladicek]

Well, here's one :-)

[manovotn]

Yeah, definitely not against it.
I am just thinking... SR propagates everything by default, so assuming we add it there, anyone using it might be in for a surprise if they are relying in TCCL in their apps...
Alternatively, we could just add it into Quarkus for now, but I would also love to hear @stuartwdouglas opinion on that.

[stuartwdouglas]

How are you going to propagate it to the FJ pool? At least on JDK 8 there are no hooks for that, which is why I have the current crappy hack, not sure if the situation has improved in 11.

[FroMage]

If you're using MP-CP we can make a ThreadContextProvider that captures and restores the TCCL.

[Ladicek]

I was only talking about using MP CP, which users must do explicitly. (I'd personally even consider removing that crappy hack, if it's only for user applications -- when users move execution to a separate thread pool, they should use ConProp.)

EDIT: of course, the error message is impossible to fix :-( At least I'm not aware of a good way how to produce an error message that directly points people towards ConProp.

[stuartwdouglas]

Yea, but you can't do it automatically.
The big problem is CompletionStage Async methods, which use the pool by default with no way to capture the context. Hopefully the move to Mutiny means this is no longer a problem.

[Ladicek]

I think my point is that user should use ConProp explicitly if they are submitting tasks to arbitrary thread pools. (It's not just TCCL, it's also CDI request context, security context, etc.)

At least for now, there's no better way.

(And it would require ConProp to propagate the TCCL, which it currently doesn't :-) )

EDIT: actually I totally ignored your point about *Async methods in CompletionStage. There's ThreadContext.withContextCapture that transforms CompletionStage or CompletableFuture to one that has captured context, but such transformed CompletionStage or CompletableFuture will throw an exception if an *Async method without Executor is called. There's nothing we can do for *Async methods that don't take an Executor. Agree that isn't totally user-friendly, but that's a fault of the CompletionStage API, not ours :-)

[FroMage]

Yea, but you can't do it automatically.

With MP-CP and CompletionStage it's indeed always manual. But if you already have to wrap your CS to get the Request Context, you'll get the TCCL for free.

The big problem is CompletionStage Async methods, which use the pool by default with no way to capture the context. Hopefully the move to Mutiny means this is no longer a problem.

Once you've wrapped (contextualised) your CS, even Async methods are contextualised, so I'm not sure what you mean. And if you're not wrapping the CS you don't get any context, Async or not, which leads to your request not working if (almost inevitably: when) it ends up on the event loop.