Quarkus: RESTEasy JAX-RS Client Issues in Native Mode

[sherl0cks]

Describe the bug
I'm using quarkus-rest-client and quarkus-jackson as described https://quarkus.io/guides/rest-client and https://quarkus.io/guides/kotlin#kotlin-and-jackson to create a JAX-RS REST client. This is a Kotlin app, built as native, and deployed to AWS Lambda.

The app build fine as a native image and deploys fine, but when the app boots, it crashes with the below stack trace. We can work around this issue by adding java.lang.String to my @RegisterForReflection, but I want to know why I see this issue and help remove the papercut.

Exception in thread "Lambda Thread" java.lang.NoClassDefFoundError: java.lang.String
    at org.apache.commons.logging.impl.LogFactoryImpl.class$(LogFactoryImpl.java:210)
    at org.apache.commons.logging.impl.LogFactoryImpl.<init>(LogFactoryImpl.java:210)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(DynamicHub.java:778)
    at org.apache.commons.logging.LogFactory.createFactory(LogFactory.java:1047)
    at org.apache.commons.logging.LogFactory$2.run(LogFactory.java:960)
    at java.security.AccessController.doPrivileged(AccessController.java:69)
    at org.apache.commons.logging.LogFactory.newFactory(LogFactory.java:957)
    at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:624)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
    at org.apache.http.conn.ssl.DefaultHostnameVerifier.<init>(DefaultHostnameVerifier.java:82)
    at org.apache.http.conn.ssl.DefaultHostnameVerifier.<init>(DefaultHostnameVerifier.java:91)
    at org.jboss.resteasy.client.jaxrs.engines.ClientHttpEngineBuilder43.build(ClientHttpEngineBuilder43.java:66)
    at org.jboss.resteasy.client.jaxrs.internal.ResteasyClientBuilderImpl.build(ResteasyClientBuilderImpl.java:393)
    at org.jboss.resteasy.microprofile.client.RestClientBuilderImpl.build(RestClientBuilderImpl.java:267)
    at io.quarkus.restclient.runtime.RestClientBase.create(RestClientBase.java:54)
    at com.acme.RestClient_b4a61e0400049cfbfd0cbc657165517695a0e196_Synthetic_Bean.create(RestClient_b4a61e0400049cfbfd0cbc657165517695a0e196_Synthetic_Bean.zig:74)
    at com.acme.RestClient_b4a61e0400049cfbfd0cbc657165517695a0e196_Synthetic_Bean.get(RestClient_b4a61e0400049cfbfd0cbc657165517695a0e196_Synthetic_Bean.zig:102)
    at com.acme.RestClient_b4a61e0400049cfbfd0cbc657165517695a0e196_Synthetic_Bean.get(RestClient_b4a61e0400049cfbfd0cbc657165517695a0e196_Synthetic_Bean.zig:241)
    at io.quarkus.arc.impl.CurrentInjectionPointProvider.get(CurrentInjectionPointProvider.java:53)
    at com.acme.Config_Bean.create(Config_Bean.zig:357)
    at com.acme.Config_Bean.create(Config_Bean.zig:32)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:80)
    at io.quarkus.arc.impl.ComputingCache$CacheFunction.lambda$apply$0(ComputingCache.java:99)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
    at io.quarkus.arc.impl.ComputingCache.getValue(ComputingCache.java:41)
    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:25)
    at com.acme.Config_ClientProxy.arc$delegate(Config_ClientProxy.zig:81)
    at com.acme.Config_ClientProxy.arc_contextualInstance(Config_ClientProxy.zig:423)
    at com.acme.Config_ProducerMethod_Manager_9eb74c56de466df63cb1d8b9c22aee2d83bf9dd7_Bean.create(Config_ProducerMethod_Manager_9eb74c56de466df63cb1d8b9c22aee2d83bf9dd7_Bean.zig:244)
    at com.acme.Config_ProducerMethod_Manager_9eb74c56de466df63cb1d8b9c22aee2d83bf9dd7_Bean.get(Config_ProducerMethod_Manager_9eb74c56de466df63cb1d8b9c22aee2d83bf9dd7_Bean.zig:135)
    at com.acme.Config_ProducerMethod_Manager_9eb74c56de466df63cb1d8b9c22aee2d83bf9dd7_Bean.get(Config_ProducerMethod_Manager_9eb74c56de466df63cb1d8b9c22aee2d83bf9dd7_Bean.zig:120)
    at com.acme.Handler_Bean.create(Handler_Bean.zig:377)
    at com.acme.Handler_Bean.get(Handler_Bean.zig:227)
    at com.acme.Handler_Bean.get(Handler_Bean.zig:305)
    at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:355)
    at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:367)
    at io.quarkus.arc.impl.ArcContainerImpl$1.get(ArcContainerImpl.java:222)
    at io.quarkus.arc.impl.ArcContainerImpl$1.get(ArcContainerImpl.java:219)
    at io.quarkus.arc.runtime.ArcRecorder$2$1.create(ArcRecorder.java:79)
    at io.quarkus.arc.runtime.BeanContainer.instance(BeanContainer.java:19)
    at io.quarkus.amazon.lambda.runtime.AmazonLambdaRecorder$2.run(AmazonLambdaRecorder.java:151)
    at java.lang.Thread.run(Thread.java:748)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)

Expected behavior

1. JAX-RS client and it's dependencies to be created during the native image build, not at runtime. If there is an issue, fail the build.

1. In the event that runtime creation of the JAX-RS client is by design, then I should not have to worry adding language primitives to @RegisterForReflection

Actual behavior
Crash at runtime.

To Reproduce
Steps to reproduce the behavior:

1. Declare a JAX-RS client
2. Make sure your code has no references to java.lang.String (ours in Kotlin, so it references Kotlin strings). This will make sure Graal doesn't see the reference.
3. Build and deploy
4. Invoke running app

Let me know if we need a reproducer app to test this.

Environment (please complete the following information):

• Darwin sunrun.lan 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64
• Output of java -version: Graal 19.2.1
• GraalVM version (if different from Java): Graal 19.2.1
• Quarkus version or git rev: 1.1.1

[geoand]

Would it be possible to test with Quarkus 1.3.0.Final (you will need to use quarkus-bom instead of quarkus-universe-bom)?

There have been a ton of changes since 1.1.1.Final.

[sherl0cks]

Sure - didn't realize it had been released @geoand. We've been waiting to upgrade to 1.3.0.Final. Resolving the dependencies now and will get back to you shortly.

[sherl0cks]

@geoand has io.quarkus:quarkus-universe-bom:1.3.0.Final been released yet? I cannot resolve it. I can resolve io.quarkus:quarkus-bom:1.3.0.Final

[geoand]

Yeah, for now you can only use the non-universe BOM.

The universe BOM will be released in a couple days

[sherl0cks]

@geoand Were their changes to @QuarkusTest? It appears that constructor injection for JUnit tests does not work anymore. I'm going to have to rewrite a fair amount of code in order to verify this in 1.3.0.Final if this feature no longer works, which will delay my feedback.

[geoand]

Yes there were major changes to the test infrastructure.

What kind of issue are you seeing with the constructor?
Asking because there is a good chance it has been fixed already. If it's not, we should definitely fix it before 1.3.1.Final

[sherl0cks]

@QuarkusTest
class InMemoryTest @Inject constructor(inMemoryTestFixture: InMemoryTestFixture) :
    OutsideInTestSuite(inMemoryTestFixture){

}

Leads to

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [com.sunrun.InMemoryTestFixture inMemoryTestFixture] in constructor [public com.sunrun.InMemoryTest(com.sunrun.InMemoryTestFixture)].

[geoand]

That has already been fixed and will be part of the 1.3.1.Final release.

In the meantime if you want to test, you will probably have to build from master

[sherl0cks]

😭 was reeeallly hoping we could move to 1.3.0.Final. When do you expect 1.3.1.Final?

I'll see if I can refactor the code in the meantime.

[geoand]

If all goes well, it should be out around the last week of the month

[sherl0cks]

@geoand Sounds good. Will keep an eye out.

What version of graal should I be building against? Is there a migration guide or docs out there on 1.3.0.Final?

[geoand]

19.3.1 is the supported GraalVM version.

We do have a migration guide in the wiki of the repo (sorry, but I'm on a phone and I don't have the link)

[gsmet]

Here it is: https://github.com/quarkusio/quarkus/wiki/Migration-Guide-1.3

[sherl0cks]

@geoand Managed to push through dependency hell faster than I thought. The issue as described above persists in 1.3.0.Final, but the work around with @RegisterForReflection on java.lang.String works as well.

The issue around Kotlin Builtins (https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/kotlin_builtins.20not.20found.20in.20Classpath.3F) is also present still, though it's a new stack trace since quarkus 1.3.0 bumps the kotlin version. I'll try to get you a reproducer on that. May take some time.

[geoand]

Thanks a lot! Looking forward to the reproducer

[sherl0cks]

@kenfinnigan bump. Is there a quick fix here where the extension registers java.lang.String for reflection?

[kenfinnigan]

TBH I don't know, my concern would be if the extension does that then it will always be present irrespective of whether it's needed. In and of itself it's not a huge impact, but if we take the same approach in all places something like this happens it becomes a bigger problem fast.

I'd also be curious to know whether a reproducer outside AWS Lambda has the same issue.

[sherl0cks]

@kenfinnigan well java.lang.String here is a reflective dependency of the logging framework in the RESTEast client. If the extension is used without swapping out the logging, that class is required.

Probably the reason this was not caught is because folks use Java so graal always sees java.lang.String on the heap. We're using Kotlin, which has its own String impl, which is what I think exposed this issue.

As to the reproducer, I'll have to test that, but may be a week or so before I can look again.

[kenfinnigan]

I wonder if we need to exclude Apache Commons Logging from the RESTEasy Client dependency?

I think Quarkus as a whole uses JBoss Logging instead

[gsmet]

@kenfinnigan Commons logging is banned from our dependencies via the enforcer.

@sherl0cks do you have something else bringing commons-logging? If so you should get rid of it and make sure you have org.jboss.logging:commons-logging-jboss-logging instead. If not, we really need a reproducer.

[sherl0cks]

Closing. I am no longer able to produce this issue in quarkus 1.5.0 with a pom that has common logging removed and jboss commons added.