Quarkus: Native compilation fails with MP Rest Client

[pilhuhn]

See also https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/Native.20and.20rest-client

Describe the bug

Create an app with the MP RestClient (or apparently also GraphQL client -- see https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/graphql.20microprofile/near/198970978 )

then try to compile this to native

This ends up in something like

Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of sun.security.provider.NativePRNG are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use -H:+TraceClassInitialization.
Detailed message:
Trace: Object was reached by
    reading field java.security.SecureRandom.secureRandomSpi of
        constant java.security.SecureRandom@4fa8cb4 reached by
    reading field sun.security.ssl.SSLContextImpl.secureRandom of
        constant sun.security.ssl.SSLContextImpl$TLSContext@67cd2da reached by
    reading field sun.security.ssl.SSLSocketFactoryImpl.context of
        constant sun.security.ssl.SSLSocketFactoryImpl@7c8befbb reached by
    reading field org.apache.http.conn.ssl.SSLConnectionSocketFactory.socketfactory of
        constant org.apache.http.conn.ssl.SSLConnectionSocketFactory@652a8d99 reached by
    reading field java.util.concurrent.ConcurrentHashMap$Node.val of
        constant java.util.concurrent.ConcurrentHashMap$Node@7e52f699 reached by
    indexing into array
        constant java.util.concurrent.ConcurrentHashMap$Node[]@31ebe5c5 reached by
    reading field java.util.concurrent.ConcurrentHashMap.table of
        constant java.util.concurrent.ConcurrentHashMap@2089b9a9 reached by
    reading field org.apache.http.config.Registry.map of
        constant org.apache.http.config.Registry@61ba6f03 reached by
    reading field org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.socketFactoryRegistry of
        constant org.apache.http.impl.conn.DefaultHttpClientConnectionOperator@1e443cae reached by
    reading field org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connectionOperator of
        constant org.apache.http.impl.conn.PoolingHttpClientConnectionManager@f07065f reached by
    reading field org.apache.http.impl.client.HttpClientBuilder$2.val$cm of
        constant org.apache.http.impl.client.HttpClientBuilder$2@544a1d92 reached by
    indexing into array
        constant java.lang.Object[]@34752250 reached by
    reading field java.util.ArrayList.elementData of
        constant java.util.ArrayList@4bf1b087 reached by
    reading field org.apache.http.impl.client.InternalHttpClient.closeables of
        constant org.apache.http.impl.client.InternalHttpClient@4c454c5 reached by
    reading field org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.httpClient of
        constant org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient43Engine@643e52f7 reached by
    reading field org.jboss.resteasy.client.jaxrs.internal.ResteasyClientImpl.httpEngine of
        constant org.jboss.resteasy.microprofile.client.impl.MpClient@165deefb reached by
    reading field org.jboss.resteasy.microprofile.client.ProxyInvocationHandler.client of
        constant org.jboss.resteasy.microprofile.client.ProxyInvocationHandler@432dfe97 reached by
    reading field java.lang.reflect.Proxy.h of
        constant com.sun.proxy.$Proxy338@392f433f reached by
    reading field com.redhat.cloud.policies.app.auth.RbacFilter.rbac of
        constant com.redhat.cloud.policies.app.auth.RbacFilter_Subclass@38e67950 reached by
    reading field org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$OnDemandInterceptorFactory.interceptor of
        constant org.jboss.resteasy.core.interception.jaxrs.JaxrsInterceptorRegistryImpl$OnDemandInterceptorFactory@1a2c343c reached by
    indexing into array

Expected behavior

App can be compiled to native

This is with Quarkus 1.8.2

[quarkusbot]

/cc @phillip-kruger

[missourian55]

Same problem exists in 1.10.0.Final

Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of sun.security.provider.NativePRNG are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=sun.security.provider.NativePRNG.
Detailed message:
Trace: Object was reached by
    reading field java.security.SecureRandom.secureRandomSpi of
        constant java.security.SecureRandom@60e2425 reached by
    reading field sun.security.ssl.SSLContextImpl.secureRandom of
        constant sun.security.ssl.SSLContextImpl$TLSContext@6b5d19cf reached by
    reading field sun.security.ssl.SSLSocketFactoryImpl.context of
        constant sun.security.ssl.SSLSocketFactoryImpl@3d18c639 reached by
    reading field org.apache.http.conn.ssl.SSLConnectionSocketFactory.socketfactory of
        constant org.apache.http.conn.ssl.SSLConnectionSocketFactory@1726ceef reached by
    reading field java.util.concurrent.ConcurrentHashMap$Node.val of
        constant java.util.concurrent.ConcurrentHashMap$Node@466b91b reached by
    indexing into array
        constant java.util.concurrent.ConcurrentHashMap$Node[]@5eae4e26 reached by
    reading field java.util.concurrent.ConcurrentHashMap.table of
        constant java.util.concurrent.ConcurrentHashMap@46ab82a8 reached by
    reading field org.apache.http.config.Registry.map of
        constant org.apache.http.config.Registry@38e1fe46 reached by
    reading field org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.socketFactoryRegistry of
        constant org.apache.http.impl.conn.DefaultHttpClientConnectionOperator@4ccf48a0 reached by
    reading field org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connectionOperator of
        constant org.apache.http.impl.conn.PoolingHttpClientConnectionManager@5714a15f reached by
    reading field org.apache.http.impl.client.HttpClientBuilder$2.val$cm of
        constant org.apache.http.impl.client.HttpClientBuilder$2@793410dd reached by
    indexing into array
        constant java.lang.Object[]@5bc84d20 reached by
    reading field java.util.ArrayList.elementData of
        constant java.util.ArrayList@4268f9f0 reached by
    reading field org.apache.http.impl.client.InternalHttpClient.closeables of
        constant org.apache.http.impl.client.InternalHttpClient@56b30498 reached by
    reading field org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.httpClient of
        constant org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient43Engine@62293a5d reached by
    reading field org.jboss.resteasy.client.jaxrs.internal.ResteasyClientImpl.httpEngine of
        constant org.jboss.resteasy.microprofile.client.impl.MpClient@b2447fe reached by
    reading field org.jboss.resteasy.microprofile.client.ProxyInvocationHandler.client of
        constant org.jboss.resteasy.microprofile.client.ProxyInvocationHandler@270331c8 reached by

[missourian55]

I am able to force the MP Rest client to use the URLConnection instead of Apache http client with below configuration. It took me hours to figure this out, would be helpful if we add this in the Quarkus Rest Client documentation until we find a permanent fix for Apache Http Client in native mode

quarkus.native.additional-build-args =\
    -H:EnableURLProtocols=https,\
    -J-Dorg.jboss.resteasy.microprofile.defaultToURLConnectionHttpClient=true

[gsmet]

@missourian55 any chance you could share a small reproducer?

@geoand I'm surprised the Apache HTTP client doesn't work in native? I would have thought we had fixed that for ages.

[geoand]

The Apache HTTP client should work in native.
We use it in some extensions without issue (in the spring cloud config extension for example).

[missourian55]

@gsmet

Here is the reproducer, I used Quarkus S3 guide as the base and added other required extensions

Steps to reproduce:

1. Clone the repo https://github.com/missourian55/s3-demo.git
2. Execute this native build
   ./mvnw clean package -Pnative -Dnative-image.docker-build=true -Dquarkus.container-image.build=true -Dnative-image.xmx=6g
3. You will see below stacktrace at build time

Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of sun.security.provider.NativePRNG are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use -H:+TraceClassInitialization.
Detailed message:
Trace: Object was reached by
    reading field java.security.SecureRandom.secureRandomSpi of
        constant java.security.SecureRandom@503562f4 reached by
    reading field sun.security.ssl.SSLContextImpl.secureRandom of
        constant sun.security.ssl.SSLContextImpl$TLSContext@668c6f89 reached by
    reading field sun.security.ssl.SSLSocketFactoryImpl.context of
        constant sun.security.ssl.SSLSocketFactoryImpl@33ebc721 reached by
    reading field org.apache.http.conn.ssl.SSLConnectionSocketFactory.socketfactory of
        constant org.apache.http.conn.ssl.SSLConnectionSocketFactory@4d9c51c7 reached by
    reading field java.util.concurrent.ConcurrentHashMap$Node.val of

1. Add below property to the application.properties and execute the build and notice build success

quarkus.native.additional-build-args =\
    -J-Dorg.jboss.resteasy.microprofile.defaultToURLConnectionHttpClient=true

Thanks in advance.

[geoand]

I'll take a look

[pilhuhn]

@geoand I reported the issue with https://github.com/RedHatInsights/policies-ui-backend
I did create a small reproducer at https://github.com/pilhuhn/quarkus-rest-json-quickstart which compiles to native on rhel8 (but fails to link).
So there must be more involved than purely the rest-client

[geoand]

@pilhuhn using your reproducer I couldn't reproduce a problem (on Ubuntu), so if there is an issue, it's probably different than the original issue

[pilhuhn]

This is what I tried to say - the minimal reproducer does not show the issue, so it must be something different in the larger context of the policy-ui-backend.
Did the minimal one link (so completely succeed)? If so, I'll open a new issue for that.

[geoand]

Cool, thanks

[geoand]

I tried the reproducer that @missourian55 linked and could indeed reproduce the problem.

There appears to be an easy solution which is to add:

quarkus.native.additional-build-args=--rerun-class-initialization-at-runtime=sun.security.provider.NativePRNG

I think it makes for the quarkus-rest-client extension to force reinitialization of this class (thus removing the burden from the user application), but I would to like to hear what @gsmet and @galderz think.

[gsmet]

Given it's a very common issue, if it's a valid workaround, I would even do it in Core. But I'm not sure of all the consequences.

Let's wait for @galderz 's opinion.

/cc @dmlloyd too

[missourian55]

@geoand Thanks I have tried the workaround, is it safe to ignore this warning?

Warning: Using a deprecated option --rerun-class-initialization-at-runtime. 
Currently there is no replacement for this option.
 Try using --initialize-at-run-time or use the non-API option -H:ClassInitialization directly.

[geoand]

@missourian55 yes, it is safe to ignore that warning. If we provide the support on our side eventually, that warning won't be present at all.

[galderz]

@gsmet @geoand The workaround might work but I feel it's hiding the real issue: why is SSL connection factory being created a cached? Seems to me something like that should be done at runtime... which if you follow back to the top, the question is, why is the HTTP client be initialised at image build time? Does it make sense to make such a thing at image build time? I feel that's the real issue to fix, avoid initialising things that really should be initialised at runtime. Connections (e.g. netty, http clients and similar) and security are things for which the majority of work should be done at runtime.

[geoand]

That is probably right, but I am not very familiar with the implementation of the client.

@michalszynkiewicz care to take a look at this one?

[michalszynkiewicz]

Thanks @geoand, added to my todos :)

[michalszynkiewicz]

In @pilhuhn 's project, the problem seems to occur only if the RbacFilter (a ContainerRequestFilter) is injecting a rest client stub.
Digging deeper :pick: