Graal: cannot build a vertx native image with 19.3.0

[pmlopes]

With 19.2.0.1 (and earlier releases) it was possible to build vertx native images without the need for substitutions. With 19.3.0 this isn't possible anymore.

The root cause seems to be related to DNS related code where Inet?Address objects are allocated in the heap.

After tracking down where these allocations came from I managed to point to the classes:

io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider
io.netty.resolver.dns.DnsServerAddressStreamProviders
io.netty.util.NetUtil
io.vertx.core.impl.AddressResolver

Adding these classes to the initialize at runtime parameter will not work because:

io.netty.util.NetUtil

Is always initialized at compile time, regardless of what the configuration I use.

A full reproducer can be found here:

https://github.com/vertx-howtos/graal-native-image-howto/tree/issues/graal-19-3/steps/step-9

I'm out of ideas on how to solve this.

[gwenneg]

Hi @pmlopes! We had a similar issue in Quarkus, maybe you'll find something useful in https://github.com/quarkusio/quarkus/pull/5353.

[pmlopes]

@gwenneg thanks for the tip. As you noted it's a workaround for the new limitations, we (vertx) would prefer not to had those to our codebase and just run with what Graal offers.

IMHO I thing if the issue affects vertx and Quarkus it probably affects others too and maybe should be addressed upstream.

[vjovanov]

It is technically possible to allow INetAddress in a heap but it would require some code duplication with the JDK. My concern is that this is not a good thing except the address points to something standard such as 127.0.0.1.

What is the address in your case?

One option would be to implement a check whether a "reasonable" address is stored in the heap.
This is all extra work, so I would not do it if this is not something very important that happens quite often.

[pmlopes]

The issue seems to come from the NetUtil class in netty: https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/NetUtil.java and it does look like they are caching localhost values in ipv4 and ipv6 representations.

My issue isn't the inet addresses per se but the fact that I cannot delay that class to be initialized at runtime at all. If you look further at the class, it really shows it should be because it caches not only the localhost values (probably harmless) but also all the inet devices and that is not good to keep in the heap as my laptop on the docking station and running docker has a ton of virtual devices ;-)

[vjovanov]

Have you tried with --initialize-at-run-time=io.netty.util.NetUtil?

[pmlopes]

Yes, but no matter what the configuration is it always gets initialized at compile time. I think I wrote a note on it before.

[vjovanov]

OK, the problem comes from the fact that the class initialization tracking agent is not working for the JDK classes. I will address this separately ASAP.

In the meanwhile, this is the magic sauce:
--initialize-at-run-time=io.netty.channel.DefaultChannelId,io.netty.util.NetUtil,io.netty.channel.socket.InternetProtocolFamily,io.netty.resolver.HostsFileEntriesResolver,io.netty.resolver.dns.DnsNameResolver,io.netty.resolver.dns.DnsServerAddressStreamProviders,io.netty.resolver.dns.PreferredAddressTypeComparator\$1,io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider

[sstubbs]

It works if I add these which is great. I'm getting these warnings. Not sure if they are important.

[main] WARN io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider - Default DNS servers: [/8.8.8.8:53, /8.8.4.4:53] (Google Public DNS as a fallback)
[vert.x-eventloop-thread-0] WARN io.netty.channel.DefaultChannelId - Failed to find the current process ID from ''; using a random value: 71011194

[pmlopes]

@sstubbs that is a warning from netty when the code that is supposed to parse the system dns config fails. It could be related as that code is usually called on a static constructor i think.

[sstubbs]

I see ok thanks. It seems to be running so I will continue to test it and see what happens.

[shaharoliver]

OK, the problem comes from the fact that the class initialization tracking agent is not working for the JDK classes. I will address this separately ASAP.

In the meanwhile, this is the magic sauce:
--initialize-at-run-time=io.netty.channel.DefaultChannelId,io.netty.util.NetUtil,io.netty.channel.socket.InternetProtocolFamily,io.netty.resolver.HostsFileEntriesResolver,io.netty.resolver.dns.DnsNameResolver,io.netty.resolver.dns.DnsServerAddressStreamProviders,io.netty.resolver.dns.PreferredAddressTypeComparator\$1,io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider

can you share how you managed to do that? i'm using 19.3.0 on MacOS , even after adding these i'm still getting :

Error: Classes that should be initialized at run time got initialized during image building:
io.netty.util.NetUtil the class was requested to be initialized at build time (from the command line). io.netty.util.NetUtil has been initialized without the native-image initialization instrumentation and the stack trace can't be tracked. Try avoiding to initialize the class that caused initialization of io.netty.util.NetUtil
io.netty.channel.socket.InternetProtocolFamily the class was requested to be initialized at build time (from the command line). io.netty.channel.socket.InternetProtocolFamily has been initialized without the native-image initialization instrumentation and the stack trace can't be tracked. Try avoiding to initialize the class that caused initialization of io.netty.channel.socket.InternetProtocolFamily

Simple vertx project with the following native-image parameters:

                    -H:+TraceClassInitialization
                    --verbose
                    --initialize-at-build-time=io.vertx,com.fasterxml.jackson
                    --initialize-at-run-time=io.netty.channel.DefaultChannelId,
                    --initialize-at-run-time=io.netty.util.NetUtil
                    --initialize-at-run-time=io.netty.channel.socket.InternetProtocolFamily
                    --initialize-at-run-time=io.netty.resolver.HostsFileEntriesResolver
                    --initialize-at-run-time=io.netty.resolver.dns.DnsNameResolver,
                    --initialize-at-run-time=io.netty.resolver.dns.DnsServerAddressStreamProviders
                    --initialize-at-run-time=io.netty.resolver.dns.PreferredAddressTypeComparator\$1,
                    --initialize-at-run-time=io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider
                    --initialize-at-run-time=io.netty.buffer.AbstractReferenceCountedByteBuf
                    --initialize-at-run-time=io.vertx.core.net.impl.PartialPooledByteBufAllocator
                    --initialize-at-run-time=io.netty.handler.codec.http.websocketx.extensions.compression.DeflateEncoder
                    --initialize-at-run-time=io.netty.handler.codec.http.websocketx.extensions.compression.DeflateDecoder
                    --initialize-at-run-time=io.netty.handler.codec.http.HttpObjectEncoder
                    --initialize-at-run-time=io.netty.handler.codec.http.websocketx.WebSocket00FrameEncoder
                    --initialize-at-run-time=io.netty.handler.codec.http2.Http2CodecUtil
                    --initialize-at-run-time=io.netty.handler.codec.http2.Http2ConnectionHandler
                    --initialize-at-run-time=io.netty.handler.codec.http2.DefaultHttp2FrameWriter
                    --initialize-at-run-time=io.netty.util.internal.logging.Log4JLogger
                    --initialize-at-run-time=io.netty.handler.ssl.ReferenceCountedOpenSslServerContext
                    --initialize-at-run-time=io.netty.handler.ssl.JdkNpnApplicationProtocolNegotiator
                    --initialize-at-run-time=io.netty.handler.ssl.ReferenceCountedOpenSslEngine
                    --initialize-at-run-time=io.netty.handler.ssl.ConscryptAlpnSslEngine
                    --initialize-at-run-time=io.netty.handler.ssl.JettyNpnSslEngine
                    --initialize-at-run-time=io.netty.handler.ssl.ReferenceCountedOpenSslContext
                    --initialize-at-run-time=io.netty.handler.ssl.ReferenceCountedOpenSslClientContext
                    --initialize-at-run-time=io.vertx.core.net.impl.transport.EpollTransport
                    --initialize-at-run-time=io.vertx.core.net.impl.transport.KQueueTransport
                    --initialize-at-run-time=io.vertx.core.http.impl.VertxHttp2ClientUpgradeCodec
                    --initialize-at-run-time=io.vertx.core.eventbus.impl.clustered.ClusteredEventBus
                    --report-unsupported-elements-at-runtime
                    --allow-incomplete-classpath
                    --enable-all-security-services
                    -Djava.net.preferIPv4Stack=true

[vjovanov]

I put breakpoints in <clinit> and <init> and see where stuff gets initialized.
I fixed the initialization tracking of Netty in master. You could try from there and the error message should tell you how this class got initialized.

[shaharoliver]

I put breakpoints in <clinit> and <init> and see where stuff gets initialized.
I fixed the initialization tracking of Netty in master. You could try from there and the error message should tell you how this class got initialized.

You mean by debugging the native image generation when it's running as a server and using --debug-attach ?

[vjovanov]

yes, doesn't have to run as a server.

[cvgaviao]

OK, the problem comes from the fact that the class initialization tracking agent is not working for the JDK classes. I will address this separately ASAP.

In the meanwhile, this is the magic sauce:
--initialize-at-run-time=io.netty.channel.DefaultChannelId,io.netty.util.NetUtil,io.netty.channel.socket.InternetProtocolFamily,io.netty.resolver.HostsFileEntriesResolver,io.netty.resolver.dns.DnsNameResolver,io.netty.resolver.dns.DnsServerAddressStreamProviders,io.netty.resolver.dns.PreferredAddressTypeComparator\$1,io.netty.resolver.dns.DefaultDnsServerAddressStreamProvider

I've tried once more to build my vertx app with native-image, but this time I used the version 20.0.0 and using the properties that @vjovanov have suggested. But still no luck.

[INFO] Error: Classes that should be initialized at run time got initialized during image building:
[INFO]  io.netty.util.NetUtil the class was requested to be initialized at build time (from the command line). io.netty.util.NetUtil has been initialized without the native-image initialization instrumentation and the stack trace can't be tracked. Try avoiding to initialize the class that caused initialization of io.netty.util.NetUtil
[INFO] 
[INFO] com.oracle.svm.core.util.UserError$UserException: Classes that should be initialized at run time got initialized during image building:
[INFO]  io.netty.util.NetUtil the class was requested to be initialized at build time (from the command line). io.netty.util.NetUtil has been initialized without the native-image initialization instrumentation and the stack trace can't be tracked. Try avoiding to initialize the class that caused initialization of io.netty.util.NetUtil
[INFO] 
[INFO]  at com.oracle.svm.core.util.UserError.abort(UserError.java:68)

-H:+TraceClassInitialization and -H:+ReportExceptionStackTraces doesn't help to show who is initializing it.

[cvgaviao]

@jkremser, may I ask you to give us any statement about this issue?

[jvican]

I'm also running into this with GraalVM Native 20.1.0 and vertx 9.3.1 (latest release so far). I already have the magic sauce mentioned by @vjovanov in my configuration but the error persists. If you have ideas about how to work around it, please let me know and I'll be happy to give them a try.

[InfoSec812]

+1 Same problem here...

Graal Version:

openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment GraalVM CE 20.1.0 (build 11.0.7+10-jvmci-20.1-b02)
OpenJDK 64-Bit Server VM GraalVM CE 20.1.0 (build 11.0.7+10-jvmci-20.1-b02, mixed mode, sharing)

Vert.x Version: 3.8.4

[vjovanov]

We are reworking class initialization tracking to work in 100% of the cases (CC @gradinac). However, here I believe the simplest solution would be not to initiaIize all of netty at build time. Then, all should work out of the box.

[borag]

+1

[gradinac]

Hello everyone!

We've changed the way TraceClassInitialization works for 20.3+. From the user perspective, you can now do --trace-class-initialization=<comma-separated-list-of-classes-to-trace>. Behind the scenes, we now use a JVMTI agent that traces class initialization/object instantiation using JVMTI events and breakpoints.

@cvgaviao Could you see if this can help you resolve your issue?
You will need a GraalVM version that includes https://github.com/oracle/graal/commit/64460fafeef63ee7b978b0238b0d5c07febbf5f9, you can either build GraalVM from master or wait until a community snapshot is out at https://github.com/graalvm/graalvm-ce-dev-builds/releases
You can then run the image build with an additional argument (based on your error message): --trace-class-initialization=io.netty.util.NetUtil

Let me know if you hit any issues

[t3rmian]

Hi there!

I've run the image build for a basic Vert.x (3.9.1) app using GraalVM CE 20.3.0-dev-20200912_0211 (graalvm-ce-java11-linux-amd64-dev) with --trace-class-initialization=io.netty.util.NetUtil and this is what I got:

Error: Classes that should be initialized at run time got initialized during image building:
 io.netty.util.NetUtil the class was requested to be initialized at run time (from the command line). io.netty.resolver.dns.DnsServerAddressStreamProviders$DefaultProviderHolder caused initialization of this class with the
 following trace:
        at io.netty.util.NetUtil.<clinit>(NetUtil.java:125)
        at io.netty.resolver.dns.UnixResolverDnsServerAddressStreamProvider.parse(UnixResolverDnsServerAddressStreamProvider.java:197)
        at io.netty.resolver.dns.UnixResolverDnsServerAddressStreamProvider.<init>(UnixResolverDnsServerAddressStreamProvider.java:95)
        at io.netty.resolver.dns.UnixResolverDnsServerAddressStreamProvider.<init>(UnixResolverDnsServerAddressStreamProvider.java:129)
        at io.netty.resolver.dns.UnixResolverDnsServerAddressStreamProvider.parseSilently(UnixResolverDnsServerAddressStreamProvider.java:69)
        at io.netty.resolver.dns.DnsServerAddressStreamProviders$DefaultProviderHolder$1.provider(DnsServerAddressStreamProviders.java:138)
        at io.netty.resolver.dns.DnsServerAddressStreamProviders$DefaultProviderHolder$1.<init>(DnsServerAddressStreamProviders.java:117)
        at io.netty.resolver.dns.DnsServerAddressStreamProviders$DefaultProviderHolder.<clinit>(DnsServerAddressStreamProviders.java:115)

So, I then added io.netty.resolver.dns.DnsServerAddressStreamProviders$DefaultProviderHolder to the properties from vertx-howtos and the build finished without issues. I was also able to start it as a native app.

Went back to the 20.2.0 version with the same settings, and it works without a snag!

[gradinac]

@t3rmian Thank you for verifying this! :)
I will be closing this issue for now but if any more problems arise we can either reopen it or create a new issue.