micrometer-registry-statsd pollutes classpath with `/META-INF/native/libnetty_transport_native_epoll_x86_64.so` from Netty 4.1.29, breaking use of later versions of Netty

Created on 30 Oct 2018 · 8Comments · Source: micrometer-metrics/micrometer

When micrometer-registry-statsd is on the classpath and it appears before netty-transport-native-epoll, libnetty_transport_native_epoll_x86_64.so is found in micrometer-registry-statsd and loaded from there. This means that, even if you're using a later version of Netty, you're stuck with the shared library from 4.1.29. This causes a failure to be thrown from native code:

15:19:18.442 [reactor-http-client-epoll-8] WARN io.netty.channel.epoll.EpollEventLoop - Unexpected exception in the selector loop.
io.netty.channel.ChannelException: timerfd_settime() failed: Invalid argument
    at io.netty.channel.epoll.Native.epollWait0(Native Method)
    at io.netty.channel.epoll.Native.epollWait(Native.java:114)
    at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:253)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:278)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
    at java.lang.Thread.run(Thread.java:748)

This happens because -1 is passed into Native.epollWait0 where it should be given special treatment. Instead it's passed into timerfd_settime() which rejects it as invalid.

See https://github.com/netty/netty/issues/8444 for the (embarrassing) background.

Source

wilkinsona

All 8 comments

I think the solution is to rename libnetty_transport_native_epoll_x86_64.so to prefix it with the prefix that's applied when shading Netty, replacing . with _. So I think it should be named io_micrometer_shaded_libnetty_transport_native_epoll_x86_64.so.

wilkinsona on 30 Oct 2018

An alternative would be to exclude the file entirely. It's not being used at the moment anyway as it's named incorrectly to be found by NativeLibraryLoader. Starting a Boot app that uses the StatsD registry with debug logging enabled for the shaded Netty loggers produces output like this:

2018-10-30 17:12:20.890 DEBUG 112397 --- [           main] i.m.s.i.n.u.i.NativeLibraryLoader        : io_micrometer_shaded_netty_transport_native_epoll cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp

java.lang.UnsatisfiedLinkError: no io_micrometer_shaded_netty_transport_native_epoll in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at io.micrometer.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
    at io.micrometer.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
    at io.micrometer.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
    at io.micrometer.shaded.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:186)
    at io.micrometer.shaded.io.netty.channel.epoll.Native.<clinit>(Native.java:61)
    at io.micrometer.shaded.io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at io.micrometer.shaded.reactor.netty.resources.DefaultLoopEpoll.<clinit>(DefaultLoopEpoll.java:47)
    at io.micrometer.shaded.reactor.netty.resources.LoopResources.preferNative(LoopResources.java:216)
    at io.micrometer.shaded.reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:156)
    at io.micrometer.shaded.reactor.netty.udp.UdpResources.onClient(UdpResources.java:141)
    at io.micrometer.shaded.reactor.netty.udp.UdpClientConnect.connect(UdpClientConnect.java:39)
    at io.micrometer.shaded.reactor.netty.udp.UdpClientOperator.connect(UdpClientOperator.java:42)
    at io.micrometer.shaded.reactor.netty.udp.UdpClientOperator.connect(UdpClientOperator.java:42)
    at io.micrometer.shaded.reactor.netty.udp.UdpClientDoOn.connect(UdpClientDoOn.java:60)
    at io.micrometer.shaded.reactor.netty.udp.UdpClient.connect(UdpClient.java:146)
    at io.micrometer.statsd.StatsdMeterRegistry.start(StatsdMeterRegistry.java:245)
    at io.micrometer.statsd.StatsdMeterRegistry.<init>(StatsdMeterRegistry.java:219)
    at io.micrometer.statsd.StatsdMeterRegistry.<init>(StatsdMeterRegistry.java:85)
    at io.micrometer.statsd.StatsdMeterRegistry.<init>(StatsdMeterRegistry.java:73)
    at org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdMetricsExportAutoConfiguration.statsdMeterRegistry(StatsdMetricsExportAutoConfiguration.java:64)
    at org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdMetricsExportAutoConfiguration$$EnhancerBySpringCGLIB$$90853531.CGLIB$statsdMeterRegistry$2(<generated>)
    at org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdMetricsExportAutoConfiguration$$EnhancerBySpringCGLIB$$90853531$$FastClassBySpringCGLIB$$65908e73.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
    at org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdMetricsExportAutoConfiguration$$EnhancerBySpringCGLIB$$90853531.statsdMeterRegistry(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:620)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:605)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1288)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1127)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:273)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1239)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1166)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:855)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:758)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:508)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1288)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1127)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:204)
    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:236)
    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:224)
    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:100)
    at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:88)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:250)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:237)
    at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:54)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5098)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1429)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:944)
    at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:839)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1429)
    at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
    at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:944)
    at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:261)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:770)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.startup.Tomcat.start(Tomcat.java:370)
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:106)
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:86)
    at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:414)
    at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:174)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:179)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:152)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:540)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
    at sample.actuator.SampleActuatorApplication.main(SampleActuatorApplication.java:31)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.maven.AbstractRunMojo$LaunchRunner.run(AbstractRunMojo.java:558)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.UnsatisfiedLinkError: no io_micrometer_shaded_netty_transport_native_epoll in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at io.micrometer.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at io.micrometer.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
        at java.security.AccessController.doPrivileged(Native Method)
        at io.micrometer.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
        at io.micrometer.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
        ... 104 common frames omitted

2018-10-30 17:12:20.892 DEBUG 112397 --- [           main] i.m.s.r.n.resources.DefaultLoopEpoll     : Default Epoll support : false
2018-10-30 17:12:20.892 DEBUG 112397 --- [           main] i.m.s.r.n.resources.DefaultLoopKQueue    : Default KQueue support : false

wilkinsona on 30 Oct 2018

Could this please be fixed in a 1.0.x release in time for next week's Boot releases? As things stand we're stuck on Netty 4.1.29. If we try to upgrade our build hangs due to the native library pollution described above.

wilkinsona on 22 Nov 2018

👍1

@wilkinsona I confirmed that 1.1.0 has /META-INF/native/libnetty_transport_native_epoll_x86_64.so but 1.0.7 doesn't have it. Am I missing something?

izeye on 23 Nov 2018

Thanks for double-checking, @izeye. You're right. Sorry, I should have checked the built jars rather than looking at the build configuration and dependencies of 1.0.x and 1.1.x. I can't spot why the native library isn't part of 1.0.x, but I can see now that it isn't.

To reword my earlier request slightly:

Could this please be fixed and 1.1.1 released in time for next week's Boot releases? As things stand we're stuck on Netty 4.1.29. If we try to upgrade our build hangs due to the native library pollution described above.

wilkinsona on 23 Nov 2018

👍1

Netty versions seem different due to dependency locking: 4.1.29.Final vs 4.1.21.Final

izeye on 23 Nov 2018

I created #1027 to fix this.

izeye on 23 Nov 2018

Due to dependency locking, master depends on reactor-netty 0.8.1.RELEASE which has a classifier for its netty-transport-native-epoll dependency but 1.0.x depends on reactor-netty 0.7.4.RELEASE which doesn't have a classifier for its netty-transport-native-epoll dependency. I guess that's why 1.0.7 doesn't have libnetty_transport_native_epoll_x86_64.so.