Reactor-netty: Apparent file descriptor leak after upgrading from 0.9.7.RELEASE to 0.9.8.RELEASE

[ginkel]

Yesterday I upgraded one of our services from reactor-netty 0.9.7.RELEASE to 0.9.8.RELEASE as well as Project Reactor from Dysprosium-SR7 to Dysprosium-SR8.

The service is deployed in multiple roles, the one affected by the regression is doing HTTP scraping and upload using WebClient (the ones just exposing REST services via WebFlux seem unaffected).

This morning the instance crashed (killed by Linux kernel OOM killer). Looking at the Micrometer Grafana dashboard shows that the instance accumulated nearly 180.000 file descriptors before crashing. I have since reverted reactor-netty to 0.9.7.RELEASE (but left Project Reactor at Dysprosium-SR8). This eliminates the issues, which hints at some kind of regression in reactor-netty 0.9.8.RELEASE.

The following dashboard panel nicely shows the behavior of the different versions:

Phase 1 (until about 19:00): reactor-netty 0.9.7 and Dysprosium-SR7 (no handles leaking)
Phase 2 (from 19:00 till 10:30): reactor-netty 0.9.8 and Dysprosium-SR8 (handles leaking)
Crash and Restart @ 10:30
Phase 3 (from 10:30 till 11:00): reactor-netty 0.9.8 and Dysprosium-SR8 (handles leaking)
Phase 4 (since 11:00): reactor-netty 0.9.7 and Dysprosium-SR8 (no handles leaking)

At the moment, that's all evidence I currently have. :-/

Your Environment

• Reactor version(s) used: Dysprosium-SR8
• Other relevant libraries versions (eg. netty, ...): netty 4.1.50.Final, Spring Boot 2.3.1.RELEASE
• JVM version (javar -version): OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)
• OS and version (eg uname -a): Linux hydra 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[violetagg]

@ginkel May I ask you to make a test (if possible) with reactor-netty 0.9.8 and Dysprosium-SR8 and Netty v4.1.49.Final?

[ginkel]

Sure, although netty was already at version 4.1.50.Final before the problems manifested:

diff --git a/pom.xml b/pom.xml
index 36a95d1..cb9d3ca 100644
--- a/pom.xml
+++ b/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-parent</artifactId>
-        <version>2.3.0.RELEASE</version>
+        <version>2.3.1.RELEASE</version>
         <relativePath />
     </parent>
     <groupId>de.tgbyte</groupId>
@@ -19,7 +19,7 @@
         <jqassistant.skip>true</jqassistant.skip>

         <!-- Dependencies -->
-        <awssdk.version>2.13.30</awssdk.version>
+        <awssdk.version>2.13.35</awssdk.version>
         <blockhound.version>1.0.3.RELEASE</blockhound.version>
         <caffeine.version>2.8.4</caffeine.version>
         <commons-math3.version>3.6.1</commons-math3.version>
@@ -37,8 +37,8 @@
         <mockito.version>3.3.3</mockito.version>
         <netty.version>4.1.50.Final</netty.version>
         <okhttp3.version>4.7.2</okhttp3.version>
-        <reactor.version>Dysprosium-SR7</reactor.version>
-        <reactor-netty.version>0.9.7.RELEASE</reactor-netty.version>
+        <reactor.version>Dysprosium-SR8</reactor.version>
+        <reactor-netty.version>0.9.8.RELEASE</reactor-netty.version>
         <s3mock.version>2.1.20</s3mock.version>
         <sentry.version>1.7.30</sentry.version>
         <zookeeper.version>3.4.14</zookeeper.version>
@@ -46,7 +46,7 @@
         <!-- Plugins -->
         <git-commit-id-plugin.version>4.0.0</git-commit-id-plugin.version>
         <jacoco-maven-plugin.version>0.8.5</jacoco-maven-plugin.version>
-        <jib-maven-plugin.version>2.3.0</jib-maven-plugin.version>
+        <jib-maven-plugin.version>2.4.0</jib-maven-plugin.version>
         <jqassistant-maven-plugin.version>1.8.0</jqassistant-maven-plugin.version>
         <maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>

Still makes sense to do the test?

[violetagg]

@ginkel No
Then try reactor-netty 0.9.8 and Dysprosium-SR7 to be sure it is Reactor Netty.

[violetagg]

Also is it possible to extract some reproducible example?

[ginkel]

Already done ;-) That's the last segment in the dashboard panel where reactor-netty is at 0.9.7 and Reactor is at Dysprosium-SR8. A slightly newer version of the dashboard better shows this:

The problem does not occur with Reactor Dysprosium-SR8 and reactor-netty 0.9.7, but does occur with Reactor Dysprosium-SR8 and reactor-netty at 0.9.8.

A reproducer may be somewhat tricky (as the overall service is a non-trivial codebase), I can see what I can do...

[violetagg]

Then can you use a profiler on your side to identify the acquisition point?

[ginkel]

There you go: https://github.com/tgbyte/reactor-netty-issue-1152-reproducer

The unit test de.tgbyte.issue1152reproducer.ReproducerTest#test won't complete for a while and you can watch the file descriptors being consumed using lsof -p <pidoftest> | wc -l while it's running. When you downgrade to 0.9.7.RELEASE the issue goes away.

[violetagg]

@ginkel Will you be able to test this fix #1154

[ginkel]

@ginkel Will you be able to test this fix #1154

I'll give it a try later today and will let you know about the result. Is the fix already available in some SNAPSHOT artifact that I could easily reference during the build?

[violetagg]

@ginkel yes 0.9.9.BUILD-SNAPSHOT

[jayzch]

When it will be release?

[violetagg]

@jayzch The release is scheduled for Monday 22.06

[ginkel]

Just tested 0.9.9.BUILD-SNAPSHOT. The file descriptor leak is gone. Thanks for the speedy fix and sorry that it took a while to verify it!

[deepakkapoor23]

I am still seeing this issue even after reverting reactor-netty back to 0.9.7.RELEASE version. What is the latest stable version to use?

[violetagg]

All

Reactor Netty 0.9.9.RELEASE is available

Thanks a lot for testing this fix.