Reactor-netty: Global resource cleanup slow on Mac OS

[sbrannen]

Overview

When using global HTTP resources for event loops and pooling on Mac OS, cleanup of those resources takes considerably longer on Mac OS than it does on Linux.

For example, after this commit in the Spring Framework, the WebClientDataBufferAllocatingTests test class began taking approximately 120 seconds instead of 2-3 seconds to complete.

After disabling the use of global resources in https://github.com/spring-projects/spring-framework/commit/f0e160fc67d276d22b2908b6421351feeba7f492#diff-cb71838b5a4c0062d77075023316ea9c, the WebClientDataBufferAllocatingTests test class once again completes in 2-3 seconds.

Based on a tip from a colleague that including a dependency on io.netty:netty-transport-native-kqueue:4.1.39.Final:osx-x86_64 might help, I tried that in the spring-webflux project, but that does not appear to make a difference.

Expected Behavior

The time taken for blocking global resource cleanup on Mac OS is comparable to cleanup on Linux.

Actual Behavior

The time taken for blocking global resource cleanup on Mac OS is considerably slower than on Linux.

Steps to Reproduce

Execute the WebClientDataBufferAllocatingTests test class with this.factory.setUseGlobalResources(false); commented out in the setUp() method.

Your Environment

• Reactor version: Dysprosium-BUILD-SNAPSHOT
• Netty version: 4.1.39.Final
• JVM version: Oracle JDK 1.8.0 update 221
• OS and version: Mac OS / 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64

[violetagg]

@sbrannen There is a problem with the test

When you initialise the WebClient, actually you still don't know the bufferFactory, so you do not create a WebClient with custom ConnectionProvider and LoopResources, but with the global ones. At the end, you dispose ConnectionProvider and LoopResources that have never been used and because of that their disposal finishes immediately.

Here is the code, you enter in the else branch and not the if branch.
https://github.com/spring-projects/spring-framework/blob/f0e160fc67d276d22b2908b6421351feeba7f492/spring-webflux/src/test/java/org/springframework/web/reactive/function/client/WebClientDataBufferAllocatingTests.java#L76-L85

When we dispose the event loop group we invoke
https://netty.io/4.1/api/io/netty/util/concurrent/EventExecutorGroup.html#shutdownGracefully--
This default implementation is with 2s quiet period and 15s timeout

[violetagg]

The log below is an execution on Ubuntu - again 2s quiet period

^A^B^Cq21:47:22.105 [Test worker] DEBUG o.s.w.r.f.c.ExchangeFunctions - [4288649a] HTTP GET http://localhost:42291$
^@^B^CFSep 23, 2019 9:47:22 PM okhttp3.mockwebserver.MockWebServer$3 execute
^@^B^C:INFO: MockWebServer[42291] starting to accept connections
^A^B^Cj21:47:22.142 [reactor-http-epoll-1] DEBUG o.s.w.r.f.c.ExchangeFunctions - [4288649a] Response 201 CREATED
^@^B^CPSep 23, 2019 9:47:22 PM okhttp3.mockwebserver.MockWebServer$4 processOneRequest
^@^B^C_INFO: MockWebServer[42291] received request: GET /json HTTP/1.1 and responded: HTTP/1.1 201 OK
^A^B^C\21:47:22.155 [Test worker] INFO  r.M.D.2 - onSubscribe([Fuseable] MonoWhen.WhenCoordinator)
^A^B^C>21:47:22.158 [Test worker] INFO  r.M.D.2 - request(unbounded)
^A^B^CD21:47:24.366 [globalEventExecutor-1-2] INFO  r.M.D.2 - onComplete()

[sbrannen]

@sbrannen There is a problem with the test

Oops. That's actually my fault: I missed that the bufferFactory was null at that point while migrating from JUnit 4 to JUnit Jupiter.

Thanks for finding that! I've fixed that now.

When we dispose the event loop group we invoke
netty.io/4.1/api/io/netty/util/concurrent/EventExecutorGroup.html#shutdownGracefully--
This default implementation is with 2s quiet period and 15s timeout

I've moved our "destroy" invocation to an @AfterAll method so that we are only hit with the 2s penalty once for the entire test class instead of for each test method.

This is acceptable and much better than the 122 seconds we were seeing.

However, is it possible to override the default quiet period and timeout?

At a glance, it looks like io.netty.util.concurrent.AbstractEventExecutor.DEFAULT_SHUTDOWN_QUIET_PERIOD is always used.

[violetagg]

@sbrannen we can expose an API and then use https://netty.io/4.1/api/io/netty/util/concurrent/EventExecutorGroup.html#shutdownGracefully-long-long-java.util.concurrent.TimeUnit-
with the provided quiet period and timeout

[sbrannen]

Closing since the bug in Spring's test suite has been fixed in https://github.com/spring-projects/spring-framework/commit/38052e77b710a0fa373483bc6e55d68c26e6ec38.

[sbrannen]

@sbrannen we can expose an API and then use https://netty.io/4.1/api/io/netty/util/concurrent/EventExecutorGroup.html#shutdownGracefully-long-long-java.util.concurrent.TimeUnit-
with the provided quiet period and timeout

I think it would be nice to make that configurable. Feel free to reopen this issue if you decide to go that route.

Thanks again for your support! 👍

[violetagg]

Reopen - we may expose an API with quiet period and timeout

[bilak]

I'd like to have this feature in 0.9.x is it possible to provide it? We discussed it on gitter @violetagg .
Thanks

[sbrannen]

Thanks for adding this support, @violetagg!

Looking forward to updating our tests once we're on reactor-netty 0.9.3.

Do you know which reactor-bom release that will go into?

[violetagg]

@sbrannen the planning is Dysprosium-SR3 for next Monday 13th of January