Reactor-netty: Sometimes deadlock is occurred due to blocking DNS resolver

Created on 11 Apr 2019 · 6Comments · Source: reactor/reactor-netty

Expected behavior

Omit

Actual behavior

In reactor-netty, Mono#block() and Mono#blockOptional() may occasionally produce deadlock.
(About twice in the last 4 months.)

stack traces:

"XXX_THREAD_NAME_XXX" #231 prio=5 os_prio=0 tid=0x00007f217c0103a0 nid=0x19123 waiting on condition [0x00007f210adb4000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000640294e38> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    at reactor.core.publisher.BlockingOptionalMonoSubscriber.blockingGet(BlockingOptionalMonoSubscriber.java:118)
    at reactor.core.publisher.Mono.blockOptional(Mono.java:1545)
    ....
    .... secret
    ....
    at at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

"XXX_THREAD_NAME_XXX" #201 prio=5 os_prio=0 tid=0x00007f0c7c061630 nid=0x19c3c waiting on condition [0x00007f0c715d4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000009a887358> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
        at reactor.core.publisher.Mono.block(Mono.java:1475)
        ....
        .... secret
        ....
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Steps to reproduce

Prepare an environment to intentionally delay DNS lookup. I used this tool.
https://github.com/brennentsmith/slodns
Use slodns for a specific domain using /etc/resolver

And I reproduced deadlock when I run such a code. (Maybe, it depends on the situation of DNS cache.)
https://gist.github.com/be-hase/9bfdf13432f0b9000260e66ec4ac0cac

This issue was resolved using netty's async DNS Resolver. (See gist code)
https://netty.io/4.1/api/io/netty/resolver/dns/DnsNameResolver.html

Reactor Netty version

0.8.4.RELEASE

JVM version (e.g. `java -version`)

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

OS version (e.g. `uname -a`)

My Local Mac:

Darwin XXXX 16.7.0 Darwin Kernel Version 16.7.0: Sun Oct 28 22:30:19 PDT 2018; root:xnu-3789.73.27~1/RELEASE_X86_64 x86_64

Production Server:

Linux XXXX 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

typbug

Source

be-hase

👍5

All 6 comments

Is there any workaround here?
I'm getting this error for each request.

I'll add the environment description soon..

newmen on 29 Jan 2020

you can provide your own DNS resolver as the author wrote

This issue was resolved using netty's async DNS Resolver. (See gist code)
https://netty.io/4.1/api/io/netty/resolver/dns/DnsNameResolver.html

HttpClient.tcpConfiguration(tcpClient -> tcpClient.resolver(...))

https://projectreactor.io/docs/netty/release/api/reactor/netty/tcp/TcpClient.html#resolver-io.netty.resolver.AddressResolverGroup-

violetagg on 29 Jan 2020

❤1

So, in my case, the problem was not about DNS resolver.
My problem was due to a conflict in dependencies and silent NoClassDefFoundError.
I'm sorry for the confusion. It because I had the same stacktrace as the author mentioned.

newmen on 31 Jan 2020

We hit this problem during a DDoS attack on one of our systems, and have been able to reproduce it fairly reliably in load tests. Having this problem solved by default, without having to configure a custom resolver would be great! Looking forward to the fix.

philsttr on 2 Mar 2020

👍1

We have hit probably the same issue in our scale testing environment.
I didn't look deep but the symptom seems same.
We have migrated our remote call service from using RestTemplate to WebClient with block() calls, in order to keep the API same.
I am now trying to set async dns resolver, but it would be nice if it has already done by default in reactor-netty level. Especially, it seems reactor-netty already has auto detection mechanism of epoll, kqueue, and nio.