Reactor-core: Mono#block() and Mono#blockOptional() may occasionally produce deadlock.

[be-hase]

Expected behavior

_Omit_

Actual behavior

Mono#block() and Mono#blockOptional() may occasionally produce deadlocks.
(About twice in the last 4 months.)

stack traces:

case 1:

"XXX_THREAD_NAME_XXX" #231 prio=5 os_prio=0 tid=0x00007f217c0103a0 nid=0x19123 waiting on condition [0x00007f210adb4000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000640294e38> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    at reactor.core.publisher.BlockingOptionalMonoSubscriber.blockingGet(BlockingOptionalMonoSubscriber.java:118)
    at reactor.core.publisher.Mono.blockOptional(Mono.java:1545)
    ....
    .... secret
    ....
    at at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

case 2:

"XXX_THREAD_NAME_XXX" #201 prio=5 os_prio=0 tid=0x00007f0c7c061630 nid=0x19c3c waiting on condition [0x00007f0c715d4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000009a887358> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
        at reactor.core.publisher.Mono.block(Mono.java:1475)
        ....
        .... secret
        ....
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Steps to reproduce

Sorry. I'm writing a minimal code to reproduce, but I do not reproduce.

Reactor Core version

3.2.5.RELEASE

JVM version (e.g. java -version)

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

[be-hase]

In addition, it occurred when I was using WebClient(spring webflux).

[bsideup]

Hi @be-hase,

BlockingMonoSubscriber is just a CountDownLatch counting down on onNext, onError or onComplete.

Since it is not reproducible with a minimal code, I would suggest trying https://github.com/reactor/BlockHound, perhaps you block parallel threads somewhere and it never completes because the delayed task never gets triggered

[be-hase]

Thanks quick reply.

BlockingMonoSubscriber is just a CountDownLatch counting down on onNext, onError or onComplete.

Yes, I'm now looking reactor's code. It is right.

I did not know BlockHound. I will look at it. thanks.

[bsideup]

See this blog post for the explanation of the deadlock scenario:
https://spring.io/blog/2019/03/28/reactor-debugging-experience#blockhound-a-new-kid-on-the-block

[be-hase]

Thank you for sharing.
I'm not doing blocking with reactor's non blocking thread ...
I will check it a bit more.

[be-hase]

It looks that the Reactor Netty's DNS-lookup blocks thread.
Ref: https://github.com/reactor/reactor-netty/issues/569

[be-hase]

I reproduced in my environment.

Prepare an environment to intentionally delay DNS lookup. I used this tool.
https://github.com/brennentsmith/slodns
Use slodns for a specific domain using /etc/resolve

And I reproduced it when I ran such a code.
(It depends on the situation of DNS cache.)
https://gist.github.com/be-hase/9bfdf13432f0b9000260e66ec4ac0cac

This issue was resolved using Netty's Async DNS Resolver. (See gist code)

Should I put a Github-Issue on reactor-netty ?

[bsideup]

Nice!

Should I put a Github-Issue on reactor-netty ?

Yes, please.

[be-hase]

I have created issue.
https://github.com/reactor/reactor-netty/issues/710