Vertx-web: deadlock in eventloop

[shysnail]

Version

• vert.x core:3.5.1~3.5.3 ,3.6.3, 3.7.0
• vert.x web:

Context

deadlock happends when I use build a gateway with vertx。
process is as follow：
step 1. I open a listener to accept http requests
step 2.I use HttpClient in vertx to request to real server
step 3. forward response to client。

the step2 code section like this：

`// I got clientRequest and clientResponse now

HttpClientRequest upstream = httpClient.requestAbs(uri);

upstream.handler(upstreamResponse->{
Pump pump = Pump.pump(upstreamResponse, clientResponse);

upstreamResponse.end(end -> {
clientResponse.end();
});

pump.start();
})`

few requests entered sequentially， it‘s ok。otherwise， boom！

`Found one Java-level deadlock:

"vert.x-eventloop-thread-3":
waiting to lock monitor 0x00007f9fe2cc1c58 (object 0x00000007995849d8, a io.vertx.core.http.impl.Http1xServerConnection),
which is held by "vert.x-eventloop-thread-6"
"vert.x-eventloop-thread-6":
waiting to lock monitor 0x00007f9fe343e988 (object 0x00000007998ea000, a io.vertx.core.http.impl.Http1xClientConnection),
which is held by "vert.x-eventloop-thread-3"

Java stack information for the threads listed above:

"vert.x-eventloop-thread-3":
at io.vertx.core.http.impl.HttpServerResponseImpl.write(HttpServerResponseImpl.java:614)
- waiting to lock <0x00000007995849d8> (a io.vertx.core.http.impl.Http1xServerConnection)
at io.vertx.core.http.impl.HttpServerResponseImpl.write(HttpServerResponseImpl.java:291)
at io.vertx.core.http.impl.HttpServerResponseImpl.write(HttpServerResponseImpl.java:54)
at io.vertx.core.streams.impl.PumpImpl.lambda$new$1(PumpImpl.java:59)
at io.vertx.core.streams.impl.PumpImpl$$Lambda$270/154416641.handle(Unknown Source)
at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:233)
- locked <0x00000007998ea000> (a io.vertx.core.http.impl.Http1xClientConnection)
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$beginResponse$0(Http1xClientConnection.java:466)
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl$$Lambda$265/340773425.handle(Unknown Source)
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:230)
at io.vertx.core.streams.impl.InboundBuffer.drain(InboundBuffer.java:217)
at io.vertx.core.streams.impl.InboundBuffer.lambda$fetch$0(InboundBuffer.java:270)
at io.vertx.core.streams.impl.InboundBuffer$$Lambda$277/344563157.handle(Unknown Source)
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:320)
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
at io.vertx.core.impl.EventLoopContext$$Lambda$13/1309129055.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)
"vert.x-eventloop-thread-6":
at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:244)
- waiting to lock <0x00000007998ea000> (a io.vertx.core.http.impl.Http1xClientConnection)
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.doFetch(Http1xClientConnection.java:356)
at io.vertx.core.http.impl.HttpClientResponseImpl.fetch(HttpClientResponseImpl.java:187)
at io.vertx.core.http.impl.HttpClientResponseImpl.resume(HttpClientResponseImpl.java:182)
at io.vertx.core.http.impl.HttpClientResponseImpl.resume(HttpClientResponseImpl.java:35)
at io.vertx.core.streams.impl.PumpImpl.lambda$new$0(PumpImpl.java:57)
at io.vertx.core.streams.impl.PumpImpl$$Lambda$269/1508911407.handle(Unknown Source)
at io.vertx.core.http.impl.HttpServerResponseImpl.handleDrained(HttpServerResponseImpl.java:527)
- locked <0x00000007995849d8> (a io.vertx.core.http.impl.Http1xServerConnection)
at io.vertx.core.http.impl.Http1xServerConnection.handleInterestedOpsChanged(Http1xServerConnection.java:415)
- locked <0x00000007995849d8> (a io.vertx.core.http.impl.Http1xServerConnection)
at io.vertx.core.http.impl.HttpServerResponseImpl.lambda$drainHandler$0(HttpServerResponseImpl.java:250)
at io.vertx.core.http.impl.HttpServerResponseImpl$$Lambda$276/1713901376.handle(Unknown Source)
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:320)
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
at io.vertx.core.impl.EventLoopContext$$Lambda$13/1309129055.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.
`

it happends when I use vertx version 3.5.1~3.5.3 and 3.7.0，3.6.3 is ok
I also read other issues and try put the code into runOnContext， but it didn‘t work。

Do you have a reproducer?

• Link to github project/gist

Steps to reproduce

1. ...
2. ...
3. ...
4. ...

Extra

• Anything that can be relevant

[vietj]

this seems related to the fact that the HttpClientResponse buffer handler is called under synchronisation and it should not, I will create an issue and fix it in vertx-core

[vietj]

see https://github.com/eclipse-vertx/vert.x/issues/2916