Armeria: Violate MAX_CONCURRENT_STREAMS on client side

[ikhoon]

After initializing HTTP2 session, Armeria client respect MAX_CONCURRENT_STREAM. #1206 #1481
However, before fully initializing HTTP2 session, Armeria client defers acquiring connection until previous request of acquiring of connection is complete and multiplex the connection.

While the parallel requests more then MAX_CONCURRENT_STREAMS which is not received yet could be registered to the callback of pendingAuquisition.handle(...).
https://github.com/line/armeria/blob/e21448a95b963cc0e6f349710f7ec1cf572b87af/core/src/main/java/com/linecorp/armeria/client/HttpChannelPool.java#L270-L277

It could cause Unexpected exception: io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint..

12:17:21.476 [armeria-eventloop-nio-4-1] WARN  c.l.a.client.HttpRequestSubscriber - [id: 0x90ccd40e, L:/127.0.0.1:61796 - R:127.0.0.1/127.0.0.1:18080][h2c] Unexpected exception:
io.netty.handler.codec.http2.Http2Exception$StreamException: Maximum active streams violated for this endpoint.
    at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:147)
    at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.checkNewStreamAllowed(DefaultHttp2Connection.java:896)
    at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:748)
    at io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultEndpoint.createStream(DefaultHttp2Connection.java:668)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:176)
    at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:151)
    at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:45)
    at com.linecorp.armeria.internal.Http2ObjectEncoder.doWriteHeaders(Http2ObjectEncoder.java:74)
    at com.linecorp.armeria.internal.HttpObjectEncoder.writeHeaders(HttpObjectEncoder.java:56)
    at com.linecorp.armeria.client.HttpRequestSubscriber.write0(HttpRequestSubscriber.java:298)
    at com.linecorp.armeria.client.HttpRequestSubscriber.writeFirstHeader(HttpRequestSubscriber.java:183)
    at com.linecorp.armeria.client.HttpRequestSubscriber.onSubscribe(HttpRequestSubscriber.java:161)
    at com.linecorp.armeria.common.stream.DefaultStreamMessage.subscribe(DefaultStreamMessage.java:121)
    at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:93)
    at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:84)
    at com.linecorp.armeria.common.HeaderOverridingHttpRequest.subscribe(HeaderOverridingHttpRequest.java:114)
    at com.linecorp.armeria.client.HttpSessionHandler.invoke(HttpSessionHandler.java:161)
    at com.linecorp.armeria.client.HttpClientDelegate.doExecute(HttpClientDelegate.java:234)
    at com.linecorp.armeria.client.HttpClientDelegate.lambda$acquireConnectionAndExecute$2(HttpClientDelegate.java:142)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2144)
    at com.linecorp.armeria.client.HttpChannelPool.notifyConnect(HttpChannelPool.java:422)
    at com.linecorp.armeria.client.HttpChannelPool.lambda$connect$5(HttpChannelPool.java:332)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
    at com.linecorp.armeria.client.HttpSessionHandler.userEventTriggered(HttpSessionHandler.java:278)
....

I think we need to reschedule the pending requests which are registered in the callback. When the number of pending requests is larger than received MAX_CONCURRENT_STREAM, a new connection should be created.

[trustin]

@jrhee17 This might also be a good issue for you. Somewhat easier than #2321.

[jrhee17]

starting work on this 😉

[ikhoon]

starting work on this 😉

Making a test case to reproduce this defect would be a good starting point. 😉

Related test:
https://github.com/line/armeria/blob/8a67c5823ab3cfd1f4a27683409c638091093d3f/core/src/test/java/com/linecorp/armeria/client/HttpClientMaxConcurrentStreamTest.java#L48

[trustin]

Fixed by @jrhee17 via #2374. Thanks a lot! :tada: