Spring-cloud-gateway: Errors in gateway during high load

Created on 19 Dec 2017 · 16Comments · Source: spring-cloud/spring-cloud-gateway

spring-cloud-dependencies: Finchley.M5
spring-boot-starter-parent: 2.0.0.M7

I configured a jmeter test which creates 200 concurrent threads and it hits my gateway. Under this load some request fail with a NullPointerException.

I created two services, the first one using spring cloud gateway and the other one using spring web react. The latter one exposes a /hello GET endpoint and with a delay of 200 ms with the following bean:

    @Bean
    public RouterFunction<ServerResponse> routes() {
        return route(GET("/hello"),
                (req)-> ok()
                        .body(BodyInserters.fromObject("Hello")).delayElement(Duration.ofMillis(200))
        );
    }

The first service has a routing to the second service. After debugging, I noticed what causes the issue is the line below in NettyRoutingFilter, as responseHeaders() returns null:

res.responseHeaders().forEach(entry -> headers.add(entry.getKey(), entry.getValue()));

Full Stacktrace:

java.lang.NullPointerException: null
    at org.springframework.cloud.gateway.filter.NettyRoutingFilter.lambda$filter$4(NettyRoutingFilter.java:105) ~[spring-cloud-gateway-core-2.0.0.M5.jar:2.0.0.M5]
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:177) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:450) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:115) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:450) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:450) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:450) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:139) ~[reactor-core-3.1.2.RELEASE.jar:3.1.2.RELEASE]
    at reactor.ipc.netty.channel.PooledClientContextHandler.fireContextActive(PooledClientContextHandler.java:84) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
    at reactor.ipc.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:176) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
    at reactor.ipc.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:342) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
    at reactor.ipc.netty.channel.ChannelOperations.onHandlerTerminate(ChannelOperations.java:420) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:108) ~[reactor-netty-0.7.2.RELEASE.jar:0.7.2.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoPeek] :
    reactor.core.publisher.Mono.doOnSignal(Mono.java:3622)
    reactor.core.publisher.Mono.doOnNext(Mono.java:1646)
    org.springframework.cloud.gateway.filter.NettyRoutingFilter.filter(NettyRoutingFilter.java:101)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:121)
    org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:40)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:103)
    org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter.filter(RouteToRequestUrlFilter.java:59)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:121)
    org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:40)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:103)
    org.springframework.cloud.gateway.filter.factory.RemoveNonProxyHeadersGatewayFilterFactory.lambda$apply$1(RemoveNonProxyHeadersGatewayFilterFactory.java:79)
    org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:40)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:103)
    org.springframework.cloud.gateway.filter.NettyWriteResponseFilter.filter(NettyWriteResponseFilter.java:52)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:121)
    org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:40)
    org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:103)
    org.springframework.cloud.gateway.handler.FilteringWebHandler.handle(FilteringWebHandler.java:87)
    org.springframework.web.reactive.result.SimpleHandlerAdapter.handle(SimpleHandlerAdapter.java:62)
    org.springframework.web.reactive.DispatcherHandler.invokeHandler(DispatcherHandler.java:168)
    org.springframework.web.reactive.DispatcherHandler.lambda$handle$1(DispatcherHandler.java:160)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
    reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:271)
    reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:803)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:115)
    reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
    reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1092)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:115)
    reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
    reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.tryOnNext(FluxFilterFuseable.java:129)
    reactor.core.publisher.FluxIterable$IterableSubscriptionConditional.fastPath(FluxIterable.java:574)
    reactor.core.publisher.FluxIterable$IterableSubscriptionConditional.request(FluxIterable.java:459)
    reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.request(FluxFilterFuseable.java:170)
    reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:102)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:156)
    reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:103)
    reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
    reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:64)
    reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onSubscribe(FluxFilterFuseable.java:79)
    reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:124)
    reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:61)
    reactor.core.publisher.FluxFilterFuseable.subscribe(FluxFilterFuseable.java:51)
    reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40)
    reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
    reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
    reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
    reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
    reactor.core.publisher.Mono.subscribe(Mono.java:3008)
    reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:418)
    reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:210)
    reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:128)
    reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:61)
    reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121)
    reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40)
    reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44)
    reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
    reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
    reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
    reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
    reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
    reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
    reactor.core.publisher.Mono.subscribe(Mono.java:3008)
    reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167)
    reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
    reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
    reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61)
    reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:383)
    reactor.ipc.netty.http.server.HttpServerOperations.onHandlerStart(HttpServerOperations.java:359)
    io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
    io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
    io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
Error has been observed by the following operator(s):
    |_  Mono.doOnNext(NettyRoutingFilter.java:101)
    |_  Mono.then(NettyRoutingFilter.java:113)
    |_  Mono.then(NettyWriteResponseFilter.java:52)
    |_  Mono.then(SimpleHandlerAdapter.java:63)
    |_  Mono.flatMap(DispatcherHandler.java:160)
    |_  Mono.flatMap(DispatcherHandler.java:161)
    |_  Mono.defer(DefaultWebFilterChain.java:71)
    |_  Mono.doOnSuccess(MetricsWebFilter.java:59)
    |_  Mono.doOnError(MetricsWebFilter.java:60)
    |_  Mono.compose(MetricsWebFilter.java:54)
    |_  Mono.defer(DefaultWebFilterChain.java:71)

And below how the gateway is configured:

server.port: 8080
spring.reactor.stacktrace-mode.enabled: true
spring:
  cloud:   
    gateway:
      routes:
      - id: hello
        uri: http://localhost:8081
        predicates:
        - Path=/hello

Source

fabiormoura

All 16 comments

Hmm, seems like a framework error

spencergibb on 19 Dec 2017

I see the same behavior too @spencergibb , @fabiormoura - this is for the sample app available here - https://github.com/bijukunjummen/boot2-load-demo

bijukunjummen on 20 Dec 2017

👍1

@bijukunjummen @fabiormoura See https://github.com/reactor/reactor-netty/commit/168b2eb94da5867facb2fe02a1c7e3b6c5e94b3a. The null pointer exception is fixed by reactor-netty 0.7.3. But that only masks an real problem

java.io.IOException: Connection closed prematurely
        at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:262) ~[reactor-netty-0.7.3.BUILD-SNAPSHOT.jar:0.7.3.BUILD-SNAPSHOT]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110) ~[reactor-netty-0.7.3.BUILD-SNAPSHOT.jar:0.7.3.BUILD-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.17.Final.jar:4.1.17.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[netty-common-4.1.17.Final.jar:4.1.17.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]

spencergibb on 20 Dec 2017

Any ideas on what we could do here?

scheuchzer on 29 Jan 2018

@scheuchzer is it still a problem? Have you tried with snapshots lately?

spencergibb on 29 Jan 2018

@spencergibb Well, the null pointer is gone but I get quite some connection closed exceptions that you mentioned before. Sometimes it's up to 10 percent of the traffic. I'm not sure where to dig. If the target server is really hanging up or if it's somewhere inside netty or reactor. I wrote a simple retry filter that retries on this specific error but this is not a good solution (and the IOException still get's logged somewhere)

java.io.IOException: Connection closed prematurely
    at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:262) [reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110) [reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.20.Final.jar:4.1.20.Final]

scheuchzer on 30 Jan 2018

java.io.IOException: Connection closed prematurely
    at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:262) [reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110) [reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.20.Final.jar:4.1.20.Final]

scheuchzer on 30 Jan 2018

Hi,
I get this error stably if the http request is greater than 1024.

java.io.IOException: Connection closed prematurely
    at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:262)
    at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420)
    at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377)
    at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
    at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282)
    at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1407)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:925)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
    at java.lang.Thread.run(Thread.java:748)

servetnikav on 5 Feb 2018

@smaldini or @violetagg any thoughts?

spencergibb on 6 Feb 2018

Can you try with Rector Netty 0.7.4.BUILD-SNAPSHOT we have fixes there that might be relevant?

Thanks,
Violeta

violetagg on 6 Feb 2018

After updating the library, the error was gone.
But the gateway does not return the response for http request (method POST) greater than 1024 bytes.

Thanks

servetnikav on 6 Feb 2018

👍1

I can confirm that the errors disappear when using 0.7.4.BUILD-SNAPSHOT

scheuchzer on 8 Feb 2018

👍1

Closing, we can reopen if there is more to discuss.

spencergibb on 31 May 2018

I still have the issue using 0.7.6. looks like race condition issue with the reactor-netty pool resources. Disabling pool in HttpClient resolved this problem, but that's not ideal.
To reproduce this, the server responds with a chunked body (>10KB in each chunk). The client loops and it will hang.