spring-cloud-gateway 🚀 - Cannot route to https URI: unexpected message type

Thanks.
Please learn how to properly format code and logs.

spencergibb on 11 Mar 2018

@spencergibb Repo is here: https://github.com/samtonyclarke/spring-cloud-gw-issue-228
Should be able to build and reproduce the issue in less than five minutes

samtonyclarke on 14 Mar 2018

I'm seeing the same error "IllegalStateException: unexpected message type: DefaultHttpRequest" on Windows 7 (haven't tried it yet on Fedora yet)

mkreider329 on 2 Apr 2018

Having same issue with similar setup here - stress load lab - on M9.

wa-one56 on 3 Apr 2018

Had the same error with M9 on high load. Latest SNAPSHOT versions seem to fix it.

<dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-gateway-dependencies</artifactId>
                <version>2.0.0.BUILD-SNAPSHOT</version>
                <type>pom</type>
                <scope>import</scope>
</dependency>

scheuchzer on 11 Apr 2018

@scheuchzer
I updated my test repo to use Finchley.BUILD-SNAPSHOT and the unfortunately the same problem occurs. Note, it doesn't take high load. I can recreate the problem with the test repo in about two minutes.

@spencergibb
I image that we can't GA with this bug? I also tried using netty core 3.1.6.RELEASE and reactor-netty 0.7.6.RELEASE but the same problem occurs.

samtonyclarke on 11 Apr 2018

The reactor team is looking at things

spencergibb on 11 Apr 2018

👍3

@spencergibb any update from the reactor team/issue to track on their end?
Running into the same issue intermittently.

cah-andrew-fitzgerald on 23 Apr 2018

👍1

There have been netty/reactor netty changes that should be available in spring boot 2.0.2.BUILD-SNAPSHOT and Finchley.BUILD-SNAPSHOT.

spencergibb on 23 Apr 2018

buildscript {
    ext {
        springBootVersion = '2.0.2.BUILD-SNAPSHOT'
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
    }
}
dependencyManagement {
    imports {
        mavenBom 'org.springframework.cloud:spring-cloud-gateway:2.0.0.BUILD-SNAPSHOT'
    }
}
dependencies {
    compile('org.springframework.cloud:spring-cloud-starter-gateway')
    compile('org.springframework.cloud:spring-cloud-starter-netflix-hystrix')
    compile('org.springframework.boot:spring-boot-starter-actuator')
}

Not seeing issues so far with this (abbreviated) build.gradle

fitzoh on 3 May 2018

👍2

I spoke a bit too soon. Still seeing some intermittently, but not nearly as often

fitzoh on 4 May 2018

The exception is gone when using org.springframework.boot:spring-boot-starter-parent version 2.0.3-BUILD-SNAPSHOT this weekend (05/19/2018). A couple of hour plus runs were successful over the weekend with the project github.com/samtonyclarke/spring-cloud-gw-issue-228

mkreider329 on 21 May 2018

I spoke to soon too. Even with 2.0.3.BUILD-SNAPSHOT, the test still fails. It can happen anywhere from a couple of minutes to a couple of hours. There is a new stacktrace:

java.io.IOException: Connection closed prematurely
        at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:269) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:113) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1028) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

mkreider329 on 30 May 2018

@spencergibb is this potentially a release blocker?

fitzoh on 30 May 2018

👍1

I don't know. It's out of the gateways control. I let the reactor-netty folks know.

spencergibb on 30 May 2018

@mkreider329 What OS are you using?
I tried several runs on MACOS with the latest Spring Boot 2.0.3.BUILD-SNAPSHOT, the test ran for about 7-8 minutes and it stopped

The test stopped

$ java -jar target/test-driver-1.0.0.jar > /dev/null
$

No errors on the gateway side

$ java -jar target/gateway-0.0.1-SNAPSHOT.jar
2018-05-31 22:17:58.414  INFO 198 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@61a52fbd: startup date [Thu May 31 22:17:58 EEST 2018]; root of context hierarchy
...
2018-05-31 22:18:00.999  INFO 198 --- [ctor-http-nio-1] r.ipc.netty.tcp.BlockingNettyContext     : Started HttpServer on /0:0:0:0:0:0:0:0:8080
2018-05-31 22:18:01.000  INFO 198 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port(s): 8080
2018-05-31 22:18:01.003  INFO 198 --- [           main] o.s.s.c.g.issue228.Issue228Application   : Started Issue228Application in 3.139 seconds (JVM running for 3.601)
^C2018-05-31 22:25:19.735  INFO 198 --- [      Thread-14] onfigReactiveWebServerApplicationContext : Closing org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext@6e1567f1: startup date [Thu May 31 22:17:58 EEST 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@61a52fbd
2018-05-31 22:25:19.738  INFO 198 --- [      Thread-14] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2018-05-31 22:25:19.739  INFO 198 --- [      Thread-14] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2018-05-31 22:25:19.739  INFO 198 --- [      Thread-14] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans
2018-05-31 22:25:19.741  INFO 198 --- [      Thread-14] r.ipc.netty.tcp.BlockingNettyContext     : Stopped HttpServer on /0:0:0:0:0:0:0:0:8080
$

violetagg on 31 May 2018

@Fitzoh any feedback?

spencergibb on 31 May 2018

My tests were running on PCF, not sure on version info.
Should be able to give it another shot w/ updated versions either tonight or tomorrow.
@spencergibb would you prefer I test RC2 or snapshots of everything?

cah-andrew-fitzgerald on 31 May 2018

@violetagg
The test as written is to short now. It only runs 200,000 calls in test-driver before the test just ends. To trigger the error, change /test-driver/src/main/java/org/samtonyclarke/test/driver/DriveIt.java
from

Flux.range(1, 200000)

to

Flux.range(1, 10000000)

I'm running on RHEL 7.3 "Linux linux03 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux"

mkreider329 on 1 Jun 2018

For reference, these appear to be the relevant reactor-netty issues:
reactor/reactor-netty#177 (closed)
reactor/reactor-netty#323 (closed)
reactor/reactor-netty#324 (open)
reactor/reactor-netty#361 (open)

fitzoh on 1 Jun 2018

Did you observe any other exceptions than Connection closed prematurely?

violetagg on 4 Jun 2018

Yes, two exceptions were observed in the output:

java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
java.io.IOException: Connection closed prematurely

Forgive the long output below, but it is the gateway output from test a run last week. Gateway output:

2018-05-30 13:38:21.517  INFO 14415 --- [-server-epoll-5] r.ipc.netty.tcp.BlockingNettyContext     : Started HttpServer on /0:0:0:0:0:0:0:0%0:8080
2018-05-30 13:38:21.518  INFO 14415 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port(s): 8080
2018-05-30 13:38:21.521  INFO 14415 --- [           main] o.s.s.c.g.issue228.Issue228Application   : Started Issue228Application in 3.015 seconds (JVM running for 3.426)
2018-05-30 15:42:37.190 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.189 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.334 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.336 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.339 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.339 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.342 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.343 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.346 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.348 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.348 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.349 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.350 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.351 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.353 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.355 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

java.io.IOException: Connection closed prematurely
        at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:269) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:113) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1028) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-05-30 15:42:37.356 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

java.io.IOException: Connection closed prematurely
        at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:269) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:113) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1028) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-05-30 15:42:37.356 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.357 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.358 ERROR 14415 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.361 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.363 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.365 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

javax.net.ssl.SSLException: SSLEngine closed already
       at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-05-30 15:42:37.367 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.369 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.372 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.374 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.377 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.378 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

2018-05-30 15:42:37.380 ERROR 14415 --- [ctor-http-nio-1] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.core.publisher.Mono.subscribe(Mono.java:3080) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:354) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE]
        at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502) ~[reactor-netty-0.7.7.RELEASE.jar!/:0.7.7.RELEASE]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        ... 30 common frames omitted

mkreider329 on 4 Jun 2018

There is a fairly simple test case to reproduce this https://github.com/samtonyclarke/spring-cloud-gw-issue-228

Just update the client DriveIt.java to make more than 200000 calls, which isn't enough to trigger the error anymore (I'd suggest at least 10000000 so it runs for several hours). https://github.com/samtonyclarke/spring-cloud-gw-issue-228/blob/master/test-driver/src/main/java/org/samtonyclarke/test/driver/DriveIt.java

mkreider329 on 5 Jun 2018

any update on this issue?

liujiong1982 on 6 Jun 2018

I think this one should fix the issue https://github.com/reactor/reactor-netty/pull/371
let us review it a bit and then we will commit it to the 0.7.x snapshot

violetagg on 6 Jun 2018

🎉1 👍1

Please try the current snapshot, it is available on the http://repo.spring.io/snapshot

violetagg on 6 Jun 2018

Thank you for the update. I've kicked off a test run using reactor-netty-0.7.8.BUILD-20180606.174842-17.jar

mkreider329 on 6 Jun 2018

👍2

@violetagg I was having a similar experience where after a connection reset error, I would get a "Only one connection receive subscriber allowed" and then I would start getting a "java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest" on future requests, which is the same symptom. This was happening under heavy load. After switching to the latest snapshot, I am no longer seeing it in initial tests.
Although this had nothing to do with ssl, I think the failure mode may be related where a pooled connection has an error condition and then it goes into a bad state.
Btw, this was with Spring Cloud Gateway 2.0.0.RC2 and I have mainly experienced this when testing with requests with a body such as a PUT as opposed to GET requests without a body.

arkin on 6 Jun 2018

@violetagg the test failed with the latest snapshot. This time there was only one exception. Here is the output from the gateway:

2018-06-06 16:56:06.706  INFO 4327 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@33a10788: startup date [Wed Jun 06 16:56:06 EDT 2018]; root of context hierarchy
2018-06-06 16:56:06.989  INFO 4327 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$3cd10333] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::  (v2.0.3.BUILD-SNAPSHOT)

2018-06-06 16:56:07.178  INFO 4327 --- [           main] o.s.s.c.g.issue228.Issue228Application   : No active profile set, falling back to default profiles: default
2018-06-06 16:56:07.191  INFO 4327 --- [           main] onfigReactiveWebServerApplicationContext : Refreshing org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext@56ef9176: startup date [Wed Jun 06 16:56:07 EDT 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@33a10788
2018-06-06 16:56:07.750  INFO 4327 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=855dcd21-224c-399f-bb24-94faa51e126e
2018-06-06 16:56:07.796  INFO 4327 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$3cd10333] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2018-06-06 16:56:08.044  INFO 4327 --- [           main] o.s.w.r.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.reactive.resource.ResourceWebHandler]
2018-06-06 16:56:08.044  INFO 4327 --- [           main] o.s.w.r.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.reactive.resource.ResourceWebHandler]
2018-06-06 16:56:08.915  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [After]
2018-06-06 16:56:08.915  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Before]
2018-06-06 16:56:08.915  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Between]
2018-06-06 16:56:08.915  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Cookie]
2018-06-06 16:56:08.915  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Header]
2018-06-06 16:56:08.915  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Host]
2018-06-06 16:56:08.916  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Method]
2018-06-06 16:56:08.916  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Path]
2018-06-06 16:56:08.916  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Query]
2018-06-06 16:56:08.916  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [ReadBodyPredicateFactory]
2018-06-06 16:56:08.916  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [RemoteAddr]
2018-06-06 16:56:08.916  INFO 4327 --- [           main] o.s.c.g.r.RouteDefinitionRouteLocator    : Loaded RoutePredicateFactory [Weight]
2018-06-06 16:56:08.967  INFO 4327 --- [           main] o.s.w.r.r.m.a.ControllerMethodResolver   : Looking for @ControllerAdvice: org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext@56ef9176: startup date [Wed Jun 06 16:56:07 EDT 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@33a10788
2018-06-06 16:56:09.294  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2018-06-06 16:56:09.307  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2018-06-06 16:56:09.308  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'environmentManager' has been autodetected for JMX exposure
2018-06-06 16:56:09.308  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'refreshScope' has been autodetected for JMX exposure
2018-06-06 16:56:09.311  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2018-06-06 16:56:09.327  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2018-06-06 16:56:09.350  INFO 4327 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=56ef9176,type=ConfigurationPropertiesRebinder]
2018-06-06 16:56:09.496  INFO 4327 --- [-server-epoll-5] r.ipc.netty.tcp.BlockingNettyContext     : Started HttpServer on /0:0:0:0:0:0:0:0%0:8080
2018-06-06 16:56:09.496  INFO 4327 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port(s): 8080
2018-06-06 16:56:09.500  INFO 4327 --- [           main] o.s.s.c.g.issue228.Issue228Application   : Started Issue228Application in 3.346 seconds (JVM running for 3.751)
2018-06-06 17:00:45.061 ERROR 4327 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

java.io.IOException: Connection closed prematurely
        at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:269) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:113) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1028) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-06-06 17:00:45.062 ERROR 4327 --- [ctor-http-nio-4] r.ipc.netty.channel.ContextHandler       : Error cannot be forwarded to user-facing Mono

javax.net.ssl.SSLException: handshake timed out
        at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

2018-06-06 17:00:45.063 ERROR 4327 --- [ctor-http-nio-4] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

java.io.IOException: Connection closed prematurely
        at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:269) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT]
        at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:113) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1028) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-06-06 17:00:45.138 ERROR 4327 --- [ctor-http-nio-4] r.ipc.netty.channel.ContextHandler       : Error cannot be forwarded to user-facing Mono

javax.net.ssl.SSLException: handshake timed out
        at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

mkreider329 on 6 Jun 2018

I get the same "Connection closed prematurely" error and it is accompanied by the following exception:

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer [java]
    at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source) ~[netty-transport-native-unix-common-4.1.25.Final.jar!/:4.1.25.Final] [java]

arkin on 6 Jun 2018

@mkreider329 @arkin do you have any issues on the server side, because Connection closed prematurely means the server closed the connection?

violetagg on 7 Jun 2018

Sprinkled in between the "Connection closed permaturely" errors, there are also the following errors:

javax.net.ssl.SSLException: handshake timed out
        at io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]

Nothing else is really running on that machine, so I doubt an external program starts using resources, or triggers this in some way. And that was one of my early suspicions when running on Windows, but since the same behavior occurs on Linux, I became a little more confident this is an internal problem.

That said, it is strange that the error can happen anywhere from a couple of minutes to a couple of hours into the run. Can you recommend any log settings that may provide more insight? Or do you have any diagnostic patches I can run with to collect more data?

mkreider329 on 7 Jun 2018

@spencergibb Is there a configuration for the ssl handshake timeout? (by default it is 10s)

violetagg on 7 Jun 2018

I've been assuming this problem was originally related to https, but maybe that's not true anymore. I'll change my test to use http and see what happens.

mkreider329 on 7 Jun 2018

👍1

@violetagg In my case, the test is somewhat long running and I can imagine the server sending TCP resets for some connections after a while for things like DOS protection. I think the default setup may not be handling these - not sure reactor or netty would be responsible for handling resets. The other aspect is keep alive, which I don't know how to turn on in the SCG2/Reactor/Netty combination. But in any case, it looks to me like the connection reset messages are not being handled. I could not see any configuration related to this, I am using default settings - hopefully there is a way to configure one of the layers to avoid this.

arkin on 7 Jun 2018

Well, at the risk of convoluting this issue a bit, the run with http lasted 3 hours before throwing a new exception. This is the only time I've seen this exception "ConnectTimeoutException connection timed out", but it seems similar to "SSLException: handshake timed out" (one of the exceptions above).

I plan to continue testing SSL, and will try to gather more system statistics to see if there are system bottlenecks.

Here is the full stacktrace from the gateway output:

2018-06-07 12:54:44.366 ERROR 11959 --- [client-epoll-11] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integrationTest/1.0/app-name]

io.netty.channel.ConnectTimeoutException: connection timed out: localhost/127.0.0.1:8081
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:125) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-06-07 12:54:44.366 ERROR 11959 --- [client-epoll-12] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integra                                                   tionTest/1.0/app-name]

io.netty.channel.ConnectTimeoutException: connection timed out: localhost/127.0.0.1:8081
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:125) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-06-07 12:54:44.366 ERROR 11959 --- [client-epoll-10] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/integra                                                   tionTest/1.0/app-name]

io.netty.channel.ConnectTimeoutException: connection timed out: localhost/127.0.0.1:8081
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:125) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

While the above exception

mkreider329 on 7 Jun 2018

If you can try with this system property -Dreactor.ipc.netty.native=false to see whether there will be any difference if the transport is NIO (currently you are using epoll as I can see from the logs)

violetagg on 7 Jun 2018

@violetagg I tried, but stacktraces were still showing native libraries. I see that this setting used be in LoopResources on master (reactor/netty), but on the 0.7.x branch it is not there anymore. After a little digging in the reactory-netty source 0.7.x branch, I ended up using: -Dreactor.netty.native=false. The problem is still there.
The exception looks slightly different:

java.io.IOException: Connection reset by peer [java]
    at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[na:1.8.0_92] [java]
    at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[na:1.8.0_92] [java]
    at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[na:1.8.0_92] [java]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504) ~[na:1.8.0_92] [java]
    at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:418) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:901) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1396) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:533) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:358) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at reactor.ipc.netty.channel.ChannelOperationsHandler$PublisherSender.onComplete(ChannelOperationsHandler.java:553) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT] [java]
    at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:130) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE] [java]
    at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.emitCachedSignals(ChannelSendOperator.java:289) ~[spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE] [java]
    at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.subscribe(ChannelSendOperator.java:314) ~[spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE] [java]
    at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:52) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE] [java]
    at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE] [java]
    at reactor.ipc.netty.FutureMono$1.subscribe(FutureMono.java:141) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT] [java]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6873) ~[reactor-core-3.1.7.RELEASE.jar!/:3.1.7.RELEASE] [java]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:460) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT] [java]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192) ~[reactor-netty-0.7.8.BUILD-SNAPSHOT.jar!/:0.7.8.BUILD-SNAPSHOT] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext.access$1500(AbstractChannelHandlerContext.java:38) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1129) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) ~[netty-transport-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.25.Final.jar!/:4.1.25.Final] [java]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_92] [java]

Please let me know if this should be handled in a separate issue since the original symptom seems to have been addressed and this is possibly a separate issue.

arkin on 7 Jun 2018

@arkin yep this seems a different issue so lets not trace it here and open another one. Also try to use Wireshark or else to see what happens with the server.

violetagg on 8 Jun 2018

@violetagg @spencergibb
Using netty-reactor 0.7.8 is much better, but it is not 100% fixed. Now I see an exception
javax.net.ssl.SSLException: SSLEngine closed already at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
This exception occurs maybe 10 times in a 5 hour test. You can use this repo to reproduce the problem.

Disabling the netty client pool solves the problems for any recent version of the spring cloud gateway.

samtonyclarke on 13 Jun 2018

👍1

is any one in netty-reactor working on this issue now?

liujiong1982 on 21 Jun 2018

@liujiong1982 yes

violetagg on 21 Jun 2018

Hello,

Can you try the reproducible scenario with this enhancement?

org.samtonyclarke.test.MicroserviceApplication

    @Bean
    public EmbeddedServletContainerFactory servletContainer() {
        TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory();
        tomcat.addConnectorCustomizers(new TomcatConnectorCustomizer() {
            @Override
            public void customize(Connector connector) {
                connector.setProperty("maxKeepAliveRequests", "-1");
                System.out.println("VILY");
            }
        });
        tomcat.addAdditionalTomcatConnectors(initiateHttpConnector());
        return tomcat;
    }

By default Tomcat closes the connection on every 100 requests (maxKeepAliveRequests).
http://tomcat.apache.org/tomcat-8.5-doc/config/http.html#Standard_Implementation

Thanks,
Violeta

violetagg on 21 Jun 2018

@violetagg When I make this change then the test does not fail. However I assume you asked this to help diagnose the problem...it is not a fix, right?

samtonyclarke on 22 Jun 2018

Well I do not see problems on the Gateway side at the moment.
I succeeded to enable the logs + tcpdump and to reproduce the provided scenario (no changes to the scenario except enabling the logs)

The relevant logs are:

2018-06-23 14:30:08.074 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.resources.DefaultPoolResources     : Created new pooled channel [id: 0xcf3533e8], now 1 active connections
2018-06-23 14:30:08.227 DEBUG 4182 --- [ctor-http-nio-7] r.i.n.resources.DefaultPoolResources     : Acquired [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444], now 174 active connections
2018-06-23 14:30:08.228 DEBUG 4182 --- [ctor-http-nio-7] r.i.n.resources.DefaultPoolResources     : Registering close event to pool release: [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:844
4]
2018-06-23 14:30:08.239 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.c.PooledClientContextHandler       : Acquired active channel: [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444]
2018-06-23 14:30:08.251 DEBUG 4182 --- [ctor-http-nio-8] r.ipc.netty.channel.ChannelOperations    : [HttpClient] [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444] handler is being applied:
 HttpClientHandler{startURI=https://localhost:8444/integrationTest/1.0/app-name, method=GET, handler=org.springframework.cloud.gateway.filter.NettyRoutingFilter$$Lambda$642/1768990065@ad81935}
2018-06-23 14:30:08.252 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.channel.ChannelOperationsHandler   : [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444] New sending options
2018-06-23 14:30:08.252 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.channel.ChannelOperationsHandler   : [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444] Writing object DefaultHttpRequest(deco
deResult: success, version: HTTP/1.1)
GET /integrationTest/1.0/app-name HTTP/1.1
user-agent: ReactorNetty/0.7.7.RELEASE
accept: */*
accept-encoding: gzip
Forwarded: proto=http;host="localhost:8060";for="127.0.0.1:58638"
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: http
X-Forwarded-Port: 8060
X-Forwarded-Host: localhost:8060
host: localhost:8444
2018-06-23 14:30:09.789 ERROR 4182 --- [ctor-http-nio-8] r.i.n.http.client.HttpClientOperations   : [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444] onOutboundError 

javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[netty-handler-4.1.25.Final.jar!/:4.1.25.Final]
2018-06-23 14:30:09.790 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.channel.ChannelOperationsHandler   : [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444] Disposing context reactor.ipc.netty.ch
annel.PooledClientContextHandler@1b66c61
2018-06-23 14:30:09.790 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.c.PooledClientContextHandler       : Releasing channel: [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444]
2018-06-23 14:30:09.790 DEBUG 4182 --- [ctor-http-nio-8] r.i.n.resources.DefaultPoolResources     : Released [id: 0xcf3533e8, L:/127.0.0.1:60537 ! R:localhost/127.0.0.1:8444], now 33 active connections

The tcpdump:

Frame 2528205: 44 bytes on wire (352 bits), 44 bytes captured (352 bits) on interface 0 (outbound)
Null/Loopback
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 8444, Dst Port: 60537, Seq: 1, Len: 0
    Source Port: 8444
    Destination Port: 60537
    [Stream index: 1942]
    [TCP Segment Len: 0]
    Sequence number: 1    (relative sequence number)
    Acknowledgment number: 0
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x004 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgment: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Warning/Sequence): Connection reset (RST)]
                [Connection reset (RST)]
                [Severity level: Warning]
                [Group: Sequence]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·········R··]
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 32]
    Checksum: 0xfe1c [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
Frame 2528206: 44 bytes on wire (352 bits), 44 bytes captured (352 bits) on interface 0 (inbound)
Null/Loopback
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 8444, Dst Port: 60537, Seq: 1, Len: 0
    Source Port: 8444
    Destination Port: 60537
    [Stream index: 1942]
    [TCP Segment Len: 0]
    Sequence number: 1    (relative sequence number)
    Acknowledgment number: 0
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x004 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgment: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Warning/Sequence): Connection reset (RST)]
                [Connection reset (RST)]
                [Severity level: Warning]
                [Group: Sequence]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·········R··]
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 32]
    Checksum: 0xfe1c [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0

So the Gateway uses a new channel, not recycled, and the server resets the connection

violetagg on 23 Jun 2018

@violetagg but then that only leaves tomcat. It seems very unlikely to be a problem in tomcat given how long its been around. WDYT?

btw how do you correlate the tcpdump with error?

samtonyclarke on 23 Jun 2018

@samtonyclarke I think that the issue is that when with the default maxKeepAliveRequests the connections are closed too often, when you set it to -1 there is not connection close and thus less connections are opened/closed on OS level.
When I reproduce I see in tcpdump also a lot of TCP Retransmissions.
When you reproduce the issue do you run everything on one machine or on three separate machines. In case you run everything on one machine, do you have a possibility to test it on separate machines?

btw how do you correlate the tcpdump with error?

It is correlated with Src/Dst Port:

in the tcpdump - Src Port: 8444, Dst Port: 60537,
in the logs - [id: 0xcf3533e8, L:/127.0.0.1:60537 - R:localhost/127.0.0.1:8444].

violetagg on 23 Jun 2018

@violetagg Yes, the previous tests were all run on the same machine. I reran the test across three different linux boxes are the failure didn't occur. However that might just be because the timing is has changed and race condition doesn't happen any more.
I am still unsure where you think the error is from: the gateway (netty-reactor), the micro service (tomcat) or the underlying OS (in this case Linux)?

samtonyclarke on 24 Jun 2018

@samtonyclarke After investigating more the possible reasons about so many TCP Retransmissions on a local communication I found these 2 articles [1] [2] . According them in BSD Derivates when TCP backlog is full the server simply does not respond to the payload packages which fully fits with the TCP dump that I recorded. The issue happens when many new connections are established (maxKeepAliveRequests=100) and the system is on a high load, it is not reproducible when connection closing on Tomcat is disabled (maxKeepAliveRequests=-1) as the already established connections are used and new connections are not established during the load. Regarding the error message, it is a little bit misleading because the behavior of the server is also misleading. The server accepts the connection, the client tries to start SSL handshake and during this operation the connection is closed (RST) by the server.
So if TCP backlog is increased, or closing of the connections is disabled the issue will be fixed. Most probably when running on different machines the issue will not be so easy to reproduce because there will be a small delay compare to local execution.
[1] http://veithen.github.io/2014/01/01/how-tcp-backlog-works-in-linux.html
[2] https://bunn.cc/2017/syn-backlog/
screen shot 2018-06-25 at 14 43 22

violetagg on 25 Jun 2018

Hi,

Can we close this issue?

Regards,
Violeta

violetagg on 12 Jul 2018

I'll close it and we can reopen if there are further issues.

spencergibb on 12 Jul 2018

Hi,

I've come across with the same problem, reproducing is trivial. Below is the stack trace.

java.io.IOException: Connection closed prematurely
    at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:269) ~[reactor-netty-0.7.8.RELEASE.jar:0.7.8.RELEASE]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:113) ~[reactor-netty-0.7.8.RELEASE.jar:0.7.8.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) ~[netty-codec-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:282) ~[netty-codec-http-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163) ~[netty-common-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.27.Final.jar:4.1.27.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171]

I'm using raw cloud-gateway and webflux which are both produced by Spring Initializr/no modifications, just added a uuid to represent each user's session.

Load is generated using Gatling, 2000rps. Below is the gatling's scala file.

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.http.request.builder.HttpRequestBuilder.toActionBuilder
import scala.concurrent.duration._
import java.util.UUID

class GatewayLoadTest extends Simulation {
  object UuidFeeder {
    val feeder = Iterator.continually(Map("uuid" -> java.util.UUID.randomUUID.toString()))
  }


  val theHttpProtocolBuilder = http
        .baseURL("http://localhost:8080")

  val theScenarioBuilder = scenario("sample")
  .feed(UuidFeeder.feeder)
  .forever(){
     exec(http("gateway")
      .get("/premature/${uuid}")
      .header("gatling-user-id", "${uuid}")
      .check(

                    status.find.in(200)))
    }

    setUp(
        theScenarioBuilder.inject(constantUsersPerSec(100) during (120 seconds))
    )
    .throttle(
            reachRps(2000) in (1 seconds),
            holdFor(120 seconds)
        )
      .protocols(theHttpProtocolBuilder)


}

After 11k requests, exception occurred. This number varies between tests.

Gateway/reactor-netty's context when exception is occurred:
gateway-premature-exception

Origin is unaware about exception
origin

Wireshark HTTP packages, user's first two request were delegated to origin, 3rd one wasn't forwarded.
premature-wireshark

Last http package's details

Frame 275732: 210 bytes on wire (1680 bits), 210 bytes captured (1680 bits) on interface 0
    Interface id: 0 (lo0)
        Interface name: lo0
    Encapsulation type: NULL/Loopback (15)
    Arrival Time: Aug  2, 2018 16:43:41.578082000 +03
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1533217421.578082000 seconds
    [Time delta from previous captured frame: 0.000009000 seconds]
    [Time delta from previous displayed frame: 0.345598000 seconds]
    [Time since reference or first frame: 233.942944000 seconds]
    Frame Number: 275732
    Frame Length: 210 bytes (1680 bits)
    Capture Length: 210 bytes (1680 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: null:ip:tcp:http]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]
Null/Loopback
    Family: IP (2)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 206
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (6)
    Header checksum: 0x0000 [validation disabled]
    [Header checksum status: Unverified]
    Source: 127.0.0.1
    Destination: 127.0.0.1
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Transmission Control Protocol, Src Port: 57374, Dst Port: 8080, Seq: 309, Ack: 299, Len: 154
    Source Port: 57374
    Destination Port: 8080
    [Stream index: 6157]
    [TCP Segment Len: 154]
    Sequence number: 309    (relative sequence number)
    [Next sequence number: 463    (relative sequence number)]
    Acknowledgment number: 299    (relative ack number)
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window size value: 12750
    [Calculated window size: 408000]
    [Window size scaling factor: 32]
    Checksum: 0xfec2 [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
        TCP Option - No-Operation (NOP)
            Kind: No-Operation (1)
        TCP Option - No-Operation (NOP)
            Kind: No-Operation (1)
        TCP Option - Timestamps: TSval 458370685, TSecr 458370364
            Kind: Time Stamp Option (8)
            Length: 10
            Timestamp value: 458370685
            Timestamp echo reply: 458370364
    [SEQ/ACK analysis]
        [iRTT: 0.000137000 seconds]
        [Bytes in flight: 154]
        [Bytes sent since last PSH flag: 154]
    TCP payload (154 bytes)
Hypertext Transfer Protocol
    GET /premature/43d91f31-5fc7-4c65-81eb-84d9865c2ff1 HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET /premature/43d91f31-5fc7-4c65-81eb-84d9865c2ff1 HTTP/1.1\r\n]
            [GET /premature/43d91f31-5fc7-4c65-81eb-84d9865c2ff1 HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /premature/43d91f31-5fc7-4c65-81eb-84d9865c2ff1
        Request Version: HTTP/1.1
    gatling-user-id: 43d91f31-5fc7-4c65-81eb-84d9865c2ff1\r\n
    Host: localhost:8080\r\n
    Accept: */*\r\n
    \r\n
    [Full request URI: http://localhost:8080/premature/43d91f31-5fc7-4c65-81eb-84d9865c2ff1]
    [HTTP request 3/3]
    [Prev request in frame: 270045]

Tcp stream of the last http package(produces using right-click>follow TCP stream):
wireshark-as-tcp-stream

All apps are run on my personal macbook, same exception also occurs on EC2/Linux per gateway/origin with genuine clients.

Macbook specs:
macOS High Sierra 10.13.6 (17G65)
Processor 2.2 GHz Intel Core i7
Memory 16 GB 1600 MHz DDR3
Tried with 3 different jdks:
jdk1.8.0_131
jdk1.8.0_171
jdk1.8.0_181

bcelenk on 2 Aug 2018

👍1

@bcelenk @violetagg
Hi Bener,
What version of the gateway did you use?

tony-clarke-amdocs on 2 Aug 2018

Hi @tony-clarke-amdocs
I've tried with 2.0.0.RELEASE and 2.0.1.RELEASE.

bcelenk on 2 Aug 2018

@violetagg @bcelenk
If you disable netty client pool does the problem go away?

tony-clarke-amdocs on 2 Aug 2018

@bcelenk more important is the version of reactor-netty

spencergibb on 2 Aug 2018

@spencergibb from stack trace we can see it is 0.7.8.Release

tony-clarke-amdocs on 2 Aug 2018

Yes, reactor netty version is 0.7.8, I didn't test it extensively by disabling pool since it significantly reduces throughput and unfortunately wouldn't be an option.

bcelenk on 2 Aug 2018

@bcelenk thanks for the detailed explanation! @spencergibb @tony-clarke-amdocs I'm experiencing this exact problem as well, is there a way to try out 0.8.0.M1 of the reactor-netty with this? When I try to use it with spring-boot-dependencies of version 2.1.0.M1, I can get the reactor-netty on board with version: 0.8.0.M1 but reactor.ipc.netty.http.server.HttpServerOptions class is missing with that so my code doesn't compile?

mulderbaba on 2 Aug 2018

👍1

Since issue title doesn't represent the "Connection closed prematurely" problem directly, and the issue itself is closed, I'm opening a new issue to reach out more people who might be able to provide a potential solution.

https://github.com/spring-cloud/spring-cloud-gateway/issues/473

bcelenk on 6 Aug 2018

@spencergibb @violetagg
Hi, we are still seeing this problem with Spring Cloud Gateway 2.0.2 which uses reactor-netty 0.7.10.RELEASE. In order to reproduce the problem you need to run for a long time. In an 8 hour run it occurs one time. The latest stack trace looks like:

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1
                at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:293) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:476) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:194) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:200) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at reactor.core.publisher.Mono.subscribe(Mono.java:3088) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.Mono.subscribe(Mono.java:3088) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:369) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:381) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:505) ~[reactor-netty-0.7.10.RELEASE.jar!/:0.7.10.RELEASE]
                at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1
                at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
                ... 30 common frames omitted

tony-clarke-amdocs on 28 Oct 2018

👍2

@tony-clarke-amdocs Any chance to test with Reactor Netty 0.7.11.RELEASE?

violetagg on 28 Oct 2018

@violetagg
No difference. Here is the new stack trace:

Failed to handle request [GET https://apigw-service-kellianp.ilocpfnd01.corp.amdocs.com/security/authorizationRoleA]
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1
                at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:293) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:476) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:194) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:200) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.core.publisher.Mono.subscribe(Mono.java:3088) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.Mono.subscribe(Mono.java:3088) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:395) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:381) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:536) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1
                at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
                ... 30 common frames omitted
2018-10-30 06:27:19.800 ERROR [amd-apigw,,,] 1 --- [tor-http-nio-15] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET https://apigw-service-kellianp.ilocpfnd01.corp.amdocs.com/integrationTest/1.0/get-headers]
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1
                at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:293) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:476) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:194) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:200) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.core.publisher.Mono.subscribe(Mono.java:3088) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.core.publisher.Mono.subscribe(Mono.java:3088) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:395) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.1.10.RELEASE.jar!/:3.1.10.RELEASE]
                at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:381) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:536) ~[reactor-netty-0.7.11.RELEASE.jar!/:0.7.11.RELEASE]
                at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) ~[netty-transport-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-common-4.1.29.Final.jar!/:4.1.29.Final]
                at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_181]
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1
                at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.29.Final.jar!/:4.1.29.Final]
                at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[netty-codec-4.1.29.Final.jar!/:4.1.29.Final]
                ... 30 common frames omitted

tony-clarke-amdocs on 30 Oct 2018

@tony-clarke-amdocs Any chance to tell us the scenario how to reproduce this or to provide some example?

violetagg on 30 Oct 2018

At a high level, take spring cloud Gateway with out of the box configuration. Then publish some API's to the gateway with https as the target URL. Then using a testing tool like Gailting send a many concurrent requests (we used 150) and run the test for 8 hours. Its difficult to reproduce in a short amount of time. We usually need about 6 to 8 hours before the first error occurs.

tony-clarke-amdocs on 30 Oct 2018

@tony-clarke-amdocs Is your problem solved? I encountered the same problem in stress testing.

gzlicanyi on 13 Feb 2019

👍2

Don't see that problem any more with latest netty (.7.14), but still seeing this one: https://github.com/spring-cloud/spring-cloud-gateway/issues/473

tony-clarke-amdocs on 13 Feb 2019

Using:

```

org.springframework.boot
spring-boot-starter-parent
2.1.6.RELEASE

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>Greenwich.SR2</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<properties>
    <spring-cloud.version>Greenwich.SR2</spring-cloud.version>
    <java.version>1.8</java.version>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-reactor-netty</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-contract-stub-runner</artifactId>
        <exclusions>
            <exclusion>
                <artifactId>spring-boot-starter-web</artifactId>
                <groupId>org.springframework.boot</groupId>
            </exclusion>
        </exclusions>
    </dependency>

    <!-- Hystrix and Cloud-Gateway and Eureka -->
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-gateway</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>

    <!-- JWT -->
    <dependency>
        <groupId>io.jsonwebtoken</groupId>
        <artifactId>jjwt</artifactId>
        <version>0.7.0</version>
    </dependency>

    <!-- Apache Httpclient -->
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.3</version>
    </dependency>

    <!-- Config service discovery -->
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-config-client</artifactId>
    </dependency>

    <!-- Actuator service like /health-->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
        <groupId>org.hibernate.validator</groupId>
        <artifactId>hibernate-validator</artifactId>
        <version>6.0.17.Final</version>
    </dependency>

</dependencies>

..mapped in bootstrap.yml:

routes:
- id: backend
uri: lb://backend
predicates:
- Path=/backend/*
filters:
- RewritePath=/backend/(?.), /${path}

I get this:

2019-07-18 19:42:21.578 ERROR 16048 --- [ctor-http-nio-2] a.w.r.e.AbstractErrorWebExceptionHandler : [f85fcd0b] 500 Server Error for HTTP GET "/backend/public/time"

java.util.regex.PatternSyntaxException: Unknown inline modifier near index 11
/backend/(?.*)
^
at java.util.regex.Pattern.error(Pattern.java:1955) ~[na:1.8.0_131]
at java.util.regex.Pattern.group0(Pattern.java:2894) ~[na:1.8.0_131]
at java.util.regex.Pattern.sequence(Pattern.java:2051) ~[na:1.8.0_131]
at java.util.regex.Pattern.expr(Pattern.java:1996) ~[na:1.8.0_131]
at java.util.regex.Pattern.compile(Pattern.java:1696) ~[na:1.8.0_131]
at java.util.regex.Pattern.(Pattern.java:1351) ~[na:1.8.0_131]
at java.util.regex.Pattern.compile(Pattern.java:1028) ~[na:1.8.0_131]
at java.lang.String.replaceAll(String.java:2223) ~[na:1.8.0_131]
at org.springframework.cloud.gateway.filter.factory.RewritePathGatewayFilterFactory.lambda$apply$0(RewritePathGatewayFilterFactory.java:60) ~[spring-cloud-gateway-core-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:44) ~[spring-cloud-gateway-core-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:118) ~[spring-cloud-gateway-core-2.1.2.RELEASE.jar:2.1.2.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeek.subscribe(MonoPeek.java:71) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3848) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3848) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:275) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:849) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:275) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:849) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onNext(MonoFilterWhen.java:140) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2071) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onSubscribe(MonoFilterWhen.java:103) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFilterWhen.subscribe(MonoFilterWhen.java:56) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeek.subscribe(MonoPeek.java:71) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3848) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onNext(FluxDematerialize.java:114) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onNext(FluxDematerialize.java:42) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drainAsync(FluxFlattenIterable.java:395) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drain(FluxFlattenIterable.java:638) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.onNext(FluxFlattenIterable.java:242) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:179) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1498) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.DrainUtils.postCompleteDrain(DrainUtils.java:131) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.DrainUtils.postComplete(DrainUtils.java:186) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMaterialize$MaterializeSubscriber.onComplete(FluxMaterialize.java:134) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drainAsync(FluxFlattenIterable.java:325) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.drain(FluxFlattenIterable.java:638) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber.onComplete(FluxFlattenIterable.java:259) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:144) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1499) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:794) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:560) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:540) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:426) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:265) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:201) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:335) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap.subscribe(FluxFlatMap.java:97) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMaterialize.subscribe(FluxMaterialize.java:40) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:40) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlattenIterable.subscribe(MonoFlattenIterable.java:101) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxDematerialize.subscribe(FluxDematerialize.java:39) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3848) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:212) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:139) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:63) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxConcatMap.subscribe(FluxConcatMap.java:121) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoNext.subscribe(MonoNext.java:40) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoSwitchIfEmpty.subscribe(MonoSwitchIfEmpty.java:44) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2071) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3848) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:60) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3848) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:70) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoPeekTerminal.subscribe(MonoPeekTerminal.java:61) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.netty.http.server.HttpServerHandle.onStateChange(HttpServerHandle.java:64) ~[reactor-netty-0.8.9.RELEASE.jar:0.8.9.RELEASE]
at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:226) ~[reactor-netty-0.8.9.RELEASE.jar:0.8.9.RELEASE]
at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:442) ~[reactor-netty-0.8.9.RELEASE.jar:0.8.9.RELEASE]
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:91) ~[reactor-netty-0.8.9.RELEASE.jar:0.8.9.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:161) ~[reactor-netty-0.8.9.RELEASE.jar:0.8.9.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
```

gewure on 18 Jul 2019

Spring-cloud-gateway: Cannot route to https URI: unexpected message type

Most helpful comment

All 68 comments

Related issues