Reactor-netty: Web client connection pool may have connections with invalid HttpObjectEncoder state
While testing WebClient acting as a proxy under load, we reproduce the issue when connection pool holds some connections with invalid state HttpObjectEncoder.
When client gets such connection from a pool and uses it to make request to a destination server, it immediately fails because DefaultHttpRequest cannot be encoded by HttpObjectEncoder due to the fact that encoders state is set to ST_CONTENT_NON_CHUNK (or any other), while it expects it to be ST_INIT. A kind of broken state machine inside the encoder.
The issue is mainly observed when the client proxies heavy upstream data (uploads content to a destination server).
Disabling connection pool fixes the thing.
Observed exception:
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
...
alexmaddriver
All 47 comments
I can confirm this issue.. hard to recreate, but often happens when the WebClient receives a 500 status from the server. @alexmaddriver How did you disable the connection pool?
lnxmad
on 5 Oct 2017
Hi,
Is it possible to test with the latest available Netty - netty-4.1.16.Final
Thanks,
Violeta
violetagg
on 5 Oct 2017
Please ignore it I saw in the stack trace that you are using it already.
violetagg
on 5 Oct 2017
@alexmaddriver could you show some snippet that's representative of how the WebClient is used in this case? In particular I presume that you're using one of the bodyToXxx methods to consume the response content? And the version for the spring-webflux module is 5.0.0.RELEASE?
rstoyanchev
on 6 Oct 2017
There might be a timing issue with the cleanup action that gets subscribed by reactor.ipc.netty.ReactorNetty#registerForClose, called from reactor.ipc.netty.ReactorNetty#addHandlerAfterReactorCodecs.
Perhaps what happens is that the action to remove the handler gets called after the new request flow has already started to use the existing connection. When the expected handler is missing from the Netty pipeline one would get an exception that was reported.
When I was working on #176 , I noticed a similar issue when I first had an invalid fix for the memory leak I had found. Coincidentally that fix might also fix this issue since it would remove the possibility that the cleanup action would get signalled out-of-order. The reason why this could help is that the onClose publisher used for the onInactive processor in reactor.ipc.netty.channel.ChannelOperations now subscribes to a processor in reactor.ipc.netty.channel.ServerContextHandler that is disposed an and unsubscribed from the upstream closeFuture before the channel gets released back to the pool. The reason I did so was to fix a memory leak that was caused by active subscribers to Channel's closeFuture being left around hanging and leaking memory. The same change might have other benefits.
Could you @alexmaddriver please try out #176 ? It might not be the most optimal fix for this issue or even fix this issue, but it would be interesting to see if it changes anything. :)
lhotari
on 6 Oct 2017
Hello,
We use the latest spring-webflux 5.0.0.RELEASE.
@lnxmad: We disable connection pool when creating WebClient as follows:
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(options -> options.disablePool()))
.build();
@rstoyanchev: we do not use bodyToXXX, instead directly "targetResponse.body(toDataBuffers())", because of a recent bug with bodyToXXX methods which used to close connections as a side affect. As it was fixed in 0.7.0.RELEASE, we can now certainly switch to bodyToFlux.
The code snippet:
protected Mono
return client
.post()
.uri(destinationUriGenerator.apply(request.getURI()))
.headers(headers -> filterRequestHeaders(request, headers))
.headers(this::setUserAgent)
.body(request.getBody(), DataBuffer.class)
.exchange()
.map(toResponseEntity());
}
private Function
return targetResponse -> new ResponseEntity<>(
targetResponse.body(toDataBuffers()),
targetResponse.statusCode()
);
}
alexmaddriver
on 6 Oct 2017
@lhotari, thank you for explanation. Sure, we can give it a try and will come back with results.
In fact, we put quite some effort to switch form VertX to web flux client to unify our product stack. And now very close to RTM version and disabling connection pool workaround is not good. Because of that willing to put our efforts to contribute in fixing the problem, or at least fully understand when the issue happens :)
alexmaddriver
on 6 Oct 2017
@alexmaddriver with regards to bodyToXxx and the side effect of closing, indeed that was fixed in 5.0 GA release. The close() method was removed in favor using bodyToMono(Void.class) if not interested in the response content along with this improvement.
rstoyanchev
on 6 Oct 2017
Hi,
As we have several fixes recently that might be relevant, is it possible to test with the latest 0.7.1.BUILD-SNAPSHOT version.
Thanks,
Violeta
violetagg
on 19 Oct 2017
Just tried the latest build and the problem is still there. It is easily reproduced after a few HTTP errors, e.g. 404. After that MessageToMessageEncoder gets into invalid state.
alexmaddriver
on 19 Oct 2017
We use reactor-netty + webflux client based proxy server between clients and a content server.
Also, we wrote client emulating tool able to generate any load using both HTTP and HTTPS connectors, with any content size, including huge packages uploaded / downloaded in parallel. Such emulator is very handy to discover performance, concurrency and memory leak type of issues. If you want our QEs can share more details for you to be able to setup such testing in-house.
alexmaddriver
on 19 Oct 2017
@alexmaddriver I tried to use the snippet above to produce some reproducible example but I'm still not able to do that. If you have something can you share with us. And yes we are interested in the tool that you are developing if you can share more details it will be great.
violetagg
on 19 Oct 2017
@violetagg I have a test based on reactor.ipc.netty.http.HttpTests#httpErrorWithRequestsFromClients which reproduces some type of problem with http client when connection pool is enabled. It is related to errors happening on the server side. It's in this gist: https://gist.github.com/lhotari/3e1c0cb7ae671c8d1ef7050ebbe08e9c . It might or not be related to this issue. Could you take a look at it?
lhotari
on 19 Oct 2017
@lhotari this is an API issue we aim to fix in 0.8 by reworking in full the way you consume a response. Right now nothing tells the flatMapMany code that extract your status code that it should also consume the body (and call receive().subscribe()). So we never fully consume it while you already have returned a code.
In 0.8 we aim to provide a receive(resp -> Mono.just(resp.code()) that would allow us to control the returned Publisher and onComplete, cancel or ignore the underlying connection.
smaldini
on 19 Oct 2017
We've put quite some efforts trying to understand what is going wrong. We can easily reproduce this on our machines however this seem to be a raise conditions thus still unable to write 100% reproducible case.
Just wanted to summarize our findings and maybe you can give some hints where the bug could hide.
The problem with HttpObjectEncoder occurs because it does not receive LastHttpContent message. Last content message should be generated inside HttpClientOperations.onOutboundComplete() method which is called by ChannelOperations.onComplete(). However sometimes onCompete is not called, and after this the encoder is broken.
According to our tests the highest probability when onCompete() for client requests is not called happens in the following conditions:
- HttpClient receives HTTP error response, 404 in our case. Most likely the error code value might be any. However, with 200 code everything seem to work fine.
- HttpClient uses HTTPS to connect a server. With HTTP it seems to work well, or at least the probability to fail is much lower. The only explanation coming to our minds is that HTTPS uses chuncked encoding, while HTTP sets content length and thus someone tracking end of package upload operation properly calls onComplete.
- One more thing, the uploaded content in HTTP request should be greater than some value. E.g. uploading 200 bytes file works fine, while 2MBs file reproduces the case almost immediately.
@smaldini Do you have any idea why ChannelOperations.onComplete() might not be called? According to HttpClientOperations.checkResponseCode(), client errors are ignored, so what might difference between handling 404 vs 200 and how it might affect the flow?
alexmaddriver
on 23 Oct 2017
@alexmaddriver we're closing in another couple issues for 0.7.2 but we need to find a way to reproduce this one.
To explain the intent on 404 in 0.7 : by default we try to just consume the data in full in order to receive a last http packet and release the connection while we are sending back to you an HttpClientException. In 0.8 this is gone because that leads to users missing body data without noticing, but since it's an API change we had to defer this to the next 0.x.
In 0.7.1 we tried fixing this problem with a client by subscribing, where in 0.7.0 there was a similar issue to the one you just mentioned (hence why we insisted on the version). We were interrupting but not fully consuming the body allowing the next request to acquire a previously non finished request.
Another thing i'm thinking about is similar to what you posted in #196, we might not just read enough and the last packet is not being received. What I don't get tho is if it wasn't the case you wouldn't acquire this connection in a next request because its still not marked as released until complete.
smaldini
on 26 Oct 2017
With the latest 0.7.2.BUILD-SNAPSHOT I can't reproduce with 1000 req (max 256 parallel). What kind of sample size you run for and how is 404 response formed (has it a body ? what headers does it present). I will reuse that to produce a smoke test. Maybe what kind of remote server ? Tomcat ? something public we can hit ?
@Test
public void httpStatusCode404IsHandledByTheClient() {
NettyContext server =
HttpServer.create(0)
.newRouter(r -> r.post("/test", (req, res) -> res.status(404)
.send(req.receive().retain().log("server-received"))))
.block(Duration.ofSeconds(30));
HttpClient client = HttpClient.create("localhost", server.address().getPort());
Mono<String> content = client.post("/test", req ->
req.addHeader("Content-Type", "text/plain")
.failOnClientError(false)
.sendString(Flux.just("Hello").log("client-send"))
)
.flatMap(res -> res.receive().asString().log("client-received").last());
StepVerifier.create(Flux.range(1, 1000).flatMap(d -> content).count())
.expectNext(1000L)
.expectComplete()
.verify(Duration.ofSeconds(30));
server.dispose();
}
Hello, i will try to help to reproduce the issue on Wednesday when back from a business trip.
alexmaddriver
on 30 Oct 2017
Hello, apologize for the late response. Just tested with 0.7.2 and unfortunately the issue is still reproduced with out product.
I am attaching reactor-netty logs. Inside are two folders - good (everything fine) and bad (the situation with an invalid encoder). In both cases we have small (24 bytes) and big (~48KB) files uploaded to a server, the only difference is the order in which files are sent. When the small file goes first - everything is fine and any subsequent number of bigger files can be uploaded successfully. However, in case the first uploaded file is the big one - it breaks encoder. All next operations using connection from the pool fail (409 error in the logs).
Involved entities are the follows:
[curl.exe to upload files] <- HTTP -> [proxy] <- HTTPS -> [content server]
- Content server always returns 404 not found HTTP code.
- The issue is reproduced only with HTTPS connection between proxy and content server. If switching to HTTP, it magically starts to work.
- The issue is 100% stable to reproduce.
The only difference inside the logs which we spotted is the time when 404 error is received by HttpClient. In the bad case, it is received and responded back to curl in the middle of proxing operation. In the good case, this happens at the very end after transferred content is fully proxied. We checked content server, it behaves identically the same in both cases, it always returns 404 immediately after receiving HTTP POST request.
alexmaddriver
on 21 Nov 2017
A few more notes to the previous post:
bad/reactor-netty.log, line 6354: HttpClient makes FLUSH, but there is no "Writing object EmptyLastHttpContent" following it. In my understanding this is the reason why connection's encoder is left in invalid state.
alexmaddriver
on 21 Nov 2017
Hi
I can confirm this bug on my side. Similar scenario, uploading file via https. Switching to http solves my problems.
Bug occurs after:
USER_EVENT: SslCloseCompletionEvent(java.nio.channels.ClosedChannelException)
Normally it returns:
USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)
kzimnicki-n26
on 1 Dec 2017
@kzimnicki-n26 any chance to provide some simple example?
violetagg
on 1 Dec 2017
Hi,
Recently we added several fixes that might be relevant to this issue.
Please check the current snapshot version.
Regards,
Violeta
violetagg
on 27 Feb 2018
After updating to 0.7.5-RELEASE I still have this issue. It seems that after at least one error appears "in WebClient" (for example 500 from some rest service ) it will always fail with io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest.
Disabling pool fixes the issue:
new ReactorClientHttpConnector(
options -> options.disablePool()
);
env:
spring-boot 2.0.0.RELEASE
reactor-core 3.1.5.RELEASE
reactor-netty 0.7.5.RELEASE
I wish I could, but "extracting" some minimal POC from my production code to reproduce this issue is not so easy ;/
myhau
on 5 Mar 2018
@myhau Can you at least describe the scenario that you are doing in order to receive the exception?
violetagg
on 5 Mar 2018
Closing this one.
If you are able to provide a described scenario or even a reproducible example you can reopen it.
violetagg
on 3 Apr 2018
@violetagg @smaldini I was able to reproduce a problem which might be similar to what was originally reported in this issue. The exception is the same, io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest.
Steps to reproduce (tested with Ubuntu 16.04):
Starting the test application, reactor-fibonacci:
git clone https://github.com/lhotari/reactor-fibonacci
cd reactor-fibonacci
./gradlew --console=plain -PfileLoggingLevel=info -PrunArgs=--post,--ssl run
then creating some requests in another terminal:
curl -k https://127.0.0.1:8888/12
# run another time if the first time is successful
curl -k https://127.0.0.1:8888/12
In my case, I get this error as the first error in log-output.log in less than 15 seconds after triggering curl -k https://127.0.0.1:8888/12:
[reactor-http-nio-4] ERROR reactor.ipc.netty.channel.ChannelOperations - [HttpServer] Error processing connection. Requesting close
the channel
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106)
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:291)
at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:465)
at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:191)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300)
at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135)
at reactor.ipc.netty.FutureMono.lambda$deferFuture$0(FutureMono.java:68)
at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:134)
at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
at reactor.ipc.netty.NettyOutbound.subscribe(NettyOutbound.java:356)
at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51)
at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:381)
at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:501)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86)
at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
... 31 more
The next errors are
[reactor-http-nio-2] ERROR reactor.ipc.netty.channel.ChannelOperations - [HttpClient] Error processing connection. Requesting close the channel
java.io.IOException: Connection closed prematurely
at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:271)
at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420)
and
[reactor-http-nio-2] ERROR reactor.ipc.netty.channel.ContextHandler - Error cannot be forwarded to user-facing Mono
java.io.IOException: Connection closed prematurely
at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:271)
at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHa
ndler.java:420)
(lots of these in the log)
There is an option to disable the connection pool. In that case these errors get printed to the log:
[reactor-http-nio-2] WARN io.netty.handler.ssl.SslHandler - [id: 0xa414cc3c, L:/127.0.0.1:44702 - R:/127.0.0.177:8888] Last write attempt timed out; force-closing the connection.
[reactor-http-nio-2] WARN io.netty.handler.ssl.SslHandler - [id: 0x25deb5ae, L:/127.0.0.1:59788 - R:/127.0.0.137:8888] Last write attempt timed out; force-closing the connection.
[reactor-http-nio-1] WARN io.netty.handler.ssl.SslHandler - [id: 0x93634478, L:/127.0.0.1:58450 - R:/127.0.0.135:8888] Last write attempt timed out; force-closing the connection.
[reactor-http-nio-1] WARN io.netty.handler.ssl.SslHandler - [id: 0x697bea43, L:/127.0.0.1:50544 - R:/127.0.0.107:8888] Last write attempt timed out; force-closing the connection.
That could be a hint of the problem.
One important detail is that reproducing the issue consistently required switching to use runtime 'io.netty:netty-tcnative-boringssl-static:2.0.8.Final' dependency.
I noticed that the default JDK SSL implementation is extremely slow compared to using netty-tcnative.
There are some differences in how flushing behaves with netty-tcnative. When running Reactor Netty tests with netty-tcnative-boringssl-static on the classpath, "reactor.ipc.netty.tcp.TcpServerTests > sendFileSecure" fails. I'm not sure if that is related to the problem. This commit reproduces the problem in TcpServerTests and shows that the particular test fails when SslProvider.OPENSSL is used on client.
With debug logging there are a lot of these errors, could be another hint that helps investigation:
[reactor-http-nio-4] DEBUG r.i.n.h.server.HttpServerOperations - Dropped HTTP content, Since response has been sent already:DefaultHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 627, cap: 627/627, unwrapped: PooledUnsafeDirectByteBuf(ridx: 8821, widx: 8821, cap: 16384)), decoderResult: success)
This is logged in HttpServerHandler.
Debug logging produces a lot of output (18GB when I tried) due to logging of request/response bodies. There is -PfileLoggingLevel=debug parameter to enable debug logging to log-output.log in the app to reproduce the issue.
I hope this helps.
Did I find something relevant or is there a problem with the application that reproduces the problem?
I've been experimenting with a lot of crazy stuff so it's possible that I've become blind to some obvious problems in the test application itself.
lhotari
on 9 Apr 2018
@lhotari thanks I can reproduce it on Ubuntu. Debugging ...
violetagg
on 10 Apr 2018
@violetagg What would be the a good way to debug this type of problem?
I also tried debugging without making real progress. In debugging the test app, the vast size of the Netty debug logs is a problem which makes it hard to inspect them. I made this temporary change in logging in my local reactor-netty snapshot to reduce the log size by skipping the logging of the convent of the messages: https://github.com/reactor/reactor-netty/commit/95ef6bf3 . I haven't had the time to inspect the logs. Are there tooling scripts that help spotting state transition problems from Netty debug logs? 263MB of logs is a lot of work to scroll through. :)
lhotari
on 13 Apr 2018
@lhotari this one is a tough one. I put several very tiny logs here and there to understand what's going on. The important thing is to correlate the connection id. The problem is really weird and I'm still debugging it.
violetagg
on 13 Apr 2018
@lhotari the test seems to be wrong here https://github.com/lhotari/reactor-fibonacci/blob/master/src/main/java/reactor/ipc/netty/profiling/ReactorFibonacci.java#L121, it should be flatMap not map.
Currently, nothing is subscribing to the server request.receive() and consuming the response, meaning that the response is terminated before body is fully read - identified separately as #323 and #324. We have worked out something locally with @violetagg today and got rid of the IllegalStateException. However even with our local fix, the test starts hanging so we're not out of the water yet :(
smaldini
on 18 Apr 2018
Thanks @smaldini and @violetagg . I fixed the problem you mentioned and also another bug in the test: https://github.com/lhotari/reactor-fibonacci/commit/06463f9dfa094f92daf467be11bdbbff683fde7d . I also see the test hanging with both http (non-ssl) and https (ssl). I wonder if the test app now reproduces issue https://github.com/reactor/reactor-netty/issues/316 ?
lhotari
on 18 Apr 2018
@lhotari try #327
I cannot see the hanging with this fix.
violetagg
on 19 Apr 2018
@violetagg Yes, #327 fixes the problem that the test app started hanging. Great work!
lhotari
on 19 Apr 2018
@lhotari thanks for testing
I'm closing this issue we created #323 and #324
violetagg
on 20 Apr 2018
@violetagg The java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest error also appears with the test app provided in https://github.com/spring-cloud/spring-cloud-gateway/issues/228 . I tried the test app with the recent snapshot versions. The issue remains, but it seems that it takes longer to reproduce when using reactor-netty 0.7.7.BUILD-SNAPSHOT. I did a small change to the test app in my fork to see if consuming the body would change the results. It didn't. When client connection pooling is disabled, the issue cannot be reproduced with the spring-cloud-gw-issue-228 test app (java -Dspring.cloud.gateway.httpclient.pool.type=DISABLED -jar target/gateway-0.0.1-SNAPSHOT.jar).
Do you believe that fixing #323 and #324 will fix the issue reported in https://github.com/spring-cloud/spring-cloud-gateway/issues/228 ?
lhotari
on 1 May 2018
Do you believe that fixing #323 and #324 will fix the issue reported in spring-cloud/spring-cloud-gateway#228 ?
@lhotari Yes I believe so
violetagg
on 1 May 2018
@lhotari can you try this fix #343 - it is for #323
violetagg
on 1 May 2018
I tested the fix #343 and I wasn't able to reproduce the java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest problem with the spring-cloud-gw-issue-228 test app . Great work @violetagg !
lhotari
on 1 May 2018
I'm seeing this issue with 0.7.7.RELEASE version (using Spring boot 2.0.2)
Before seeing this exception there's this one in the logs:
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
and then:
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106)
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290)
at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464)
at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300)
at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135)
at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197)
at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141)
at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60)
at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47)
at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:46)
at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380)
at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86)
at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
... 32 common frames omitted
madgnome
on 15 May 2018
@madgnome There might be different causes that lead to this IllegalStateException any chance to provide reproducible scenario?
Thanks,
Violeta
violetagg
on 15 May 2018
I tried to reproduce locally but no luck. Disabling the pool fixed the issue in prod.
@violetagg Do you have any tips to reproduce this type of issue?
madgnome
on 16 May 2018
I've turned on more logging but not all of it because it's too verbose unfortunately.
Successful log sequences
5/16/184:34:59.857 PM => POST "https://xyz.net/v1/events"
5/16/184:34:59.857 PM Acquiring existing channel from pool: DefaultPromise@6c7d861b(success: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]) SimpleChannelPool{activeConnections=0}
5/16/184:34:59.858 PM Acquired [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 1 active connections
5/16/184:34:59.858 PM Acquired active channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/184:34:59.858 PM [HttpClient] [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1365/1092126209@71733611}
5/16/184:35:00.517 PM [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] Received response (auto-read:false) : [Access-Control-Allow-Credentials=true, Access-Control-Allow-Headers=Origin, X-Requested-With, Content-Type, Accept, Access-Control-Allow-Methods=POST, OPTIONS, Access-Control-Allow-Origin=*, Access-Control-Max-Age=3600, Content-Type=application/json, Date=Wed, 16 May 2018 06:35:00 GMT, Server=nginx/1.12.2, Content-Length=66, Connection=keep-alive]
5/16/184:35:00.518 PM <= [200] POST "https://xyz.net/v1/events" 661ms
5/16/184:35:00.518 PM [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
5/16/184:35:00.518 PM [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] Received last HTTP packet
5/16/184:35:00.518 PM Releasing channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/184:35:00.518 PM Released [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 0 active connections
Last log sequences before the first exception:
5/16/184:39:23.085 PM => POST "https://xyz.net/v1/events"
5/16/184:39:23.085 PM Acquiring existing channel from pool: DefaultPromise@4009105c(incomplete) SimpleChannelPool{activeConnections=0}
5/16/184:39:23.085 PM Acquired [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 1 active connections
5/16/184:39:23.085 PM Acquired active channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/184:39:23.085 PM [HttpClient] [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1365/1092126209@6a5ecd33}
5/16/184:39:23.329 PM [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] Received response (auto-read:false) : [Access-Control-Allow-Credentials=true, Access-Control-Allow-Headers=Origin, X-Requested-With, Content-Type, Accept, Access-Control-Allow-Methods=POST, OPTIONS, Access-Control-Allow-Origin=*, Access-Control-Max-Age=3600, Content-Type=application/json, Date=Wed, 16 May 2018 06:39:23 GMT, Server=nginx/1.12.2, Content-Length=65, Connection=keep-alive]
5/16/184:39:23.329 PM <= [200] POST "https://xyz.net/v1/events" 244ms
5/16/184:39:23.329 PM [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
5/16/184:39:23.330 PM [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] Received last HTTP packet
5/16/184:39:23.330 PM Releasing channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/184:39:23.330 PM Released [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 0 active connections
Shit starts to hit the fan here
Log when the SSL exception happens, just before getting in the bad state
5/16/18 4:40:54.154 PM => POST "https://xyz.net/v1/events"
5/16/18 4:40:54.169 PM Acquiring existing channel from pool: DefaultPromise@1574de1d(success: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]) SimpleChannelPool{activeConnections=0}
5/16/18 4:40:54.169 PM Acquired [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 1 active connections
5/16/18 4:40:54.169 PM Acquired active channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/18 4:40:54.169 PM [HttpClient] [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1365/1092126209@562fd009}
5/16/18 4:40:54.170 PM <= [-1] POST "https://xyz.net/v1/events" 16ms
5/16/18 4:40:54.170 PM Error publishing analytic events
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
5/16/18 4:40:54.171 PM Releasing channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/18 4:40:54.171 PM Released [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 0 active connections
Bad state log statements:
5/16/18 4:40:55.169 PM => POST "https://xyz.net/v1/events"
5/16/18 4:40:55.170 PM Acquiring existing channel from pool: DefaultPromise@48c4b3fe(incomplete) SimpleChannelPool{activeConnections=0}
5/16/18 4:40:55.170 PM Acquired [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 1 active connections
5/16/18 4:40:55.170 PM Acquired active channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/18 4:40:55.170 PM [HttpClient] [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1365/1092126209@4735686a}
5/16/18 4:40:55.171 PM <= [-1] POST "https://xyz.net/v1/events" 2ms
5/16/18 4:40:55.172 PM Error publishing analytic events
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106)
at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290)
at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464)
at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300)
at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135)
at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197)
at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141)
at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60)
at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47)
at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:46)
at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380)
at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86)
at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
... 32 common frames omitted
5/16/18 4:40:55.172 PM Releasing channel: [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443]
5/16/18 4:40:55.172 PM Released [id: 0x90557300, L:/10.255.0.3:50690 - R:xyz.net/52.23.127.36:443], now 0 active connections
Hopefully that helps
madgnome
on 16 May 2018
I've managed to reproduce with HttpClient logging handler logs enabled:
Last successful log sequences:
鈿狅笍It's in reverse order (older log are at the end)
5/17/18 1:28:15.650 Releasing channel: [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443]
5/17/18 1:28:15.650 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Disposing context reactor.ipc.netty.channel.PooledClientContextHandler@7dbb06e5
5/17/18 1:28:15.650 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] USER_EVENT: [Handler Terminated]
5/17/18 1:28:15.649 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Received last HTTP packet
5/17/18 1:28:15.649 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
5/17/18 1:28:15.649 <= [200] POST "https://xyz.net/v1/events" 87ms
5/17/18 1:28:15.649 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Received response (auto-read:false) : [Access-Control-Allow-Credentials=true, Access-Control-Allow-Headers=Origin, X-Requested-With, Content-Type, Accept, Access-Control-Allow-Methods=POST, OPTIONS, Access-Control-Allow-Origin=*, Access-Control-Max-Age=3600, Content-Type=application/json, Date=Thu, 17 May 2018 03:28:15 GMT, Server=nginx/1.12.2, Content-Length=65, Connection=keep-alive]
5/17/18 1:28:15.648 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] READ: 444B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d |HTTP/1.1 200 OK.| |00000010| 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d |.Access-Control-| |00000020| 41 6c 6c 6f 77 2d 43 72 65 64 65 6e 74 69 61 6c |Allow-Credential| |00000030| 73 3a 20 74 72 75 65 0d 0a 41 63 63 65 73 73 2d |s: true..Access-| |00000040| 43 6f 6e 74 72 6f 6c 2d 41 6c 6c 6f 77 2d 48 65 |Control-Allow-He| |00000050| 61 64 65 72 73 3a 20 4f 72 69 67 69 6e 2c 20 58 |aders: Origin, X| |00000060| 2d 52 65 71 75 65 73 74 65 64 2d 57 69 74 68 2c |-Requested-With,| |00000070| 20 43 6f 6e 74 65 6e 74 2d 54 79 70 65 2c 20 41 | Content-Type, A| |00000080| 63 63 65 70 74 0d 0a 41 63 63 65 73 73 2d 43 6f |ccept..Access-Co| |00000090| 6e 74 72 6f 6c 2d 41 6c 6c 6f 77 2d 4d 65 74 68 |ntrol-Allow-Meth| |000000a0| 6f 64 73 3a 20 50 4f 53 54 2c 20 4f 50 54 49 4f |ods: POST, OPTIO| |000000b0| 4e 53 0d 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 |NS..Access-Contr| |000000c0| 6f 6c 2d 41 6c 6c 6f 77 2d 4f 72 69 67 69 6e 3a |ol-Allow-Origin:| |000000d0| 20 2a 0d 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 | *..Access-Contr| |000000e0| 6f 6c 2d 4d 61 78 2d 41 67 65 3a 20 33 36 30 30 |ol-Max-Age: 3600| |000000f0| 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 |..Content-Type: | |00000100| 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e |application/json| |00000110| 0d 0a 44 61 74 65 3a 20 54 68 75 2c 20 31 37 20 |..Date: Thu, 17 | |00000120| 4d 61 79 20 32 30 31 38 20 30 33 3a 32 38 3a 31 |May 2018 03:28:1| |00000130| 35 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 6e |5 GMT..Server: n| |00000140| 67 69 6e 78 2f 31 2e 31 32 2e 32 0d 0a 43 6f 6e |ginx/1.12.2..Con| |00000150| 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 36 35 0d |tent-Length: 65.| |00000160| 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 |.Connection: kee| |00000170| 70 2d 61 6c 69 76 65 0d 0a 0d 0a 7b 0a 20 20 22 |p-alive....{. "| |00000180| 69 6e 76 61 6c 69 64 5f 70 72 6f 64 75 63 74 22 |invalid_product"| |00000190| 3a 20 30 2c 20 0a 20 20 22 70 72 6f 63 65 73 73 |: 0, . "process| |000001a0| 65 64 20 22 3a 20 31 33 2c 20 0a 20 20 22 73 6b |ed ": 13, . "sk| |000001b0| 69 70 70 65 64 22 3a 20 30 0a 7d 0a |ipped": 0.}. | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.565 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] FLUSH
5/17/18 1:28:15.565 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 5B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 30 0d 0a 0d 0a |0.... | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.565 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] FLUSH
5/17/18 1:28:15.564 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 2B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 0d 0a |.. | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.564 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 7391B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 7b 22 65 76 65 6e 74 73 22 3a 5b 7b 22 6e 61 6d |{"events":[{"nam| |00000010| 65 22 3a 22 61 74 6c 61 73 73 69 61 6e 2e 66 61 |e":"atlassian.fa| |00000020| 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 70 75 62 |bric.frontendpub| |00000030| 73 75 62 2e 63 68 61 6e 6e 65 6c 73 2e 73 75 62 |sub.channels.sub| |00000040| 73 63 72 69 62 65 64 22 2c 22 73 65 72 76 65 72 |scribed","server| |00000050| 22 3a 22 70 72 6f 64 75 63 74 69 6f 6e 22 2c 22 |":"production","| |00000060| 70 72 6f 64 75 63 74 22 3a 22 68 69 70 63 68 61 |product":"hipcha| |00000070| 74 22 2c 22 70 72 6f 70 65 72 74 69 65 73 22 3a |t","properties":| |00000080| 7b 22 63 6c 69 65 6e 74 2e 70 6c 61 74 66 6f 72 |{"client.platfor| |00000090| 6d 22 3a 22 57 45 42 22 2c 22 63 6c 69 65 6e 74 |m":"WEB","client| |000000a0| 2e 76 65 72 73 69 6f 6e 22 3a 22 31 2e 30 22 2c |.version":"1.0",| |000000b0| 22 63 68 61 6e 6e 65 6c 43 6f 75 6e 74 22 3a 31 |"channelCount":1| |000000c0| 32 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |2},"serverTime":| |000000d0| 31 35 32 36 35 32 37 36 39 35 32 33 35 2c 22 75 |1526527695235,"u| |000000e0| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |000000f0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000100| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000110| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00000120| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000130| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000140| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00000150| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00000160| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000170| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000180| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000190| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |000001a0| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |000001b0| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |000001c0| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |000001d0| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |000001e0| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |000001f0| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000200| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00000210| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000220| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000230| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00000240| 2f 35 35 32 33 35 35 33 37 2d 34 38 37 34 2d 35 |/55235537-4874-5| |00000250| 34 30 34 2d 61 38 62 34 2d 33 36 32 61 65 39 37 |404-a8b4-362ae97| |00000260| 34 62 37 39 64 22 2c 22 63 68 61 6e 6e 65 6c 54 |4b79d","channelT| |00000270| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00000280| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00000290| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |000002a0| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |000002b0| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |000002c0| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |000002d0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |000002e0| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |000002f0| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00000300| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |00000310| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |00000320| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00000330| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000340| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000350| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00000360| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00000370| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000380| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000390| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |000003a0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000003b0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |000003c0| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |000003d0| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |000003e0| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |000003f0| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00000400| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00000410| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00000420| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00000430| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00000440| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00000450| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00000460| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000470| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00000480| 65 72 73 61 74 69 6f 6e 2f 32 63 30 39 39 30 66 |ersation/2c0990f| |00000490| 36 2d 63 30 61 39 2d 35 64 31 62 2d 39 33 30 36 |6-c0a9-5d1b-9306| |000004a0| 2d 36 63 37 64 63 64 66 62 33 36 66 39 22 2c 22 |-6c7dcdfb36f9","| |000004b0| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |000004c0| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |000004d0| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |000004e0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |000004f0| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00000500| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00000510| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00000520| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00000530| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00000540| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00000550| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00000560| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00000570| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00000580| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000590| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |000005a0| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |000005b0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |000005c0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |000005d0| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |000005e0| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |000005f0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000600| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000610| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000620| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00000630| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00000640| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00000650| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00000660| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00000670| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00000680| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000690| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |000006a0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000006b0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |000006c0| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |000006d0| 2f 32 64 33 66 39 30 30 31 2d 31 65 61 39 2d 34 |/2d3f9001-1ea9-4| |000006e0| 38 34 31 2d 38 32 34 37 2d 35 38 37 30 64 30 32 |841-8247-5870d02| |000006f0| 63 38 66 36 31 22 2c 22 63 68 61 6e 6e 65 6c 54 |c8f61","channelT| |00000700| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00000710| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00000720| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00000730| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00000740| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00000750| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00000760| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00000770| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |00000780| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00000790| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |000007a0| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |000007b0| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |000007c0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |000007d0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |000007e0| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |000007f0| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00000800| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000810| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000820| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00000830| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000840| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000850| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00000860| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00000870| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00000880| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00000890| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |000008a0| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |000008b0| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |000008c0| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |000008d0| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |000008e0| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |000008f0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000900| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00000910| 65 72 73 61 74 69 6f 6e 2f 37 36 30 32 66 64 33 |ersation/7602fd3| |00000920| 31 2d 65 32 62 32 2d 35 36 61 33 2d 39 35 66 31 |1-e2b2-56a3-95f1| |00000930| 2d 36 36 35 36 31 32 39 66 37 61 31 61 22 2c 22 |-6656129f7a1a","| |00000940| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00000950| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00000960| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00000970| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00000980| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00000990| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |000009a0| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |000009b0| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |000009c0| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |000009d0| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |000009e0| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |000009f0| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00000a00| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00000a10| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000a20| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000a30| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00000a40| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000a50| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000a60| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00000a70| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00000a80| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000a90| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000aa0| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000ab0| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00000ac0| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00000ad0| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00000ae0| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00000af0| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00000b00| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00000b10| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000b20| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00000b30| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000b40| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000b50| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00000b60| 2f 34 34 66 36 34 33 35 62 2d 36 36 32 35 2d 35 |/44f6435b-6625-5| |00000b70| 36 34 31 2d 62 31 34 64 2d 65 38 37 32 39 38 65 |641-b14d-e87298e| |00000b80| 65 61 39 31 30 22 2c 22 63 68 61 6e 6e 65 6c 54 |ea910","channelT| |00000b90| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00000ba0| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00000bb0| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00000bc0| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00000bd0| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00000be0| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00000bf0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00000c00| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |00000c10| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00000c20| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |00000c30| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |00000c40| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00000c50| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000c60| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000c70| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00000c80| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00000c90| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000ca0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000cb0| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00000cc0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000cd0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000ce0| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00000cf0| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00000d00| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00000d10| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00000d20| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00000d30| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00000d40| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00000d50| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00000d60| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00000d70| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00000d80| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000d90| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00000da0| 65 72 73 61 74 69 6f 6e 2f 35 33 64 34 34 31 39 |ersation/53d4419| |00000db0| 62 2d 34 61 34 32 2d 34 61 62 62 2d 39 65 61 35 |b-4a42-4abb-9ea5| |00000dc0| 2d 32 37 66 65 35 37 62 36 34 32 30 33 22 2c 22 |-27fe57b64203","| |00000dd0| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00000de0| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00000df0| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00000e00| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00000e10| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00000e20| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00000e30| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00000e40| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00000e50| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00000e60| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00000e70| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00000e80| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00000e90| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00000ea0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000eb0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000ec0| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00000ed0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000ee0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000ef0| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00000f00| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00000f10| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000f20| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000f30| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000f40| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00000f50| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00000f60| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00000f70| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00000f80| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00000f90| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00000fa0| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000fb0| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00000fc0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000fd0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000fe0| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00000ff0| 2f 63 35 39 32 36 35 64 61 2d 33 33 61 30 2d 35 |/c59265da-33a0-5| |00001000| 35 63 35 2d 62 37 31 36 2d 64 66 36 39 30 37 65 |5c5-b716-df6907e| |00001010| 37 34 39 31 35 22 2c 22 63 68 61 6e 6e 65 6c 54 |74915","channelT| |00001020| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00001030| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00001040| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00001050| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00001060| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00001070| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00001080| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00001090| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |000010a0| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |000010b0| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |000010c0| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |000010d0| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |000010e0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |000010f0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001100| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00001110| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00001120| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001130| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001140| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00001150| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00001160| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001170| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00001180| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00001190| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |000011a0| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |000011b0| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |000011c0| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |000011d0| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |000011e0| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |000011f0| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00001200| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00001210| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001220| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00001230| 65 72 73 61 74 69 6f 6e 2f 66 64 62 38 38 31 38 |ersation/fdb8818| |00001240| 31 2d 61 39 63 30 2d 35 37 65 37 2d 38 38 30 34 |1-a9c0-57e7-8804| |00001250| 2d 35 37 63 33 36 64 36 64 64 33 39 30 22 2c 22 |-57c36d6dd390","| |00001260| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00001270| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00001280| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00001290| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |000012a0| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |000012b0| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |000012c0| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |000012d0| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |000012e0| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |000012f0| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00001300| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00001310| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00001320| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00001330| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001340| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001350| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00001360| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001370| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001380| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00001390| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |000013a0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |000013b0| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |000013c0| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |000013d0| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |000013e0| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |000013f0| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00001400| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00001410| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00001420| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00001430| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00001440| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00001450| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00001460| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001470| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00001480| 2f 32 35 30 34 31 61 34 33 2d 63 65 62 36 2d 35 |/25041a43-ceb6-5| |00001490| 65 62 35 2d 61 34 38 38 2d 64 66 39 63 63 38 63 |eb5-a488-df9cc8c| |000014a0| 39 31 38 38 64 22 2c 22 63 68 61 6e 6e 65 6c 54 |9188d","channelT| |000014b0| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |000014c0| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |000014d0| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |000014e0| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |000014f0| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00001500| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00001510| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00001520| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |00001530| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00001540| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |00001550| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |00001560| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00001570| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001580| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001590| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |000015a0| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |000015b0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |000015c0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |000015d0| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |000015e0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000015f0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001600| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00001610| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00001620| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00001630| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00001640| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00001650| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00001660| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00001670| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00001680| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00001690| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |000016a0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |000016b0| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |000016c0| 65 72 73 61 74 69 6f 6e 2f 33 31 39 62 39 32 36 |ersation/319b926| |000016d0| 63 2d 62 66 30 61 2d 35 39 33 63 2d 62 36 38 32 |c-bf0a-593c-b682| |000016e0| 2d 34 37 64 34 62 61 65 64 37 62 31 62 22 2c 22 |-47d4baed7b1b","| |000016f0| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00001700| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00001710| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00001720| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00001730| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00001740| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00001750| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00001760| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00001770| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00001780| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00001790| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |000017a0| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |000017b0| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |000017c0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |000017d0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |000017e0| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |000017f0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001800| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001810| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00001820| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00001830| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001840| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00001850| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00001860| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00001870| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00001880| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00001890| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |000018a0| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |000018b0| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |000018c0| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |000018d0| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |000018e0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000018f0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001900| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00001910| 2f 31 34 36 63 66 64 35 61 2d 37 63 32 37 2d 35 |/146cfd5a-7c27-5| |00001920| 62 65 66 2d 62 39 31 31 2d 39 39 38 38 38 64 34 |bef-b911-99888d4| |00001930| 36 65 38 63 65 22 2c 22 63 68 61 6e 6e 65 6c 54 |6e8ce","channelT| |00001940| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00001950| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00001960| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00001970| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00001980| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00001990| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |000019a0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |000019b0| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |000019c0| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |000019d0| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |000019e0| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |000019f0| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00001a00| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001a10| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001a20| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00001a30| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00001a40| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001a50| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001a60| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00001a70| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00001a80| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001a90| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00001aa0| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00001ab0| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00001ac0| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00001ad0| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00001ae0| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00001af0| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00001b00| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00001b10| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00001b20| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00001b30| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001b40| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00001b50| 65 72 73 61 74 69 6f 6e 2f 39 32 63 31 65 35 35 |ersation/92c1e55| |00001b60| 66 2d 64 39 35 63 2d 35 36 32 34 2d 39 31 63 64 |f-d95c-5624-91cd| |00001b70| 2d 37 64 36 32 62 32 34 61 34 37 33 38 22 2c 22 |-7d62b24a4738","| |00001b80| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00001b90| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00001ba0| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00001bb0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00001bc0| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00001bd0| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00001be0| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00001bf0| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00001c00| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00001c10| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00001c20| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00001c30| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00001c40| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00001c50| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001c60| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001c70| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00001c80| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001c90| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001ca0| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00001cb0| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00001cc0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001cd0| 35 30 64 36 64 33 38 36 62 65 33 22 7d 5d 7d |50d6d386be3"}]} | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.564 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 6B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 31 63 64 66 0d 0a |1cdf.. | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.563 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] FLUSH
5/17/18 1:28:15.563 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 299B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 50 4f 53 54 20 2f 76 31 2f 65 76 65 6e 74 73 20 |POST /v1/events | |00000010| 48 54 54 50 2f 31 2e 31 0d 0a 75 73 65 72 2d 61 |HTTP/1.1..user-a| |00000020| 67 65 6e 74 3a 20 52 65 61 63 74 6f 72 4e 65 74 |gent: ReactorNet| |00000030| 74 79 2f 30 2e 37 2e 37 2e 52 45 4c 45 41 53 45 |ty/0.7.7.RELEASE| |00000040| 0d 0a 74 72 61 6e 73 66 65 72 2d 65 6e 63 6f 64 |..transfer-encod| |00000050| 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 68 6f |ing: chunked..ho| |00000060| 73 74 3a 20 6d 67 61 73 2e 70 72 6f 64 2e 70 75 |st: mgas.prod.pu| |00000070| 62 6c 69 63 2e 61 74 6c 2d 70 61 61 73 2e 6e 65 |blic.atl-paas.ne| |00000080| 74 0d 0a 41 63 63 65 70 74 3a 20 61 70 70 6c 69 |t..Accept: appli| |00000090| 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 |cation/json;char| |000000a0| 73 65 74 3d 55 54 46 2d 38 0d 0a 58 2d 42 33 2d |set=UTF-8..X-B3-| |000000b0| 54 72 61 63 65 49 64 3a 20 33 61 39 66 33 35 36 |TraceId: 3a9f356| |000000c0| 37 38 33 62 39 31 62 34 62 0d 0a 58 2d 42 33 2d |783b91b4b..X-B3-| |000000d0| 53 70 61 6e 49 64 3a 20 33 61 39 66 33 35 36 37 |SpanId: 3a9f3567| |000000e0| 38 33 62 39 31 62 34 62 0d 0a 58 2d 42 33 2d 53 |83b91b4b..X-B3-S| |000000f0| 61 6d 70 6c 65 64 3a 20 30 0d 0a 43 6f 6e 74 65 |ampled: 0..Conte| |00000100| 6e 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 |nt-Type: applica| |00000110| 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 73 65 |tion/json;charse| |00000120| 74 3d 55 54 46 2d 38 0d 0a 0d 0a |t=UTF-8.... | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.563 [HttpClient] [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1321/1871970505@2bc6c88c}
5/17/18 1:28:15.563 Acquired active channel: [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443]
5/17/18 1:28:15.562 Acquired [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443], now 1 active connections
5/17/18 1:28:15.562 Acquiring existing channel from pool: DefaultPromise@15d7d7a5(success: [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443]) SimpleChannelPool{activeConnections=0}
5/17/18 1:28:15.562 => POST "https://xyz.net/v1/events"
Shit starts to hit the fan from here:
Log when the SSL exception happens:
鈿狅笍It's in reverse order (older log are at the end)
5/17/18 5:33:11.548 Released [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443], now 0 active connections
5/17/18 5:33:11.548 Releasing channel: [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443]
5/17/18 5:33:11.548 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] Disposing context reactor.ipc.netty.channel.PooledClientContextHandler@7dbb06f4
5/17/18 5:33:11.548 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] USER_EVENT: [Handler Terminated]
5/17/18 5:33:11.548 Error publishing analytic events
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
5/17/18 5:33:11.547 <= [500] POST "https://xyz.net/v1/events" 2ms
5/17/18 5:33:11.546 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] FLUSH
5/17/18 5:33:11.546 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] WRITE: 299B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 50 4f 53 54 20 2f 76 31 2f 65 76 65 6e 74 73 20 |POST /v1/events | |00000010| 48 54 54 50 2f 31 2e 31 0d 0a 75 73 65 72 2d 61 |HTTP/1.1..user-a| |00000020| 67 65 6e 74 3a 20 52 65 61 63 74 6f 72 4e 65 74 |gent: ReactorNet| |00000030| 74 79 2f 30 2e 37 2e 37 2e 52 45 4c 45 41 53 45 |ty/0.7.7.RELEASE| |00000040| 0d 0a 74 72 61 6e 73 66 65 72 2d 65 6e 63 6f 64 |..transfer-encod| |00000050| 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 68 6f |ing: chunked..ho| |00000060| 73 74 3a 20 6d 67 61 73 2e 70 72 6f 64 2e 70 75 |st: mgas.prod.pu| |00000070| 62 6c 69 63 2e 61 74 6c 2d 70 61 61 73 2e 6e 65 |blic.atl-paas.ne| |00000080| 74 0d 0a 41 63 63 65 70 74 3a 20 61 70 70 6c 69 |t..Accept: appli| |00000090| 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 |cation/json;char| |000000a0| 73 65 74 3d 55 54 46 2d 38 0d 0a 58 2d 42 33 2d |set=UTF-8..X-B3-| |000000b0| 54 72 61 63 65 49 64 3a 20 62 32 34 34 33 32 30 |TraceId: b244320| |000000c0| 35 39 63 33 30 66 30 64 30 0d 0a 58 2d 42 33 2d |59c30f0d0..X-B3-| |000000d0| 53 70 61 6e 49 64 3a 20 62 32 34 34 33 32 30 35 |SpanId: b2443205| |000000e0| 39 63 33 30 66 30 64 30 0d 0a 58 2d 42 33 2d 53 |9c30f0d0..X-B3-S| |000000f0| 61 6d 70 6c 65 64 3a 20 30 0d 0a 43 6f 6e 74 65 |ampled: 0..Conte| |00000100| 6e 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 |nt-Type: applica| |00000110| 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 73 65 |tion/json;charse| |00000120| 74 3d 55 54 46 2d 38 0d 0a 0d 0a |t=UTF-8.... | +--------+-------------------------------------------------+----------------+
5/17/18 5:33:11.546 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] Writing object DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) POST /v1/events HTTP/1.1 user-agent: ReactorNetty/0.7.7.RELEASE transfer-encoding: chunked host: xyz.net Accept: application/json;charset=UTF-8 X-B3-TraceId: b24432059c30f0d0 X-B3-SpanId: b24432059c30f0d0 X-B3-Sampled: 0 Content-Type: application/json;charset=UTF-8
5/17/18 5:33:11.546 [HttpClient] [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1325/830898785@60bee0e3}
5/17/18 5:33:11.545 Acquired active channel: [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443]
5/17/18 5:33:11.545 Acquired [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443], now 1 active connections
5/17/18 5:33:11.545 Acquiring existing channel from pool: DefaultPromise@45009327(success: [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443]) SimpleChannelPool{activeConnections=0}
5/17/18 5:33:11.545 => POST "https://xyz.net/v1/events"
5/17/18 5:33:11.540 PM [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] READ COMPLETE
5/17/18 5:33:11.540 PM [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] USER_EVENT: SslCloseCompletionEvent(SUCCESS)
It looks like the ssl connection gets closed but the channel doesn't.
5/17/18 5:33:11.540 PM [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] READ COMPLETE
5/17/18 5:33:11.540 PM [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] USER_EVENT: SslCloseCompletionEvent(SUCCESS)
Looking at previous logs following a SslCloseCompletionEvent we can see the channel being unregistered
5/17/18 5:30:47.852 [id: 0x8f4fc18c, L:/10.255.0.3:44856 ! R:xyz.net/34.230.139.16:443] UNREGISTERED
5/17/18 5:30:47.852 [id: 0x8f4fc18c, L:/10.255.0.3:44856 ! R:xyz.net/34.230.139.16:443] INACTIVE
5/17/18 5:30:47.852 [id: 0x8f4fc18c, L:/10.255.0.3:44856 ! R:xyz.net/34.230.139.16:443] CLOSE
5/17/18 5:30:47.851 [id: 0x8f4fc18c, L:/10.255.0.3:44856 - R:xyz.net/34.230.139.16:443] READ COMPLETE
5/17/18 5:30:47.770 [id: 0x8f4fc18c, L:/10.255.0.3:44856 - R:xyz.net/34.230.139.16:443] READ COMPLETE
5/17/18 5:30:47.770 [id: 0x8f4fc18c, L:/10.255.0.3:44856 - R:xyz.net/34.230.139.16:443] USER_EVENT: SslCloseCompletionEvent(SUCCESS)
I've created a separate issue to track this #361
madgnome
on 18 May 2018
We have to disable connection pool too. When the post requests contain a large body, it's easy to reproduce this issue. It's frustrating without the pool, the number of requests per second is very limited due to the limit of available ports.
anonymousXrandom
on 28 Jun 2018
@anonymousXrandom What is the Reactor Netty version that you use? We have fixes for this issue.
Please create new issue and describe your scenario there.
violetagg
on 28 Jun 2018
Related issues
rreynaud
路
3Comments
rstoyanchev
路
5Comments
vchekan
路
8Comments
eugene-sadovsky
路
8Comments
izeye
路
3Comments
Most helpful comment
Hello,
We use the latest spring-webflux 5.0.0.RELEASE.
@lnxmad: We disable connection pool when creating WebClient as follows:
WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(options -> options.disablePool()))
.build();
@rstoyanchev: we do not use bodyToXXX, instead directly "targetResponse.body(toDataBuffers())", because of a recent bug with bodyToXXX methods which used to close connections as a side affect. As it was fixed in 0.7.0.RELEASE, we can now certainly switch to bodyToFlux.
The code snippet:
protected Mono>> proxyPostRequest(ServerHttpRequest request) {
return client
.post()
.uri(destinationUriGenerator.apply(request.getURI()))
.headers(headers -> filterRequestHeaders(request, headers))
.headers(this::setUserAgent)
.body(request.getBody(), DataBuffer.class)
.exchange()
.map(toResponseEntity());
}
private Function>> toResponseEntity() {
return targetResponse -> new ResponseEntity<>(
targetResponse.body(toDataBuffers()),
targetResponse.statusCode()
);
}