Spring-cloud-gateway: Gateway don't close websocket session when client close websocket.

[zjengjie]

Gateway won't close websocket session of backend when client close it, and when the backend continue to send message to client, the gateway will show error:

2018-03-30 16:41:53.432 ERROR 9255 --- [ctor-http-nio-2] .a.w.r.e.DefaultErrorWebExceptionHandler : Failed to handle request [GET http://localhost:8080/ws-test]

java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_151]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_151]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_151]
    at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[na:1.8.0_151]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_151]
    at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:901) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1376) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at reactor.ipc.netty.channel.ChannelOperationsHandler.lambda$scheduleFlush$0(ChannelOperationsHandler.java:333) ~[reactor-netty-0.7.5.RELEASE.jar:0.7.5.RELEASE]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.22.Final.jar:4.1.22.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[netty-common-4.1.22.Final.jar:4.1.22.Final]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]

2018-03-30 16:41:53.482 ERROR 9255 --- [ctor-http-nio-2] o.s.w.s.adapter.HttpWebHandlerAdapter    : Failed to handle request [GET http://localhost:8080/ws-test]

java.lang.IllegalStateException: Status and headers already sent
    at reactor.ipc.netty.http.server.HttpServerOperations.status(HttpServerOperations.java:345) ~[reactor-netty-0.7.5.RELEASE.jar:0.7.5.RELEASE]
    at org.springframework.http.server.reactive.ReactorServerHttpResponse.applyStatusCode(ReactorServerHttpResponse.java:69) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at org.springframework.http.server.reactive.AbstractServerHttpResponse.lambda$doCommit$4(AbstractServerHttpResponse.java:213) ~[spring-web-5.0.4.RELEASE.jar:5.0.4.RELEASE]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_151]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380) ~[na:1.8.0_151]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_151]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_151]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_151]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_151]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_151]
...

version Finchley.M9

[gengxiaoxiaoxin]

What method do you use to maintain the session? _spring-security-websocket_ or _spring oauth2_

[zjengjie]

I mean gateway received the CloseWebSocketFrame of client, but don't send it to the backend.

[amizurov]

It's looks like 'Reactor-Netty' when received a close frame immediately respond close handshake and terminate channel.

[dharezlak]

I am experiencing the same behavior. When I close the web socket in the browser the ws connection stays in the CLOSING state for a minute or so and then eventually closes, however, it looks like the close event is not propagated to the downstream server. Where should I look to check for more details? The gateway has security set to permitAll() for the WS endpoint.

[ghost]

我也遇到的同样的问题，前端调用了websocket的close()方法后，后端的@OnClose 方法并没有执行。

[dharezlak]

I traced the problem of not closing the WebSocket connection to the following piece of code in the org.springframework.cloud.gateway.filter.WebsocketRoutingFilter class:

@Override
        public Mono<Void> handle(WebSocketSession session) {
            // pass headers along so custom headers can be sent through
            return client.execute(url, this.headers, new WebSocketHandler() {
                @Override
                public Mono<Void> handle(WebSocketSession proxySession) {
                    // Use retain() for Reactor Netty
                    Mono<Void> proxySessionSend = proxySession
                            .send(session.receive().doOnNext(WebSocketMessage::retain));
                            // .log("proxySessionSend", Level.FINE);
                    Mono<Void> serverSessionSend = session
                            .send(proxySession.receive().doOnNext(WebSocketMessage::retain));
                            // .log("sessionSend", Level.FINE);
                    return Mono.when(proxySessionSend, serverSessionSend).then();
                }

                /**
                 * Copy subProtocols so they are available downstream.
                 * @return
                 */
                @Override
                public List<String> getSubProtocols() {
                    return ProxyWebSocketHandler.this.subProtocols;
                }
            });
        }

When I change Mono.when(proxySessionSend, serverSessionSend).then() to Mono.when(proxySessionSend).then() the connection closes immediately after the client closes it but upstream messages are not working obviously. I suppose better reactive setup is needed here, however, I couldn't make it work so far.

[dharezlak]

This issue can be easily replicated with a simple Spring Boot app:

Maven deps:

<properties>
  <spring-cloud.version>Finchley.M6</spring-cloud.version>
</properties>

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-gateway</artifactId>
  </dependency>
</dependencies>

Web Socket handler:

@Component
public class MyWebSocketHandler implements WebSocketHandler {

    @Override
    public Mono<Void> handle(WebSocketSession session) {
        return session
                .receive()
                .doOnNext(System.out::println)
                .log()
                .doFinally(sig -> System.out.println("Web socket session finished: " + sig.name()))
                .then();
    }
}

Config:

@SpringBootApplication
public class WsdemoApplication {

    @Autowired
    private WebSocketHandler webSocketHandler;

    public static void main(String[] args) {
        SpringApplication.run(WsdemoApplication.class, args);
    }

    @Bean
    public HandlerMapping websocketmapping() {
        Map<String, WebSocketHandler> map = new HashMap<>();
        map.put("/ws2", webSocketHandler);

        SimpleUrlHandlerMapping handlerMapping = new SimpleUrlHandlerMapping();
        handlerMapping.setOrder(1);
        handlerMapping.setUrlMap(map);
        return handlerMapping;
    }

    @Bean
    public WebSocketHandlerAdapter handlerAdapter() {
        return new WebSocketHandlerAdapter();
    }

    @Bean
    public RouteLocator wsGateway(RouteLocatorBuilder builder) {
        return builder.routes()
                .route("ws", predicate -> predicate
                        .path("/ws")
                        .uri("ws://localhost:8080/ws2"))
                .build();
    }
}

In the browser you can test with:

var ws = new WebSocket("ws://localhost:8080/ws");
ws.send("hello");
ws.close();

and

var ws = new WebSocket("ws://localhost:8080/ws2");
ws.send("hello");
ws.close();

For the /ws endpoint (proxied) the connection is not closed immediately, but for /ws2 (not proxied) it is.

[dharezlak]

Has anyone managed to come up with a workaround for this problem? I am revolving around reactor.ipc.netty.http.server.HttpServerWSOperations and noticed that the reactor.ipc.netty.channel.ChannelOperations.onHandlerTerminate() is not called to complete the receive Flux of WS messages.

[dharezlak]

I just tried with Spring Boot 2.0.2.RELEASE and Gateway Finchley.RC1 and the problem still occurs.

[dharezlak]

@spencergibb Could you advice on how to debug this matter further? It prevents us from using the Gateway for all websocket traffic to our application. If there is anything else I could provide to trace it just let me know.

[dharezlak]

I created a simple Spring Boot app reproducing the problem at https://github.com/dharezlak/spring-cloud-gateway-websocket.

[spencergibb]

Thanks, this is a high priority

[spencergibb]

@dharezlak I'm unable to reproduce a similar problem. I'm sure I'm not doing the same thing as you. Can you please include an html file and javascript that recreates the problem?

Can anyone else provide a simple project that recreates the problem?

[spencergibb]

This may be related to https://jira.spring.io/browse/SPR-16774

[spencergibb]

@dharezlak @zjengjie @amizurov can you folks try with the latest Finchley SNAPSHOT when the build is done (shortly) and boot 2.0.2? We can reopen if there are still issues.

[dharezlak]

@spencergibb With the latest Finchley.BUILD-SNAPSHOT and Boot 2.0.2.RELEASE build it is working as expected. Is the fix going to be a part of RC2 release?

[spencergibb]

That is the milestone and project assigned.

[liuwish]

@spencergibb This problem still occurs with Finchley.v2.0.0.RELEASE. I find that tcp four wave is abnormal when client close websocket through network packet.

[bzl969]

Hi @spencergibb ,
We are trying to use Spring Cloud Gateway as an edge-proxy to our Websocket endpoints, but seemingly the problem mentioned in this thread is still in place.
If I just simply close the Websocket connection right after opened it, it closes without any issue.
But if the client sends at least one message and tries to close after that, the connection seemingly will remain open, but neither the client nor the server will be able to send any more messages on that.
I've tried even with 2.1.0.M1 version and still the same.
Can we expect a fix on this in the near future?

[michaelpaesold]

For us it looks like using 2.1.1 SNAPSHOT (version 2.1.1.BUILD-SNAPSHOT of the parent, Greenwich.BUILD-SNAPSHOT) fixed the problem that websocket connections are not properly closed when the client sends a close message.

Any previous released version (e.g. 2.1.0.RELEASE) did not work.

[bolinzhang]

there is still this problem with Finchley.v2.0.0.RELEASE.

[spencergibb]

@bolinzhang
Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to a new issue as a zip file.

If so please open a new issue.

[xurui8691413]

spring-cloud-gateway-websocket-master.zip

Please run application and run GatewayTest can reproduce this problem ,and then run GatewayTest1 can close websocket connection because GatewayTest1 doesn't access through gateway

[xurui8691413]

I think this issue is urgent that blocked us.

[maniek45]

This issue is also blocking us. I've also sent how to reproduce this issue in https://github.com/spring-cloud/spring-cloud-gateway/issues/480.

[spencergibb]

Please stop commenting on multiple issues

[sparkcai]

I hava the same problem
my partion pom.xml

,there is my heap dump infomation, because can't close ws connections, there haven a memory leak。
gateway server info

websocket server info