Caddy: Caddy sent `Connection: close` header to upstream when proxying websocket

[syndtr]

Apparently caddy sent both Connection: Upgrade and Connection: close headers to upstream. This behavior can cause issue on some _upstream webserver_ that doesn't tolerate this.

Header generated by browser:

GET ws://localhost:5050/:/websockets/notifications HTTP/1.1
Host: localhost:5050
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Origin: http://localhost:5050
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: _ga=GA1.1.262888070.1471697271
Sec-WebSocket-Key: R8cA3wDt1Dp+gz/vFQFlSg==
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits

Header generated by caddy:

GET /:/websockets/notifications HTTP/1.1
Host: localhost:5050
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cache-Control: no-cache
Connection: Upgrade
Cookie: _ga=GA1.1.262888070.1471697271
Origin: http://localhost:5050
Pragma: no-cache
Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits
Sec-Websocket-Key: R8cA3wDt1Dp+gz/vFQFlSg==
Sec-Websocket-Version: 13
Upgrade: websocket
X-Forwarded-For: ::1
X-Forwarded-Proto: http
X-Real-Ip: ::1
Connection: close

This seems related to #886.

1. What version of Caddy are you running (caddy -version)?

Caddy 0.9.1 (+1d3212a Sun Aug 21 02:37:23 UTC 2016)

2. What are you trying to do?

Proxying websocket.

3. What is your entire Caddyfile?

:5050 {
    redir 302 {
        if {uri} is /
        if {>Referer} not_has /web/
        if_op and
        / /web/
    }
    gzip {
        level 1
        not /:/websockets
    }
    proxy /:/websockets 10.0.4.15:32400 {
        transparent
        websocket
        max_fails 0
    }
    proxy / 10.0.4.15:32400 {
        transparent
        max_fails 0
    }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

./caddy

5. What did you expect to see?

Websocket proxied correctly.

6. What did you see instead (give full error messages and/or log)?

HTTP/1.1 400 Bad Request

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

Inspect header generated by caddy. This does seems to happen persistently from my observation.

[LukeHandle]

Ahh, was about to file this one as well - looks like another person using Plex as well ;)

If it helps with the report, I replicated by performing a tcpdump during a websocket proxy.

I used a WebSocket echo server on :5180 from another comment

This test server doesn't care about the extra connection: close header, but you can see the header added in which causes issues on other backends

When the WebSocket works, the curl must be exited with the Ctrl-C (^C)

$ /usr/local/bin/caddy19 -log stdout -port 8080 "proxy /echo 127.0.0.1:5180/ { websocket }"

$ curl -Nik 'http://172.16.10.10:8080/echo' -H 'Origin: https://caddy:8080' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: <key>' -H 'Upgrade: websocket' -H 'Connection: Upgrade'
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: <accept>

^C

tcpdump of the request that Caddy makes:

16:12:10.557888 IP localhost.33074 > localhost.5180: [...]
GET /echo HTTP/1.1
Host: 127.0.0.1:5180
User-Agent: curl/7.29.0
Accept: */*
Connection: Upgrade
Origin: https://caddy:8080
Sec-Websocket-Key: <key>
Sec-Websocket-Version: 13
Upgrade: websocket
X-Forwarded-For: <localip>
Accept-Encoding: gzip
Connection: close

The extra header causes issues against Plex and against EdgeOS (Ubiquiti). Plex returns a 502 via Caddy and EdgeOS gave a 400 for it's websocket.

$ curl -Nik 'https://EdgeOSDevice/ws/stats' -H 'Origin: https://EdgeOSDevice' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: <key>' -H 'Upgrade: websocket' -H 'Connection: Upgrade'
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: <accept>

^C

$ curl -Nik 'https://EdgeOSDevice/ws/stats' -H 'Origin: https://EdgeOSDevice' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: <key>' -H 'Upgrade: websocket' -H 'Connection: Upgrade' -H 'Connection: close'
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: <accept>

curl: (52) Empty reply from server



$ curl -Nik 'http://Plex:32400/:/websockets/notifications' -H 'Origin: http://Plex:32400' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: <key>' -H 'Upgrade: websocket' -H 'Connection: Upgrade'
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: <accept>
X-Plex-Protocol: 1.0
Cache-Control: no-cache
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Location

^C

$ curl -Nik 'http://Plex:32400/:/websockets/notifications' -H 'Origin: http://Plex:32400' -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: <key>' -H 'Upgrade: websocket' -H 'Connection: Upgrade'  -H 'Connection: close'
curl: (52) Empty reply from server

[mholt]

Hmm, I also see the same thing with the WS backend in my tests, however when I add a line in Caddy just before it calls RoundTrip() which prints the value of outreq.Header["Connection"], I see this output:

outreq.Header["Connection"]: [Upgrade]

So it is not also passing a value of "close" upstream. I'm not sure where it is coming from...

[tw4452852]

@mholt I think that header is from here and we set it here. Is it expected?

[mholt]

@tw4452852 Woah, nice find. Not sure. @nemothekid - do you know if that was intended for hijacked connections?

Edit: Permalink to referenced std lib code: https://github.com/golang/go/blob/550caa1c87ea11aa54c6482ff66f98b4036c474f/src/net/http/transport.go#L1806

[nemosupremo]

I intended to set DisableKeepAlives (to prevent the underlying Transport from doing funny things to the connection so we can reliably hijack the connection), but I didn't realize go was also setting Connection: Close. Looking into a fix...

[nemosupremo]

1062 should fix the issue, thanks @tw4452852 for the narrowing down the issue, made the fix really easy.

[lbguilherme]

@mholt Please make a release with this change. 0.9.1 is completely broken for proxying websockets with this bug.

[mholt]

@lbguilherme Done :smile: See version 0.9.2.

[Nixtren]

Hey guys,

Would this change have any impact on HTTP reverse proxy connections? I upgraded from 0.9.1 to 0.9.2 and pages got messed up. For example, I request page "/blog" and I get served page "/about" instead. Looks like requests are getting mixed. This happened a few seconds later after starting Caddy 0.9.2 on a (very) busy server. I couldn't replicate the same on a test server with the exact same configs, looks like some stress is needed to make the bug appear. I rolled back to Caddy 0.9.1 and the issue stopped. This is either a bug of Caddy 0.9.2 or one of it's plugins. For reference, I'm using plugins git+ipfilter+realip+ratelimit.

The configuration I'm using is the following:
https://forum.caddyserver.com/t/has-anyone-paired-varnish-with-caddy/554/5?u=nixtren

[mholt]

@Nixtren Can you open a new issue (refer to this one if you think it's related) with more details, following the issue template? Thanks, this way we won't forget about it.

[nemosupremo]

@Nixtren This fix that was introduced by this PR would not have affected your setup as this PR only affects websockets.

[yingshaoxo]

I've been spending a whole night to solve this problem when I start to use https or wss or ssl. It always says connection stopped before establish with 400 error code.

Just a minutes ago, I found a solution for that:

0. Cloudflare

At the SSL/TLS tab:

• If you have your own cert or SSL or HTTPS: set it to Full. (The following 123 steps assume you have your own https certification)

• If you only have an http server: set it to Flexible. (The Cloudflare will add https or ssl to your website automatically.)

• After that, go to DNS tab, set Proxied.

If you are not sure what you are doing, just go to DNS tab, set DNS only

1. Make sure you have a right proxy configuration.

ai-tools-online.xyz {
    proxy / 127.0.0.1:5000 {
        except /socket.io
        transparent
    }

    proxy /socket.io 127.0.0.1:5000 { 
        websocket 
        transparent
    } 
}
import sites/*

or

ai-tools-online.xyz {
    proxy / 127.0.0.1:5000 {
        except /socket.io
    }
    proxy /socket.io 127.0.0.1:5000 { 
        header_upstream Host {host} 
        header_upstream X-Real-IP {remote} 
        header_upstream X-Forwarded-For {remote} 
        websocket 
    } 
}
import sites/*

ai-tools-online.xyz is your domain, http://127.0.0.1:5000 is your socket server.

2. Make sure your server Cross-Origin Controls is set to '*' to allow Cross-Origin Access

For flask-socketio, is to use flask_socketio.SocketIO(app, cors_allowed_origins = '*')

3. You must restart the caddy to let the new config work

I don't know how to restart caddy, but reboot may do that job

4. For more details about how to set nginx, see the following links:

https://github.com/yingshaoxo/Web-Math-Chat#reverse-proxy-configuration-for-https
https://www.nginx.com/blog/nginx-nodejs-websockets-socketio/
https://caddy.community/t/using-caddy-0-9-1-with-socket-io-and-flask-socket-io/508/6