Caddy: Unable to setup reverse-proxy for weechat websocket

[joelnb]

Following up on a forum post about something which seems like a possible issue with caddy.

On that thread I suggested it was related to headers sent by Caddy which could not be removed but I am now not convinced that is the issue after further testing. Should I create a separate issue for the fact that those headers cannot be removed? As it could possibly do with being added to the docs anyway.

1. What version of Caddy are you running (caddy -version)?

Caddy 0.9.4

2. What are you trying to do?

Setup a reverse-proxy for weechat equivalent to this nginx config:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}
upstream weechat {
    server 127.0.0.1:8001;
}

server {
    listen       9000 ssl;
    server_name  _;

    ssl_certificate ...;
    ssl_certificate_key ...;

    location /weechat {
        proxy_pass http://weechat;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 8h;
    }
}

3. What is your entire Caddyfile?

irc.something.com:9000 {
    proxy /weechat 127.0.0.1:8001 {
        websocket
    }

    log stdout
    errors stdout
}

And also:

irc.something.com:9000 {
    proxy /weechat 127.0.0.1:8001 {
        header_upstream -Accept-Encoding
        header_upstream -User-Agent
        header_upstream -X-Forwarded-For

        websocket
    }

    log stdout
    errors stdout
}

4. How did you run Caddy (give the full command and describe the execution environment)?

I'm running using the systemd service file:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target
; Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-failure
StartLimitInterval=86400
StartLimitBurst=5

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy

; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp [email protected]
ExecReload=/bin/kill -USR1 $MAINPID

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=64

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy

; The following additional security directives only work with systemd v229 or later.
; They further retrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
;AmbientCapabilities=CAP_NET_BIND_SERVICE
;NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

I don't think this is important though as I have also tried running without the service and it didn't change the results.

5. What did you expect to see?

The websocket connection was correctly established to the weechat backend.

6. What did you see instead (give full error messages and/or log)?

I see a 400 Bad Request error from Weechat. This log shows the request/response from the app I am using to weechat:

23:48 --> [1] weechat GET /weechat HTTP/1.1
23:48 Host: 127.0.0.1:8001
23:48 User-Agent: Go-http-client/1.1
23:48 Connection: Upgrade
23:48 Sec-Websocket-Key: 57Ss+Zf4hO5yIr6yx5AumA==
23:48 Sec-Websocket-Version: 13
23:48 Upgrade: websocket
23:48 X-Forwarded-For: xxx.xxx.xxx.xxx
23:48 Accept-Encoding: gzip
23:48
23:48 <-- [1] weechat HTTP/1.1 400 Bad Request

Here is the connection request made through the nginx reverse-proxy:

23:33 --> [1] weechat GET /weechat HTTP/1.1
23:33 Upgrade: websocket
23:33 Connection: upgrade
23:33 Host: weechat
23:33 Sec-WebSocket-Key: YG3fFz5tp38lwOsxz45/Yw==
23:33 Sec-WebSocket-Version: 13
23:33
23:33 <-- [1] weechat HTTP/1.1 101 Switching Protocols
23:33 Upgrade: websocket
23:33 Connection: Upgrade
23:33 Sec-WebSocket-Accept: 3ZqCrujw+6rssCmfO+aOH4zczFU=

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

You can run weechat in a docker image using the following commands:

mkdir /tmp/.weechat
chmod 700 /tmp/.weechat
docker run -v /tmp/.weechat:/home/guest/.weechat -t -i -p 8001:8001 fstab/weechat-otr

Alternately if you are not a docker user you can install weechat according to the instructions for your distribution: https://weechat.org/files/doc/stable/weechat_user.en.html#install

Once you have a working weechat setup the relay with these commands (typed at the window which opens when you start weechat):

/relay add weechat 8001
/set relay.network.password yourpassword
/relay raw

This takes you to a window where you should see all requests received by weechat.

Then start Caddy using one of the examples Caddyfiles from above. You can then try and connect by entering the details into a weechat client (e.g. Glowing Bear if running on something accessible remotelyor possibly weechat android if running totally inside your own network).

It should also be possible to demonstrate the issue using curl rather than setting up a weechat client - this request goes direct to the weechat server (and gets a 101 Switching Protocolos response):

$ curl --include \
     --no-buffer \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
    localhost:8001/weechat
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=

This request goes through the caddy server and fails with the error mentioned above:

curl --include \
     --no-buffer \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
    https://irc.something.com:9000/weechat
HTTP/1.1 400 Bad Request
Server: Caddy
Date: Sat, 07 Jan 2017 13:13:45 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8

If the same request is made when using the NGINX config above (rather than caddy). Response from nginx:

HTTP/1.1 101 Switching Protocols
Server: nginx/1.6.2
Date: Sat, 07 Jan 2017 13:17:28 GMT
Connection: upgrade
Upgrade: websocket
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=

[tw4452852]

The culprit is Sec-Websocket-* vs Sec-WebSocket-*. I have just worked out a possible fix for it.

[ghost]

Did you also look into the host header sent by caddy?

[tw4452852]

@tehscorpion The host header comes from here.

[ghost]

@tw4452852 that just caught my eye because he is using 2 different upstream hosts in his configs.

[mholt]

@joelnb This turns out to be a bug in weechat, and it looks like they're fixing it (yay) -- see the linked issue above. Thanks for the report!

[joelnb]

Thanks @mholt looks like they have sorted it now as well so I can start using only Caddy when the next version of weechat lands.

[mholt]

@joelnb Could you test it against the latest weechat before they release it? Just let us/them know if it doesn't work. ;) Thanks!

[joelnb]

@mholt Thanks for your help with this. I tested the fix on the weechat end and it seems to work perfectly with a normal build of Caddy. I've let them know on the issue you raised with them.

[mholt]

Awesome, thank you Joel!