Caddy: Websocket fails with rancher

[SoreGums]

1. What version of Caddy are you running (caddy -version)?

0.8.2

2. What are you trying to do?

proxy https://github.com/rancher/rancher

3. What is your entire Caddyfile?

#points to 127.0.0.1, using startssl cert to avoid prompts and dealing with LetsEncrypt
n1.xps15.plus1.io:443 {
  tls n1.xps15.plus1.io_bundle.crt n1.xps15.plus1.io.key
  proxy / http://192.168.99.100:8999 {
    proxy_header Host {host}
    proxy_header X-Forwarded-Proto {scheme}
    websocket
  }
}

n1.xps15.plus1.io:80 {
  redir https://{host}{uri}
}

4. How did you run Caddy (give the full command and describe the execution environment)?

./caddy (Caddyfile in current dir with caddy)

5. What did you expect to see?

I expected the backend app to "just work" like it does when proxied by nginx

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    upstream rancher {
        server 192.168.99.100:8999;
    }

    server {
        listen 443 ssl;
        server_name n1.xps15.plus1.io;
        ssl_certificate n1.xps15.plus1.io_bundle.crt;
        ssl_certificate_key n1.xps15.plus1.io.key;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Port $server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://rancher;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            # This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
            proxy_read_timeout 900s;
        }
    }

    server {
        listen 80;
        server_name n1.xps15.plus1.io;
        return 301 https://$server_name$request_uri;
    }

}

6. What did you see instead (give full error messages and/or log)?

for some reason websocket headers are not returned - chrome errors log in the console saying header missing. the rancher backend complains about the websocket connection in the logs.

Attempting to view the differences between what nginx sends to the backend and what caddy sends via wireshark proved useless as it was not exactly clear what to look for.

End of the day the situation is, Rancher works fine with nothing proxying it. Rancher works fine with nginx proxying it. Rancher does NOT work with caddy proxing it with the above config. I am not sure if the issue is caddy or rancher as testing with websocketd shows caddy does the correct thing :/

running with --http2=false made no difference

https://gist.github.com/SoreGums/15ccb9f5de5a171bec6b - extra info where I used websocketd and it worked fine

[hemsleyk]

Can confirm that I was having this same issue when trying out Rancher and 0.8.2

[edrex]

See discussion on #594 (muddled). Proxy is also not working with endpoints provided by Caddy's websocket directive. Might be the same cause.

[mholt]

@edrex When you say "websocket directive", are you referring to the websocket directive? Or the websocket preset in the proxy middleware?

[edrex]

@mholt I made a test case and couldn't reproduce the problem, see https://github.com/mholt/caddy/issues/594#issuecomment-199485675. Now it appears proxying to the Caddy websocket directive works fine.

[edrex]

@SoreGums I've also been unable to get some real-world websocket endpoints to work with the proxy directive (specifically https://github.com/magne4000/quassel-webserver and http://cockpit-project.org).

[5an1ty]

Same issue for me when using rancher:
Error during WebSocket handshake: 'Upgrade' header is missing.

To replicate try this caddyfile:

https://sub.domain {
    proxy / rancherip:8080 {
        websocket
        proxy_header Host {host}
        proxy_header X-Real-IP {remote}
        proxy_header X-Forwarded-Proto {scheme}
    }
    tls email
    errors stdout
}

And to set up a rancher container to test:
sudo docker run -d --restart=always -p 8080:8080 rancher/server

Errors should show up in the console on any page of the rancher application saying:
Error during WebSocket handshake: 'Upgrade' header is missing.

The thing is this does work fine with a fairly standard nginx config as already shown in the above posts. So this must be a caddy issue right?

[5an1ty]

I'm onto something...

for more details on the subject:
https://github.com/rancher/rancher/issues/4278

Caddy seems to drop the response header!
Upgrade: WebSocket
Websockets that work just fine with caddy have this in the response:
Upgrade: websocket
This is all via tcp dump if I look in the client chrome for the websocket that isn't working. The upgrade header isn't there...

This probably is a case sensitive thing. (WebSocket vs websocket)

[hemsleyk]

^ Fixed by https://github.com/mholt/caddy/pull/713/files

Thanks

[hemsleyk]

Re-opening because this is back in 0.9.

The following Caddyfile does not work w/ abiosoft/caddy:0.9.0 (websockets return 200):

rancher.mydomain.com {
    proxy / rancher:8080 {
        header_upstream Host {host}
        header_upstream X-Real-IP {remote}
        header_upstream X-Forwarded-Proto {scheme}
        websocket
    }
    tls myemail
    errors stderr
}

The following Caddyfile DOES work with abiosoft/caddy:0.8.3

rancher.mydomain.com {
    proxy / rancher:8080 {
        proxy_header Host {host}
        proxy_header X-Real-IP {remote}
        proxy_header X-Forwarded-Proto {scheme}
        websocket
    }
    tls myemail
    errors stderr
}

[mholt]

@hemsleyk I think this is probably fixed on master; can you build from source and confirm if the problem is still present?

[cking]

I just did a go get ... and replaced the executable from getcaddy with the newly built one.
Proxied websockets work again like expected.

[hemsleyk]

Thanks for the second opinion - had an (ongoing) work situation that made this difficult to find time at home for testing.

[marcbachmann]

My rancher ends up in a 101 Switching Protocols loop when I use abiosoft/caddy:0.9.1. abiosoft/caddy:0.8.3 works for me.

So it looks like there's still something wrong.

[mholt]

Anyone else experiencing that? ^

[abiosoft]

oh really 😳 ?

[abiosoft]

@marcbachmann does 0.9.0 work ?

[clangley]

@mholt I am currently experiencing this loop issue

Version: Caddy 0.9.1

Config:

<DOMAIN>
tls off
proxy / localhost:8080 {
  header_upstream Host {host}
  header_upstream X-Real-IP {remote}
  header_upstream X-Forwarded-For {remote}
  header_upstream X-Forwarded-Proto {scheme}
  websocket
}

edit: 0.8.3 works perfectly

[mholt]

@clangley @marcbachmann Please open a new issue and file a bug report so we can take a look at this. (Also, are you sure you want to proxy ALL requests as websockets?)

[InAnimaTe]

I know this is closed but I don't see a new issue on it, so I'll post here.

Wanted to mention that with 0.9.1, we were also trying to proxy Rancher Server with SSL. This seemed to work fine for normal http connections but deployed hosts who tried opening websocket connections back to us would not get through.

Trying 0.9.2 yesterday, this issue seems to have disappeared. Here is the config we are successfully using:

rancher-master.example.net {
    proxy / localhost:14443 {              
        transparent                        
        websocket                          
    }                                      
    tls {                                  
        max_certs 1                        
    }                                      
}                                          

So, afaik, any problems with Rancher behind Caddy (at least using SSL) seem to be gone....