Caddy: Server Sent Events don't work

[rsdy]

1. What version of Caddy are you running (caddy -version)?

0.9.4

2. What are you trying to do?

Proxy SSE through Caddy

3. What is your entire Caddyfile?

:80 
gzip {
    not /application/events/
}
maxrequestbody 200MB

log stdout
errors stderr

root html/

proxy /application 127.0.0.1:9000 {
    transparent
    header_upstream X-Real-IP {remote}
    header_upstream X-Forwarded-Server {host}
    header_upstream X-Forwarded-Host {host}
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy

5. What did you expect to see?

I expected my SSE requests to establish a long-running connection that can send data to the client.

6. What did you see instead (give full error messages and/or log)?

The HTTP request does seem to work (returns a 200, and headers), and the connection is kept alive, but SSE frames don't arrive to the client.

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

I've tried the https://github.com/jraedisch/go_sse_example to confirm the bug, but streaming _does_ work with that setup. I suspect this error may be specific to interacting with the Java upstream I'm using. nginx, however, works well in the place of caddy, without additional changes to the upstream config. I'm using a python requests script as reference client.

[mholt]

Can you please build from source (go get -u github.com/mholt/caddy && $GOPATH/bin/caddy) and try again? We've made some changes recently that may resolve this.

[rsdy]

I have just built master as you suggested, and it doesn't resolve my issue.

[mholt]

@rsdy Are you sure? People are saying in https://github.com/mholt/caddy/issues/677 that they work. Try with an older version perhaps? What is it about your scenario that causes this, what can we do to reproduce it? You say even that's only one upstream that causes it, what is it doing differently?

[rsdy]

I can try and mock up a minimal example with Dropwizard + Jersey SSE to demonstrate/test this in the next few days. There's no intentional magic in how the application handles SSE, so I assume the difference is coming from the framework. For full reference, here's the nginx config that does work:

worker_processes  1;
user nginx nginx;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    client_max_body_size 200M;
    sendfile        on;
    keepalive_timeout 600s;
    send_timeout 600s;

    gzip  off;

    upstream backend {
        keepalive 100;
        server 127.0.0.1:9000;
    }

    server {
        listen       80;
        server_name  localhost;

        location / {
            root   /var/www/html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }

        location /application/ {
            proxy_read_timeout 24h;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;

            proxy_http_version 1.1;
            proxy_set_header Connection "";

            proxy_pass http://backend;
        }
    }
}

I've tried tweaking the proxy keepalive parameter and the Connection header, but to no avail.

[rsdy]

https://github.com/rsdy/caddy-dropwizard-sse is a project that demonstrates this issue with master. You'll need a working JDK8+maven, and caddy to run it.

[tw4452852]

Hi @rsdy , I have found the reason. There is a mistake in your CaddyFile, change this line

not /application/events/

to

not /application/events

no tail slash!

Also in your https://github.com/rsdy/caddy-dropwizard-sse project, you should disable gzip for /application/events in your CaddyFile

[rsdy]

Thanks @tw4452852 for looking into this. Having made the modifications you recommended, I still can't see traffic coming through caddy. I'm trying the HEAD on master, using both Chromium and curl. Directly calling the Java server reliably works.

Even if I disable gzip completely, the SSE call just hangs through caddy.

[tw4452852]

Weird, Do you use the https://github.com/rsdy/caddy-dropwizard-sse environment? Could you please take a snapshot of response header in Chromium?

[rsdy]

~/w/caddy-dropwizard-sse$ curl -D - localhost:8000/application/events
HTTP/1.1 200 OK
Content-Type: text/event-stream
Date: Thu, 19 Jan 2017 10:43:47 GMT
Server: Caddy
Transfer-Encoding: chunked

^C
~/w/caddy-dropwizard-sse$ curl -D - localhost:9000/application/events
HTTP/1.1 200 OK
Date: Thu, 19 Jan 2017 10:43:54 GMT
Content-Type: text/event-stream
Transfer-Encoding: chunked

event: config-change
id: 1484822634858
data: 1484822634858

event: config-change
id: 1484822635860
data: 1484822635860

Same in Chromium:

[tw4452852]

Everything seems right. Is there any logs after you started start.sh? What's go version? The Head of master you mean is ecf852e?

[rsdy]

$ go version
go version go1.7.3 linux/amd64

I'm using Alpine Linux as a build box, and ecf852e.

There is nothing special popping up after running start.sh. The only thing I'm just noticing now is that I caddy leaves the connections open even if kill the client.

1. Run curl localhost:9000/application/events & pid=$! sleep 5 && kill $pid will print a few events and then log a Java exception indicating the connection is closed, as expected.

2. Run curl localhost:8000/application/events & pid=$! sleep 5 && kill $pid, and no Java exception is logged. Looking netstat -pn will list a TCP connection between caddy and the java process is still ESTABLISHED, and will only close after several minutes. Only then is the Java exception logged to the console.

[rsdy]

I updated https://github.com/rsdy/caddy-dropwizard-sse with a Dockerfile and a caddy executable to help debugging. You'll need to swap the caddy executable and rebuild if you want to test a change.

[tw4452852]

Oh, they run with docker, I suspect this is same as #1281. Does it work w/o docker?

[rsdy]

No, it doesn't.

On n19 Jan 2017, at 12:32, Tw [e-mail] wrote:

Oh, they run with docker, I suspect this is same as #1281. Does it work w/o docker?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[tw4452852]

After some investigation, the reason is found. It's actually a bug in go 1.7...

Firstly, this issue doesn't happen if you build caddy with go tip which I use, however it happens if you use go 1.7 which @rsdy uses.

The bug is that the chunk reader will expect two more bytes which is \r\n after reading a chunk before it returns. So it will read until the offered buffer is full (which is 32K in our case, so big...). Some more details are in this issue.

So @rsdy , you could use the newer go (at least contains this commit) to rebuild the caddy, this issue should be fixed then.

[mholt]

Woah, great find @tw4452852!

I haven't tried working on this, but I'm convinced by looking at that, and I trust it works for Tw because he says it does. ;) @rsdy, you're welcome to try it with Go tip (1.8 is in RC2; stable will be coming out in a few weeks) and if it still happens -- make sure you verify you are using Caddy built on Go 1.8 -- then you can re-open this with more details. Thanks!

[tw4452852]

@mholt You're welcome.

Shall we add go version information in the issue template if the reporter build Caddy himself?

[rsdy]

I can confirm the bug doesn't happen with Go 1.8rc2. Thanks @tw4452852 & @mholt for chasing this!

[laszbalo]

@tw4452852

Also in your https://github.com/rsdy/caddy-dropwizard-sse project, you should disable gzip for /application/events in your CaddyFile

Clould you please explain why disabling gzip is required?

[tw4452852]

@laszbalo
I think it needs some works on server side to implement sse with gzip.