Caddy: How to proxy Server Sent Events?

[kidwm]

I used a nodejs backend to offer event source, but failed to access it through Caddy proxy.

Is it possible to do this way in Caddy server?

[aek]

I'm wondering the same for a Python Odoo module and Chrome App integration. If helps in Nginx I solve it with this directives in the location:

    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding off;
    proxy_buffering off;
    proxy_cache off;

Maybe the websocket presets in the proxy directive it's doing the same??

[dmitshur]

In order to support server sent events going through an httputil.ReverseProxy, I had to do one thing, set a non-zero value for FlushInterval.

See example here.

[mholt]

@shurcooL Ha! I remember when you showed me that code when you first wrote it. I was actually looking at that struct field but wasn't quite sure if it was the culprit, and I haven't had the time to investigate it right now.

Would somebody please try it in Caddy for server-sent events? Here is the value to set: https://github.com/mholt/caddy/blob/f1ba7fa34353682b15a2e15d10b15413d60a6c12/middleware/proxy/reverseproxy.go#L47 - try 1 * time.Second.

[Luit]

It surprises me that your Nginx configuration turns chunked-encoding _off_ @aek, because I'd expect it to be necessary (or at least preferential) for SSE. If you'd want this to work in Caddy, I think you'd need to convince it to flush data as soon as it comes in from the destination connection. I think the parts of the above Nginx config that make it work there are proxy_buffering off; and proxy_cache off;. To be sure I'd have to test this first though.

As far as I can tell, the FlushInterval [1] in middleware/proxy is never set, so the proxy directive might need this as a new parameter to make this work.

[1] https://github.com/mholt/caddy/blob/f4c729bd2205b939f93fad068bbbd7161802f695/middleware/proxy/reverseproxy.go#L47

[Luit]

ha, didn't refresh this page in a while, it seems I'm late to the party...

[Luit]

I think you could set it here, just to make sure setting the value fixes it: https://github.com/mholt/caddy/blob/f4c729bd2205b939f93fad068bbbd7161802f695/middleware/proxy/reverseproxy.go#L102

[mholt]

Aye, that should do it! ^ If somebody who has a SSE backend would like to set that value and report how it goes, that would be fantastic. If that works, then we can make plans to implement a fix in Caddy.

[larskluge]

I tried to add a 1 second flush interval with
rp := &ReverseProxy{Director: director, FlushInterval: 1 * time.Second}
here: https://github.com/mholt/caddy/blob/master/middleware/proxy/reverseproxy.go#L114
as suggested, but unfortunately with my SSE backend it does not work. The request just hangs and never completes—same as before the change.

[mholt]

@larskluge Are you sure you were running the modified binary? Use the build script build.bash then ./ecaddy -version should confirm that it is in fact Caddy with your changes.

[larskluge]

I built w/ go build and made sure to use the correct bin. Rebuild it now with the build script, unfortunately the same result—just hangs.

./ecaddy -version
Caddy 0.8.2 (+c21ff83 Fri Apr 08 18:36:34 UTC 2016)
 1 file changed, 1 insertion(+), 1 deletion(-)
middleware/proxy/reverseproxy.go

[mholt]

Interesting; is there any more details about the upstream you can give us to reproduce the issue? Maybe compare to @shurcooL's setup...

[larskluge]

Perhaps try my config, Caddyfile:

:2345 {
  gzip
  log stderr
  header / {
    Access-Control-Allow-Origin *
  }
  proxy / 104.236.207.89:31438 {
    proxy_header Host {host}
    proxy_header X-Real-IP {remote}
    proxy_header X-Forwarded-Proto {scheme}
  }
}

This is a temp deploy, the address won't work for too long. Anyway, compare calling it directly:

curl -i http://104.236.207.89:31438/larskluge/s3

vs. running the above config locally:

curl -i http://localhost:2345/larskluge/s3

[mholt]

@larskluge Thanks for the details. I was able to see the same behavior. The culprit is this line -- in other words, dst is not a writeFlusher type.

Notice that if you remove the log directive from your Caddyfile, SSE works. This is because the middleware.ResponseRecorder type is not also an http.Flusher.

It's really late here, so I'm gonna sleep on it; somebody is welcome to fix this before I get around to it if they want to. :smile:

[mholt]

K, nevermind, I just took care of it... why let it linger.

@larskluge and @kidwm and @aek please try the latest changes and let me know how it goes!

[larskluge]

Awesome, works perfectly well—thank you @mholt !!

[gyulavoros]

I've run into the same issue using the latest version of Caddy (0.9.3). I try to run a YouTrack 7.0 install behind Caddy and it seems SSE doesn't work. I tried to play with FlushInterval but with no luck so far. I'm not really familiar with Go and the internals of Caddy, so I put a few Printlns into the code to see what's happening, but can't figure out what goes wrong. If someone can help me out with this, here are the steps to reproduce the issue:

1. Run YouTrack

docker run --rm -p 8080:8080 -e BASE_URL=http://localhost:9000 wearemakery/youtrack:7.0.28110

1. Create Caddyfile:

localhost:9000 { proxy / localhost:8080 { transparent } }

1. Open localhost:9000, setup YouTrack
2. Go to Agile Boards, create a new project with e.g. a Scrum board and wait a few seconds. (SSE only used by Agile Boards)
3. The following warning displays

The live update connection is taking longer than expected. Please refer to the YouTrack reverse proxy configuration and verify that you have disabled buffering for connections that support live updates.

YouTrack has an official proxy guide for e.g. Nginx where they recommend to use the following options:

proxy_cache off;
proxy_buffering off;
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
proxy_set_header Connection '';
chunked_transfer_encoding off;

If you can point me in any directions I would be more than happy to help as well.

[gyulavoros]

@mholt May I open a new issue for this or it would be more practical to reopen this one?

[mholt]

You can open a new issue, thanks. :)