Caddy: Proxy server pushes

[VanCoding]

Currently, it seems that caddy does not proxy server pushes.
So if my app relies on server push, I cannot put it behind a caddy-proxy, because the browser won't get the pushes that way.

Will you implement this in the future?

[wendigo]

You can use Preload links and Caddy will push resources to client event if using proxy.

[VanCoding]

That won't work for all applications.

I'm currently implementing a web push server, that implements the web push protocol.
The protocol makes heavy use of server push.

For example, to listen for push notifications, the listener makes a never-ending request to the server, and then the server pushes notifications via server push over that request.

[mholt]

We could look into this, but I don't know how to do it right now. I think we're going to implement auto-push first, but then maybe @wendigo and/or I could look into this?

[nilslice]

@mholt, @VanCoding -

I had a similar question on twitter, and would love to see Caddy support this so more of my project's users could put their Ponzu apps behind Caddy and still leverage all of h2.

My problem sounds similar to @VanCoding's in which typical server push usage does not apply. Rather than using server push for resources like CSS/JS/etc to make a "page load" faster, Ponzu provides a Push mechanism to basically use the network for n+1 queries. As mentioned in the tweet, an example of this can be illustrated below:

Say you have 2 data types, Song and Artist, where each song has an artist field, referencing the Artist type:

type Song struct {
    Title   string `json:"title"`
    Year    int    `json:"year"`
    Artwork string `json:"artwork"`
    Artist  string `json:"artist"`
}

type Artist struct {
    Title   string   `json:"title"`
    Logo    string   `json:"logo"`
    Members []string `json:"members"`
}

In a Ponzu CMS app, the Pushable interface allows the programmer to define which fields from the receiver struct to server push, as long as they are URLs.. (the Artist field in the Song struct would be a URL to the JSON representation of the Artist, i.e. ?type=Artist&id=42)

So, by adding the Push method to Song, like below, a request for a Song would also send a server push for the embedded Artist endpoint too. Now the artist information for that song is available to the client without making an additional roundtrip:

func (s *Song) Push() []string {
    return []string{
        "artist", // this being the json tag for the Artist field in Song struct
    }
}

e.g. /api/content?type=Song&id=21

{
"data": [{
    "title": "Where's the Revolution",
    "artist": "?type=Artist&id=42",
    "year": 2017,
    "artwork": "/api/uploads/2017/02/single-cover.png",
 }]
}

would send a push promise for the data[0].artist node as per the Push method to send down the JSON at ?type=Artist&id=42

{
"data": [{
    "title": "Depeche Mode",
    "logo": "/api/uploads/2017/01/depeche-logo.png",
    "members": ["Dave Gahan", "Martin Gore", "Andrew Fletcher"]
 }]
}

Currently, Caddy will not proxy the my explicit h2 server push to the client which I believe is what @VanCoding is also trying to achieve, versus the implicit server pushes done through an "auto-push" mechanism, or through the ~Preload~ Link headers which would require direct access to the ResponseWriter.

This certainly could be done, and may be achieved with some of Caddy's outstanding PRs for 1.8 HTTP/2 stuff.

@VanCoding - does this sound at all like the higher level of flexibility from server push you're looking for?

[VanCoding]

@nilslice yes, exactly.

As I understood it, with Go 1.7, this would not have been possible, but Go 1.8 does provide the features needed to implement this. So this seems to be the right time to ask for this :P

[wendigo]

As far as I know reverse proxy (https://golang.org/pkg/net/http/httputil/) does not support http push handling as of go 1.8. I don't know if it can be achieved straight away.

[mholt]

Seems this issue would be your best bet: https://github.com/golang/go/issues/18594

[wendigo]

Exactly @mholt. I think that best option for now is to leverage Link headers so instead of backend pushing resources - if it's running behind the proxy it should emit Link headers that will be handled by Caddy.

[nilslice]

@wendigo - that does seem to be the only option until go's h2 client and reverse proxy support reading the push_promise frames.

[mholt]

Closing for now, but labeled as deferred so we won't forget about it.

[xfalcox]

So @mholt currently caddy won't work with push in reverse proxy mode using either config or Link headers right?

[mholt]

If a backend responds with Link headers, Caddy will push those resources, but currently it won't send push frames from the backend.

[xfalcox]

If a backend responds with Link headers, Caddy will push those resources

I modified Discourse to add the Link headers but I don't get any pushes.

My test bed is the main resource at https://forum.overwatchbrasil.org/

[nilslice]

@xfalcox, I don't have an answer for you at the moment, but thought I'd share findings in case someone else in this thread could pick up. Here is an example Link header in a response from the server behind the link you shared:

link:</assets/ember_jquery-60dedd1dca1d8b8ce48b6d0087db3b96f3149b9b5af2d0c7c6357967d29031fb.js>; rel=preload; as=script,</assets/locales/pt_BR-8401d46807446030e20eedd74421b9de4a6affb6ccb6a1e6c17ebee18db3b070.js>; rel=preload; as=script,</assets/preload-store-9a5d6688476c54b0eb6e11cf444ab9dc19602f99f10114228f41fe605c31ac4d.js>; rel=preload; as=script,</assets/vendor-52ed1f7ea87a692ca7ec8f3b75bb76997f730cae151edfe8f467849ef9ddde78.js>; rel=preload; as=script,</assets/pretty-text-bundle-04ac5b2eae6ead3ce5d5994ccd1e264320fb14970bb0dd812e392773769b9e7f.js>; rel=preload; as=script,</assets/application-948438e84d8c167a85c9956a04607d4594120ca272c4908d67483e618e7a0303.js>; rel=preload; as=script,</assets/plugin-7373dcf40422d8d18b41a284662de686332690da6d4a136c29812e7312fcc268.js>; rel=preload; as=script,</assets/plugin-third-party-0f321a002dca51d38eac23960537a0213ce3077d7ecd462c580a1b0e050bc8c0.js>; rel=preload; as=script,</assets/admin-682daf361f129b9362a9575cd8ea1b3ce30f50444b46b62b35b0c5924eec2b5b.js>; rel=preload; as=script,</assets/browser-update-f57286e74ddbc53aa899689b01ef467078911e4138050c561939955849af35dd.js>; rel=preload; as=script,</stylesheets/desktop_27a3b29bee7eade3b9b036e5f1e5227565cd82f1.css?__ws=forum.overwatchbrasil.org>; rel=preload; as=style,</stylesheets/admin_27a3b29bee7eade3b9b036e5f1e5227565cd82f1.css?__ws=forum.overwatchbrasil.org>; rel=preload; as=style,</site_customizations/7e202ef2-56d7-47d5-98d8-a9c8d15e57dd.css?target=desktop&v=f3a81517563b77eb3476b9c214eaf5fc&__ws=forum.overwatchbrasil.org>; rel=preload; as=style

I suppose the first question is if these are in the correct syntax/format?

Also, the server header, though easily overwritten/spoofed, is "nginx", not "Caddy" as commonly left in Caddy configurations. Are you definitely serving the site you linked from Caddy?

[xfalcox]

I suppose the first question is if these are in the correct syntax/format?

I do believe they are :thinking:

Also, the server header, though easily overwritten/spoofed, is "nginx", not "Caddy" as commonly left in Caddy configurations. Are you definitely serving the site you linked from Caddy?

Yes I am, nginx is listening a sock and Caddy is the owner of 80 and 443 port.

[xfalcox]

@wendigo You said:

Exactly @mholt. I think that best option for now is to leverage Link headers so instead of backend pushing resources - if it's running behind the proxy it should emit Link headers that will be handled by Caddy.

Does that work on latest master, or is a planned feature? I added all the Link headers on my response, but Caddy doesn't push.

[mholt]

@xfalcox I missed your reply; but that feature is released now; you can use Link headers upstream to initiate pushes downstream.