Firebase-tools: Use zopfli instead of zlib for improved gzip compression on upload

[davidmurdoch]

I've forked firebase-tools to do this myselfhere: https://github.com/davidmurdoch/firebase-tools/commit/0afb5969cab16f23355c636da18838413178876c

Zopfli improves overall file size savings beyond what gzip at level 9 can do at the expense of extra CPU cycles at compression time.

In my branch I just default to using zopfli compression. But it could easily opt-in.

I'm also running with a relatively high number of zopfli iterations that may not be suitable for users with large file sizes. Defaulting to a lower number of iterations or just checking the uncompressed size of the file in order to choose an acceptable number of iterations may be a better choice.

[mbleigh]

Thanks for the suggestion! This is something we could consider for the future, but there's a lot to unpack -- we use the content hash of the gzipped file to uniquely identify files, so any change to the gzip compression is going to mean losing the advantages of "change-only" uploads for everyone who switches to the new version.

If the savings are significant, it's still worth considering, but it's not something we can just drop in.

[davidmurdoch]

Sounds like opt-in would be the way to go then for something like this. The savings are generally pretty minimal (3-8% according to https://developers.googleblog.com/2013/02/compress-data-more-densely-with-zopfli.html), especially considering the upfront CPU cost.

Another potential issue with zopfli is that it uses node-gyp, which can be problematic on Windows builds.

Implementing Brotli in the future would provide much more savings (for browsers that support it), but there would have to be a change to the Firebase back-end to handle it (it used to be possible to serve brotli encoded text-based files from firebase hosting, but a recent serverside change broke this behavior).

[mbleigh]

Brotli support is something that's on our radar, but no timeline to announce. node-gyp is definitely a big downside to this, as we've seen tons of problems with it in the past (functions emulation used to depend on grpc, which is a node-gyp library. nothing but pain).

[samtstern]

@Memeriaj this would be a server-side feature right? If so should we close this issue here on firebase-tools since there's nothing that will be done in this repo to address it?

[davidmurdoch]

gzipping is currently a client-side feature (or it was when I filed this issue).

[samtstern]

@davidmurdoch oops, I'll show myself out.

[matthewharwood]

@mbleigh any update on brotli support?

[davidmurdoch]

@matthewharwood They used to support it by setting a Content-Encoding header
in your firebase.json after brotli encoding the files yourself in a build processes before calling firebase deploy. They broke this back in August and don't plan on fixing it.

They did say back in August that we plan to support brotli in the future (automatically) and only serve the smallest content when possible but I didn't tell me a timeline for when.

[matthewharwood]

@davidmurdoch yeah heard something like this on Twitter too. Long shot but do you know any cdn that offers brotli without crazy lambda file rewrites on orgin?

[davidmurdoch]

Cloudflare will do it and it's free. https://support.cloudflare.com/hc/en-us/articles/200168396-What-will-Cloudflare-compress-

[kmcnellis]

Brotli is available by default on static hosting content now: https://firebase.googleblog.com/2020/08/firebase-hosting-new-features.html

[matthewharwood]

@kmcnellis Thank you so much for tagging thihs issue. Firebase has always been my favorite static site hosting and now I can finally go back!