Iris: [DOCUMENTATION] No Content-Encoding: gzip header on gzip-ed bindata

Created on 10 Jul 2020  ·  28Comments  ·  Source: kataras/iris

Describe the bug
I use kataras/bindata to embeding some files, but iris reponse without Content-Encoding: gzip header

To Reproduce

go get -u github.com/kataras/bindata/cmd/bindata
bindata ./public/...
go build

    app.HandleDir("/static", "./public", iris.DirOptions{
        Asset:      GzipAsset,
        AssetInfo:  GzipAssetInfo,
        AssetNames: GzipAssetNames,
    })

The public dir has only one file index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>NMac Mirror</title>
</head>
<body>
Hello!
</body>
</html>

Screenshots
image

Desktop
MacOS 10.14.6

resolved documentation
Source
picture xiaozhuai

All 28 comments

Hello @xiaozhuai, your bug report is correct however it is explicitly written in the kataras/bindata that the caller should be responsible to add the encoding headers. you can fix it with the AssetValidator option as shown below:

    app.HandleDir("/static", "./public", iris.DirOptions{
        Asset:      GzipAsset,
        AssetInfo:  GzipAssetInfo,
        AssetNames: GzipAssetNames,
        AssetValidator: func(ctx iris.Context, name string) bool {
            ctx.Header("Content-Encoding", "gzip")
            ctx.Header("Vary", "Accept-Encoding")
            return true
        },
    })

There is nothing more we can do there, the GzipAsset or Asset functions are not providing any information about the algorithm is used to compress the data, so we can't know. The kataras/bindata specifies that you must set the headers by yourself.

There is also no way to "extract" the content-encoding by reading the first "x" bytes of the data (as far as I know) so you must use the AssetValidator solution, which is proper and probably the most performant solution for this one. (a TODO for me would be to add this to the examples).

However, the upcoming Iris v12.2.0, which lives on master branch, has a solution for all major compression algorithms (gzip, deflate, brotli, snappy) not just gzip, there is an example:

https://github.com/kataras/iris/blob/c94da3b994d711936600e7b903e6738c97dd93aa/_examples/file-server/file-server/main.go#L56-L75

kataras picture kataras on 10 Jul 2020

It works now! One more thing, for small file, it's better to not gzip. Gzipped size may be more greater. It's there any solution?
image

xiaozhuai picture xiaozhuai on 11 Jul 2020

Yes, for small files the result file size may be larger, as official gzip documentation specifies. This is why gzip is not enabled by default into Iris, you have to select when it is worth and when it doesn't. You can use brotli or snappy instead, but these are only available on the master branch currently (go get -u github.com/kataras/iris/v12@master in your project's directory). See the example code I attached to you above and test it by yourself. Write down your results!

kataras picture kataras on 11 Jul 2020

Yes, this is just an edge case.
I will try snappy.
And thanks for your kind help!

xiaozhuai picture xiaozhuai on 11 Jul 2020

You are welcome @xiaozhuai , I've also changed the title and the labels to describe the issue with more details, if that's OK with you. Let me know if you find anything useful we can add on _examples!

kataras picture kataras on 11 Jul 2020

I am a newbie of golang and iris.
I did some research on golang web framework, and iris is the one I like best!
I will see what I can do to contribute to this project.
Thanks!

xiaozhuai picture xiaozhuai on 11 Jul 2020
❤1

That's awesome, thanks for the feedback @xiaozhuai. You will love golang and Iris as you learn more and more!

kataras picture kataras on 11 Jul 2020

Yes, this is just an edge case.
I will try snappy.
And thanks for your kind help!

Hello @xiaozhuai again, two things:

  • Do NOT compress files smaller than 1.4KB (files smaller than size are still transmitted as one 1500-bytes TCP packet)
  • Do NOT compress image and pdf files (they are already compressed by nature)

I am working to improve the HandleDir as we speak, thinking of adding and implementing two new fields for that matter:

type DirOptions {
        // [other options...]

    // If Compress is set to true then this value
    // is compared against the file size to decide
    // if it should be eventually compressed or not.
    // A good value is 1.4*iris.KB, so smaller files are not compressed and waste CPU.
    //
    // Defaults to 0, it compress everything (when Compress is true).
    CompressMinSize int64
    // CompressImages enables compression over
    // already compressed file types too:
    // .pdf, .jpg, .jpeg, .gif, .png, .tif, .tiff (.svg and .bmp are not compressed).
    //
    // Defaults to false, keep it that way.
    CompressImages bool
}

Sounds good to you?

EDIT:

I also found a way to automatically add the Content-Encoding to gzip when using kataras/bindata.GzipAsset, so there is not need to add that AssetValidator we talk about previously. There is actually a way to know if the data are gzip-compressed: https://stackoverflow.com/a/6059342

kataras picture kataras on 20 Jul 2020

@kataras Yes! That's awesome! I think this feature should be add to kataras/bindata, it should provided an option of min size and ignored file types, then provide a function to tell whether a file is compressed, so that iris can serve files with the right header.

xiaozhuai picture xiaozhuai on 20 Jul 2020

Yes I was thinking about it but, if we read the first two bytes of each file to see if it's gzip-compressed or not, is still a performance cost that I'm not willing to give it unless a new option set by user. I don't know if it's a good idea, why not just keep the requirement of writing one line inside the AssetValidator when using GzipAsset instead of Asset?

Also, in bindata it's a good idea to add it as an argument option in terminal.

I am putting those two fields on Iris because they can be quite useful on go-bindata/go-bindata where data are not gzip-compressed already (it's more popular/more developers using it than kataras/bindata) too. Also, don't forget, you can use compression all over your Iris handlers, not just using HandleDir, e.g. ctx.ServeFile/Content and e.t.c.

then provide a function to tell whether a file is compressed, so that iris can serve files with the right header.

Iris doesn't know if the asset functions are came from kataras/bindata or go-bindata or anything else, it just accepts the function types, so this is not possible, indeed Iris is not even aware of kataras/bindata or go-bindata packages at all.

kataras picture kataras on 20 Jul 2020

@kataras I have a question, whether iris cache compressed data of files in memory when use go-bindata? If the answer is yes, we can use go-bindata and abandon kataras/bindata. If so, we can provide any number of compression types. And that won't cause performance lose.

xiaozhuai picture xiaozhuai on 20 Jul 2020

@xiaozhuai iris does not cache anything, the files served through Asset, AssetNames and AssetInfo functions are presented as go source code of []byte, they are already in-memory; compression happens on the fly, no it does not cache compressed data from go-bindata that's why I created the kataras/bindata at the first place, kataras/bindata exports the contents as gzip-compressed []byte.

However, your idea is not bad at all, we could scan all files and cache their gzip-bytes them before server ran (as we do on kataras/bindata, after all they wont change, they are embedded. <-- One problem here, the compression algorithm is decided on the request-time based on the client's Accept-Encoding, so some clients require gzip some others may snappy or br(brotli).

kataras picture kataras on 20 Jul 2020

@kataras Yes, kataras/bindata provided compressed data so it don't need compress during runtime. But if we cache compress data in memory, then we can use go-bindata and have the same performance like kataras/bindata. And we can provided not only the gzip encoding. It's just a suggestion. : )

xiaozhuai picture xiaozhuai on 20 Jul 2020
👍1

We can provide cached data for all supported compression algorithms when Compress == true && Asset != nil && AssetInfo !=nil && AssetNames !=nil -> Cache data through a map[alg:string].... on request -> cache[compression requested] ..... however this will result on larger memory.

kataras picture kataras on 20 Jul 2020

OK, I will see what I can do, performance is everything here, we can do that. There will be probably options like above (for common file system, not embedded) plus ForceCompressAssets: bool or AssetCache: bool or AssetPrecache: bool or AssetCompressionCache: bool

kataras picture kataras on 20 Jul 2020

@kataras Yes, that's a problem. Maybe we can provided an option that cache should be created on application initlized or the file was request first time, and provide a strategy to eliminate caches that not hit for a long time. Sorry for my poor English, hope that helps. : )

xiaozhuai picture xiaozhuai on 20 Jul 2020

No, we can do that on build-time no serve-time (it will require locks and hassle while encoding ALL files without reason). There are just 4 builtin web compression algorithms (gzip, (deflate -> this is compatible with gzip as I can remember, so three caches), brotli, snappy).

Your english are fine, probably better than mines, so don't worry about it :)

kataras picture kataras on 20 Jul 2020

@kataras Thank you very much! Thanks for your idea and sharing it to me!

xiaozhuai picture xiaozhuai on 20 Jul 2020
❤1

Sure, I'll keep you informed about the progress @xiaozhuai. I thank YOU!

kataras picture kataras on 20 Jul 2020

@kataras If you don’t mind, I’m willing to buy you some coffee. : )
Thank you for bring us iris.
image

xiaozhuai picture xiaozhuai on 20 Jul 2020

Woww... I have no words... I am very touched 🥺 thank you @xiaozhuai this is very generous of you. I'm dedicated and loyal to Iris and all of you, thanks again

kataras picture kataras on 20 Jul 2020

OK @xiaozhuai, it's almost ready to cache and compress any http.FileSystem (http.Dir (system files), and go-bindata). The go-bindata, on its latest release some months ago, has an -fs argument which generates an AssetFile() http.FileSystem function (Iris had its own version of it for views and files for years). As an extra bonus, I already implemented simple verbose to check how much each compression will help:

# cache, but, don't compress files of: .pdf, .jpg, .jpeg, .gif, .png, .tif, .tiff (already compressed by nature)
# compress files larger or equal than 1500 bytes
# with sample assets directory
$ go run .
/js/main.js.map (33 B)
/vendor/bootstrap/css/bootstrap-grid.css.map (158.9 KB)
gzip    (31.5 KB)
deflate (31.5 KB)
br      (23.6 KB)
snappy  (47.7 KB)
/vendor/bootstrap/js/bootstrap.min.js.map (191.5 KB)
br      (44.4 KB)
snappy  (77.0 KB)
gzip    (51.4 KB)
deflate (51.4 KB)
/css/main.css (41 B)
/vendor/bootstrap/js/bootstrap.bundle.min.js (81.1 KB)
deflate (23.3 KB)
br      (20.7 KB)
snappy  (34.8 KB)
gzip    (23.3 KB)
/vendor/bootstrap/css/bootstrap-grid.css (68.0 KB)
deflate (9.3 KB)
br      (5.6 KB)
snappy  (15.1 KB)
gzip    (9.3 KB)
/app2/app2app3/index.html (312 B)
/vendor/bootstrap/css/bootstrap.min.css.map (646.4 KB)
deflate (120.0 KB)
br      (75.1 KB)
snappy  (183.2 KB)
gzip    (120.0 KB)
/vendor/bootstrap/js/bootstrap.bundle.js.map (409.1 KB)
gzip    (100.7 KB)
deflate (100.7 KB)
br      (85.4 KB)
snappy  (157.0 KB)
/vendor/bootstrap/js/bootstrap.min.js (60.2 KB)
gzip    (16.2 KB)
deflate (16.2 KB)
br      (14.1 KB)
snappy  (23.4 KB)
/big_files/6KB.parts (6.3 KB)
gzip    (172 B)
deflate (154 B)
br      (117 B)
snappy  (451 B)
/app2/index.html (25 B)
/vendor/bootstrap/js/bootstrap.js.map (253.9 KB)
gzip    (63.8 KB)
deflate (63.7 KB)
br      (53.8 KB)
snappy  (98.9 KB)
/vendor/bootstrap/css/bootstrap.css.map (508.2 KB)
gzip    (112.7 KB)
deflate (112.7 KB)
br      (88.4 KB)
snappy  (170.6 KB)
/vendor/bootstrap/css/bootstrap-grid.min.css.map (115.8 KB)
gzip    (17.5 KB)
deflate (17.5 KB)
br      (12.2 KB)
snappy  (27.5 KB)
/app2/app2app3/dirs/text.txt (27 B)
/favicon.ico (15.1 KB)
snappy  (5.8 KB)
gzip    (4.1 KB)
deflate (4.1 KB)
br      (3.8 KB)
/vendor/bootstrap/css/5kb.css (4.7 KB)
gzip    (1.8 KB)
deflate (1.7 KB)
br      (1.5 KB)
snappy  (2.5 KB)
/vendor/bootstrap/css/bootstrap-reboot.min.css.map (32.3 KB)
gzip    (8.5 KB)
deflate (8.5 KB)
br      (7.9 KB)
snappy  (12.5 KB)
/vendor/bootstrap/css/bootstrap-reboot.css.map (77.3 KB)
gzip    (18.9 KB)
deflate (18.9 KB)
br      (16.8 KB)
snappy  (27.6 KB)
/big_files/Advanced_API_Security_OAuth_2.0_and_Beyond_2nd_edition.epub (8.9 MB)
gzip    (8.9 MB)
deflate (8.9 MB)
br      (8.8 MB)
snappy  (8.9 MB)
/vendor/bootstrap/css/bootstrap-grid.min.css (51.0 KB)
gzip    (7.8 KB)
deflate (7.8 KB)
br      (4.8 KB)
snappy  (13.1 KB)
/vendor/bootstrap/css/4kb.css (3.9 KB)
snappy  (2.2 KB)
gzip    (1.6 KB)
deflate (1.6 KB)
br      (1.4 KB)
/index.html (469 B)
/app2/mydir/text.txt (12 B)
/vendor/bootstrap/css/bootstrap.css (198.3 KB)
gzip    (31.1 KB)
deflate (31.1 KB)
br      (22.8 KB)
snappy  (48.2 KB)
/vendor/bootstrap/js/bootstrap.js (136.3 KB)
snappy  (43.5 KB)
gzip    (29.0 KB)
deflate (28.9 KB)
br      (23.7 KB)
/big_files/yarn-1.22.4.msi (1.6 MB)
deflate (1.4 MB)
br      (1.3 MB)
snappy  (1.4 MB)
gzip    (1.4 MB)
/js/main.js (87 B)
/vendor/bootstrap/css/bootstrap.min.css (160.4 KB)
br      (20.9 KB)
snappy  (42.5 KB)
gzip    (27.7 KB)
deflate (27.6 KB)
/vendor/bootstrap/js/bootstrap.bundle.min.js.map (315.3 KB)
deflate (90.0 KB)
br      (77.6 KB)
snappy  (135.2 KB)
gzip    (90.0 KB)
/vendor/bootstrap/js/bootstrap.bundle.js (229.2 KB)
br      (44.8 KB)
snappy  (80.2 KB)
gzip    (54.0 KB)
deflate (54.0 KB)
/app2/app2app3/css/main.css (38 B)
/app2/app2app3/dirs/dir2/text.txt (32 B)
/big_files/123kb.zip (126.5 KB)
gzip    (108.9 KB)
deflate (108.9 KB)
br      (36.1 KB)
snappy  (99.5 KB)
Time to complete compression and cache of [25/36] files multiple by [4] algorithms: 1.0430372s
Reduce total size from 14.4 MB to:
deflate (11.2 MB) [-28.76%]
br      (10.8 MB) [-32.72%]
snappy  (11.6 MB) [-23.59%]
gzip    (11.2 MB) [-28.75%]

Stay tuned!

kataras picture kataras on 22 Jul 2020

@kataras brotli seems great. Waiting for your further good news!

xiaozhuai picture xiaozhuai on 22 Jul 2020

@xiaozhuai I think we are ready with: https://github.com/kataras/iris/commit/d259324f7671e769b94af06cae3a91dcd03ab929. Example: https://github.com/kataras/iris/blob/d259324f7671e769b94af06cae3a91dcd03ab929/_examples/file-server/embedding-gzipped-files-into-app/main.go#L7-L41

All examples have been updated. Please send a feedback if that's the expected result you searching for or not

kataras picture kataras on 24 Jul 2020

@kataras Sorry for the late reply, I am so happy that you make progress so quickly! I will try this new feature.

xiaozhuai picture xiaozhuai on 28 Jul 2020
❤1

How it goes @xiaozhuai ?

kataras picture kataras on 10 Aug 2020

@kataras Sorry, I am so busy these days. And haven't try this so far. I will try it tomorrow.

xiaozhuai picture xiaozhuai on 10 Aug 2020

Sure! No worries, contact me if I can help you at any way

kataras picture kataras on 10 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rhcarvalho picture rhcarvalho  ·  4Comments
stefanwuthrich picture stefanwuthrich  ·  4Comments
rebootcode picture rebootcode  ·  3Comments
Dexus picture Dexus  ·  4Comments
caimingyi484 picture caimingyi484  ·  4Comments