Caddy: gzip compression_level default

[nodesocket]

The gzip directivecompression_level by default is 9. I'm fairly certain any value greater than 5 for compression_level rarely results in smaller files. Essentially the majority of the time after level 5 is just wasted cpu cycles for nearly no smaller output. NGINX by default uses a level of 1. I propose changing the default to level 5 which should be nearly fully optimized for size without wasting extra cpu cycles.

[mholt]

The hypothesis seems sound. Anyone want to run a quick experiment or find a reference of already-done experiments about compression level?

I do think you're right, though, the compression factor might flatten out quite a bit after 5 or 6.

[nodesocket]

@mholt so I found the following via https://serverfault.com/questions/253074/what-is-the-best-nginx-compression-gzip-levelO. Basically seems about right using a value of 5.

application/x-javascript - jQuery 1.8.3 (Uncompressed):

0    261.46 KiB (100.00% of original size)
1     95.01 KiB ( 36.34% of original size)
2     90.60 KiB ( 34.65% of original size)
3     87.16 KiB ( 33.36% of original size)
4     81.89 KiB ( 31.32% of original size)
5     79.33 KiB ( 30.34% of original size)
6     78.04 KiB ( 29.85% of original size)
7     77.85 KiB ( 29.78% of original size)
8     77.74 KiB ( 29.73% of original size)
9     77.75 KiB ( 29.74% of original size)

[mholt]

LGTM. Let's make it so.

[nodesocket]

Happy to make my first PR to Caddy 👍 . Where is this default compression_level located? I searched through https://github.com/mholt/caddy/blob/master/caddyhttp/gzip/gzip.go but did not see the default set here.

[cez81]

@nodsocket https://github.com/mholt/caddy/blob/master/caddyhttp/gzip/setup.go#L149

[centminmod]

@mholt I thought this was fixed ages ago in Caddy? I have been defaulting Caddy vhosts I create with gzip level compression = 5 since I found out Caddy was using level 9 though so haven't noticed in my testing.

But you don't want to be using level 9 due to compression times and resource usage. I did some compression algorithm benchmarks a while back and you can see how gzip level 1-9 fair https://community.centminmod.com/threads/compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.12764/

[nodesocket]

@cez81 where is gzip.DefaultCompression coming from in https://github.com/mholt/caddy/blob/master/caddyhttp/gzip/setup.go#L149? Is this a built in from the gzip lib? Can we just pass 5 here instead?

Where are gzip.BestSpeed and gzip.BestCompression coming from?

Thanks.

[cez81]

https://golang.org/pkg/compress/gzip/

[tobya]

@nodesocket I think perhaps if you make a const set to 5 and assign that as the default it should be acceptable. You could place it in the gzip/setup.go file

[crvv]

The default level is set at
https://github.com/mholt/caddy/blob/c4dfbb9956095c92d0586a52723748c070c7b459/caddyhttp/gzip/setup.go#L163

And the default level is 6, which is set at
https://github.com/golang/go/blob/07c01e396894cc2b6d2528fe6b00e910d24a9384/src/compress/flate/deflate.go#L595

5 and 6 is very close so I don't think changing is necessary.

[midzer]

I would love to implement this one for caddy! My firsts in GO :)

[nodesocket]

@crvv if the default is really 6, then the documentation is wrong as it states the default settings is 9.

See https://caddyserver.com/docs/gzip

[midzer]

@crvv thanks for your links. As far as i understand GO code, i think Caddy initializes default with level 9 compression:

var i int
[...]
for i = gzip.BestSpeed; i <= gzip.BestCompression; i++ {
        writerPool[i] = newWriterPool(i)
}
// According to golang docs "i" has a value of "9" now
defaultWriterPoolIndex = i
writerPool[defaultWriterPoolIndex] = newWriterPool(gzip.DefaultCompression)
^why add again? it's already done in previous loop, DefaultCompression ^= -1 in golang gzip docs

[crvv]

According to golang doc "i" has a value of "9" now

gzip.BestCompression is 9, but here i has a value of 10.(https://play.golang.org/p/zcNbZhqkR5)
The last line is writerPool[10] = newWriterPool(gzip.DefaultCompression)

defaultWriterPoolIndex(10) is a global variable, and it is used at

https://github.com/mholt/caddy/blob/c4dfbb9956095c92d0586a52723748c070c7b459/caddyhttp/gzip/setup.go#L167

[midzer]

Thanks for clarification and example @crvv

We initialize all possible writers from BestSpeed (1) to BestCompression (9). Now we can pick the requested one out of them. For default compression defaultWriterPoolIndex could be set to 5 or 6 (depending on what we want as default). Why add another "default" instance additionally?

[crvv]

In Go standard library, gzip.DefaultCompression is -1. The actual value is implementation dependent.
If Caddy want to use whatever Go defines as default, adding another "default" is necessary.

However, in documentation of GNU Gzip, the default compression level is defined as 6 explicitly.
I think assuming 6 as standard default value is acceptable.

[tobya]

Thanks guys for all your investigations, I'm a little confused as to what the 'actual' value is being set to? Can you do a simple

fmt.Println(gzip.DefaultCompression)

to find the actual value if we are not 100% sure? If it is 5 or 6 we can close this issue.

[djhworld]

@tobya

In summary:

• fmt.Println(gzip.DefaultCompression) will print -1

In https://github.com/golang/go/blob/07c01e396894cc2b6d2528fe6b00e910d24a9384/src/compress/flate/deflate.go#L595 there is a switch statement that determines the level

    case level == DefaultCompression:
        level = 6
                fallthrough

So the default is 6

[nodesocket]

@djhworld ok great, so then updating the documentation from saying the default compression level is 9 to 6 should be completed right?

[tobya]

When I fmt.Println the actual compression level, I only get an output value of 10 unless I set the level myself like this

site.com 
gzip {
 level 5
 }

I think the default config.level should be set about here if someone wants to do that.

[tw4452852]

As go standard lib already determines the default level which is good enough, and we follow it. Why should we set another default value? Anyway, if you really want, you could set it explicitly with gzip { level x } .

[tobya]

the problem is I dont think it is set to 6 in code I think it is set to 10

[nodesocket]

Surely there must be a way to debug this and prove the defaut?

[crvv]

@tobya
10 is NOT compression level.
10 is defaultWriterPoolIndex.

writerPool[defaultWriterPoolIndex] = newWriterPool(gzip.DefaultCompression)

And the default compression level of the default writer is gzip.DefaultCompression

[djhworld]

I've been having a look at the code, and agree with @crvv

What happens is the caddy gzip module creates a map of writers, indexed with keys 1 - 9

e.g.

map[1: writerWithLevel(1), 2: writerWithLevel(2), 3: writerWithLevel(3)...

Then at the end it creates a new key called 10 with writer configured with the level gzip.DefaultCompression and stores this key (10) in a variable called defaultWriterPoolIndex

https://github.com/mholt/caddy/blob/baf6db5b570e36ea2fee30d50f879255a5895370/caddyhttp/gzip/setup.go#L144

If you don't specify a level for the gzip compression, the variable level will default to 0

When caddy attempts to get a gzip writer, it passes a the configured level to

https://github.com/mholt/caddy/blob/baf6db5b570e36ea2fee30d50f879255a5895370/caddyhttp/gzip/setup.go#L168

Which checks if the level is between 1-9,

If it isn't it then uses the variable defaultWriterPoolIndex (which is set to 10) which contains a writer that passes the level as -1, which in turn defaults to 6 in the gzip standard library

[djhworld]

I guess from a casual observation it could be confusing as the keys in the writerPool map don't necessarily reflect the compression level being set.

1: writerWithLevel(1)
2: writerWithLevel(2)
3: writerWithLevel(3)
4: writerWithLevel(4)
5: writerWithLevel(5)
6: writerWithLevel(6)
7: writerWithLevel(7)
8: writerWithLevel(8)
9: writerWithLevel(9)
10: writerWithLevel(-1)

I think there could be some slight confusion if a user purposefully configures level 0 (for whatever reason), because this would return a writer with the default compression of 6 - which might not be desired behavior?

[tobya]

Ok. That makes more sense now. We can get an update pushed to the docs.

[mholt]

So am I right to understand that all we have to do is update the docs to say the default level is 6, which is the Go standard library's default?

I've just done that locally and they will go out with the next site update. If I misunderstand, we can re-open the issue.

Thanks!