Kong: [request] GZip outbound compression

[sonicaghi]

Compressing each (or selected) requests with gzip allows to drastically reduce the amount of bandwidth used over the network.

example: https://segment.com/blog/lifecycle-of-a-mobile-message

[lucamaraschi]

:+1:

[subnetmarco]

nginx natively supports gzip, so this shouldn't be hard to activate. gzip may also be already enabled by default (and properly handling the Accept-Encoding header).

This shouldn't be a plugin, but a default feature of kong.

[sonicaghi]

@lucamaraschi thoughts?

[lucamaraschi]

I agree on having it in the core and just a simple "decorator" of an api. But can we fine grain it to location/endpoint level?

[montanaflynn]

The nginx gzip module has many options for controlling what and how gzip is applied to responses.

Personally I only gzip certain mime types and even then you have to think about wether or not gzip is actually worth the reward. For instance gzipping a string like {"error": "something blew up"} will not result in a much lower size and could in fact slow down the proxy time due to having to gzip it before sending it out. For this reason I think it should be a plugin so that you can enable it for only certain APIs, endpoints, and other methods of applying plugins.

[tyiss]

@montanaflynn, i also there not much gain gzipping small responses.
therefore i suggest adding the "gzip_min_length" directive so that it could be set.
as you suggested, like the shared dict size seem right to me.

I'm not sure its best do wrap that in a plugin, it is usually a _client_ decision, if the client support read with gzip it will ask for gzipped content, that is why i don't see much benefit using it for just certain APIs.

[montanaflynn]

Adding that directive would be a good a idea, but there are a lot more options as well. As I noted in the PR https://github.com/Mashape/kong/pull/481 there is also the issue of which mime type to apply gzipping to which would be a necassary option to include in the kong.yml.

Personally I would prefer to have this type of functionality be configured through a plugin as there are many available options. Having them in kong.yml would add more initial configuration, effect every request and response, and not be able to be changed programatically.

I agree that whatever is chosen Kong should respect the Accept-Encoding header just as Nginx does but as a lightweight high-performance proxy I don't think Kong should do it for every request unless told to.

[subnetmarco]

The trick with this plugin will be to beat the native nginx gzip performance with an OpenResty module.

[Diwann]

I have a similar issue: when I make a request with the "Accept-Encoding: gzip" header, my answer is empty ...
And gzip on the nginx server seems already activated as it already answer "Accept-Encoding: gzip" in the response header

[daddykotex]

We have the same problem as @Diwann here.

We use kong as a proxy and while the service behind it returns a compressed body correctly, when the response go through kong, the body is discarded. All headers are kept though, so tools like cURL and Postman complain that the Content-Length does not match the body.

Here is the output of cURL:

* upload completely sent off: 345 out of 345 bytes
< HTTP/1.1 201 Created
< Date: Thu, 09 Jun 2016 17:07:44 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 270
< Connection: close
< Server: spray-can/1.3.3
< Content-Encoding: gzip
< Location: /blabla/id
< X-Kong-Upstream-Latency: 39
< X-Kong-Proxy-Latency: 0
< Via: kong/0.8.2
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization, Content-Range, Content-Disposition, Content-Description, X-Custom-Header, oauthio
< Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, PATCH
< Access-Control-Allow-Credentials: true
<
* transfer closed with 270 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 270 bytes remaining to read

[subnetmarco]

@daddykotex what plugins have you installed?

[sebastienc]

We've got TCP logging, JWT. We did quite a bit of debugging to try and have this work via nginx configuration but it broke a lua plugin of which I can't remember the details. I think we also tried to use the transformers plugins.

[rca]

In the meantime, to add gzip compression to my requests, I updated the http block in /usr/local/share/lua/5.1/kong/templates/nginx.lua to look like this:

http {
    gzip on;
    gzip_disable "msie6";
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;

    # do not add text/html to gzip_types because it's added by default and will generate
    # nginx: [warn] duplicate MIME type "text/html"
    # http://stackoverflow.com/a/6475493/703144
    gzip_types application/json application/x-javascript application/xml application/xml+rss text/css text/javascript text/plain text/xml;

    include 'nginx-kong.conf';
}

I don't have any transformation plugins enabled and this seems to be working just fine.

Thanks.

[thibaultcha]

@rca Instead of updating the template itself, you should really consider doing so in your own nginx configuration, by copying your current template as-is (without the Lua part), and starting Kong with it: kong start -c kong.conf --nginx-conf nginx.conf. See https://getkong.org/docs/0.9.x/configuration/#custom-nginx-configuration