Ingress-nginx: Setting nginx 'proxy-buffer-size' to 16k causes nginx config error

[whereisaaron]

TL;DR: Increasing nginx proxy_buffers_size with the proxy-buffer-size setting needs to be coupled with a suitable increase in the proxy_buffers buffer pool.

@vyshane added a useful patch that adds a proxy-buffer-size ConfigMap setting that sets the proxy_buffer_size in the nginx config. This causes a problem though, because proxy_buffer_size is tightly coupled to proxy_buffers; You can't make the proxy_buffer_size larger if you can't also increase the size of the proxy_buffers buffer pool. E.g. currently if you set proxy-buffer-size to 16k, the nginx controller will fail with:

[emerg] 19#19: "proxy_busy_buffers_size" must be less than the size of all "proxy_buffers" minus one buffer in /tmp/nginx-cfg612772959:2175

This is because the default proxy_buffers setting is 4 x 4k buffers = 16k total, and you need at least one buffer left over for buffering in the other direction. Increasing proxy_buffers_size needs to be coupled with a matching increase in the buffer pool.

(Note that the proxy_buffers pool is still used, evening when proxy_buffering is set to off. This is because 'proxy_buffering' relates only to buffering to disk. Yeah. Thanks nginx 😃)

You could either add user control of the the proxy_buffers setting or you apply proxy-buffer-size to both settings, e.g. if proxy-buffer-size is set to 16k, the nginx config could be:

proxy_buffer_size                       16k;
proxy_buffers                           4 16k;

or, if you believe more buffers that are page-sized and page-aligned are more efficient:

proxy_buffer_size                       16k;
proxy_buffers                           16 4k;

@aledbf had a proposal too:

@whereisaaron please open an issue with ^^ and we will calculate proxy_busy_buffers_size using the proxy_buffers size (like the server hash tables)

[whereisaaron]

I created PR #364 with the patch I have been using for both 0.8.3 and 0.9.0-beta.2. It increases the size of the proxy_buffers (memory allocations) to match the size of the proxy buffer. This leaves the default values (with no ingress configmap settings) unchanged:

proxy_buffer_size     4k
proxy_buffers         4 4k

If 'proxy-buffer-size' is set then, with this patch, both the buffer size and the memory allocation size is increased (or decreased) in lock step, providing a compatible setting that also maintains the ratio of the two related settings:

proxy_buffer_size     "{{ $location.Proxy.BufferSize }}";
proxy_buffers         4 "{{ $location.Proxy.BufferSize }}";

[aledbf]

Closing. Fixed in #364

[rubo77]

Great post!

Important to know, if you want to fix the error :

proxy_buffer_size 4096 is not enough for cache key