Ingress-nginx: Upgrade from 0.27.1 to 0.28.0 failed: could not build variables_hash, you should increase variables_hash_bucket_size: 128

[juliohm1978]

NGINX Ingress controller version: 0.28.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: baremetal
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):

Linux k8s-master01-dev-20190607.dis.tjpr.jus.br 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: Kubespray / helm

What happened:

Just tried upgrading from 0.27.1 to 0.28.0 and this error pops up:

E0206 17:33:32.923953       6 controller.go:149] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2020/02/06 17:33:32 [emerg] 43#43: could not build variables_hash, you should increase variables_hash_bucket_size: 128
nginx: [emerg] could not build variables_hash, you should increase variables_hash_bucket_size: 128
nginx: configuration file /tmp/nginx-cfg855255757 test failed

-------------------------------------------------------------------------------

What you expected to happen:

I expect 0.28.0 to work with the same workload and configurations as 0.27.0.

How to reproduce it:

I haven't been abe to pinpoint the cause of the error message. So, I'm not sure how to reproduce this. It happened in one of our clusters with a larger workload, close to 100 deployments and statefulsets.

I tried using the value variables-hash-bucket-size: "128" in the ConfigMap, but that did not make any difference.

/kind bug

[juliohm1978]

Following up. Inside the container, the value for variables_hash_bucket_size seems to be correctly placed in /tmp/nginx-cfg470948177, the temporary config used before replacing the actual /etc/nginx/nginx.conf.

        variables_hash_bucket_size      128;
        variables_hash_max_size         2048;

If I change the ConfigMap to variables-hash-bucket-size: "64", the error message changes to

nginx: [emerg] could not build variables_hash, you should increase variables_hash_bucket_size: 64

[aledbf]

@juliohm1978 I think this is related to https://github.com/kubernetes/ingress-nginx/pull/4993

You can use quay.io/kubernetes-ingress-controller/nginx-ingress-controller-amd64:prometheus to test the fix.

[juliohm1978]

Unfortunately, that version also did not work. Same error.

E0207 01:28:58.128061       7 controller.go:149] Unexpected failure reloading the backend:                                                                                                 │
-------------------------------------------------------------------------------                                                                                                            │
Error: exit status 1                                                                                                                                                                       │
2020/02/07 01:28:58 [emerg] 45#45: could not build variables_hash, you should increase variables_hash_bucket_size: 128                                                                     │
nginx: [emerg] could not build variables_hash, you should increase variables_hash_bucket_size: 128                                                                                         │
nginx: configuration file /tmp/nginx-cfg454025061 test failed                                                                                                                              │
-------------------------------------------------------------------------------     

[aledbf]

variables-hash-bucket-size: "128"

This is the default. Please increase the value to 256.

[juliohm1978]

Using variables-hash-bucket-size: "256" actually fixed the issue for 0.28.0. I'm still trying to understand what it means, though.

[juliohm1978]

Closing now. Thanks @aledbf for the help!

For anyone facing this issue, the reference link provided in the nginx-ingress docs is outdated.

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#variables-hash-bucket-size

We are sent to this page:

http://nginx.org/en/docs/http/ngx_http_map_module.html#variables_hash_bucket_size

... but I think it should link users to this page:

http://nginx.org/en/docs/http/ngx_http_core_module.html#variables_hash_bucket_size

... which contains the actual reference for variables_hash_bucket_size.

An even better explanation would be: http://nginx.org/en/docs/hash.html

Setting up hashes

To quickly process static sets of data such as server names, map directive’s values, MIME types, names of request header strings, nginx uses hash tables. During the start and each re-configuration nginx selects the minimum possible sizes of hash tables such that the bucket size that stores keys with identical hash values does not exceed the configured parameter (hash bucket size). The size of a table is expressed in buckets. The adjustment is continued until the table size exceeds the hash max size parameter. Most hashes have the corresponding directives that allow changing these parameters, for example, for the server names hash they are server_names_hash_max_size and server_names_hash_bucket_size.

From what I can gather, the nginx-ingress-controller generates a significant number of variables in its final /etc/nginx/nginx.conf and the default value for variables_hash_bucket_size (64 buckets) is not enough to store values for ~100 Deployments/Statefulsets.

Increasing variables-hash-bucket-size to 256 fixed the issue here.

Thanks!

[demisx]

Thank you so much! Just ran into the same issue after upgrading nginx-ingress helm chart to v1.30.0 that uses 1.28.0 app version. Adding this to the Nginx config map fixed the problem:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-controller
data:
  ... 
  variables-hash-bucket-size: "256"