Ingress-nginx: "client_max_body_size" directive invalid value
BUG REPORT:
NGINX Ingress controller version:
0.13.0
Kubernetes version (use kubectl version):
1.9.6
Environment:
- Cloud provider or hardware configuration:
- Self hosted, Dell 620's. PXE Boot
- OS (e.g. from /etc/os-release):
- CoreOS Stable
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened:
When using version 0.9.0 setting the client_max_body_size works fine, however when trying to upgrade to 0.13.0 nginx fails to start with the following error:
Error: exit status 1
2018/05/11 11:36:56 [emerg] 65#65: "client_max_body_size" directive invalid value in /tmp/nginx-cfg219046792:764
nginx: [emerg] "client_max_body_size" directive invalid value in /tmp/nginx-cfg219046792:764
nginx: configuration file /tmp/nginx-cfg219046792 test failed
qrpike
All 15 comments
@qrpike please check the value using kubectl exec <ing pod> cat /tmp/nginx-cfg219046792
aledbf
on 11 May 2018
@aledbf I also saw this while upgrading from 0.14.0 to 0.15.0 (rolling deployment via kubectl edit deployment, running in GKE, 1.9.6-gke.1).
For me, some of the configured values initially contained an empty string:
kubectl -n ingress-nginx exec nginx-ingress-controller-XXX-YYY grep client_max_body_size /tmp/nginx-cfg187873392
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
An nginx restart/reload (during the rolling update) seems to have fixed it though (this is the same pod as above):
kubectl -n ingress-nginx exec nginx-ingress-controller-XXX-YYY grep client_max_body_size /etc/nginx/nginx.conf
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
Looking at the specific nginx config block where an empty value was present, it seems they correspond to nginx locations for specific Ingress paths configured like so:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
...
spec:
tls:
...
rules:
- host: ...
http:
paths:
- path: /first
backend:
...
- path: /foo/second-path
backend:
...
- path: /bar/third-path
backend:
...
jpalomaki
on 17 May 2018
I've also hit this while doing a helm upgrade:
grep client_max /tmp/nginx-cfg492895181
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
Interestingly, the only thing I had done was enable mod_security in the configmap:
jpds
on 31 May 2018
We are also seeing the same issue with 0.13.0.
W0726 18:37:02.514973 5 queue.go:113] requeuing , err
-------------------------------------------------------------------------------
Error: exit status 1
2018/07/26 18:37:01 [emerg] 33#33: "client_max_body_size" directive invalid value in /tmp/nginx-cfg377740575:721
nginx: [emerg] "client_max_body_size" directive invalid value in /tmp/nginx-cfg377740575:721
nginx: configuration file /tmp/nginx-cfg377740575 test failed
kubectl exec internal-nginx-nginx-ingress-controller-65588f5998-chhlb -- grep client_max_body_size /tmp/nginx-cfg377740575
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
bismarck
on 26 Jul 2018
NGINX Ingress controller version:
0.15.0
Kubernetes version (use kubectl version):
1.9.6
Environment:
AWS
Tectonic CoreOS cluster
What happened:
Experiencing the same issue as the others above. We are suddenly getting lots of these errors:
2018/07/31 21:04:00 [emerg] 37#37: "client_max_body_size" directive invalid value in /tmp/nginx-cfg613783888:1610
nginx: [emerg] "client_max_body_size" directive invalid value in /tmp/nginx-cfg613783888:1610
nginx: configuration file /tmp/nginx-cfg613783888 test failed
We've attempted to set a global value for client_max_body_size using proxy-body-size but this doesn't seem to work. Even after setting proxy-body-size to 1m, the default value for nginx, half of the values are still empty.
Further, we have two ingresses with nginx.ingress.kubernetes.io/proxy-body-size annotations; one set at 32m and one set at 256m. Only one of them, the 256m, seems to work. The other appears in the nginx conf with empty quotes like the rest.
kubectl exec -it my-nginx-ingress-controller-6bd7cdc647-zxtlj -- grep client_max_body_size /tmp/nginx-cfg613783888
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "256m";
client_max_body_size "256m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "1m";
client_max_body_size "1m";
client_max_body_size "";
client_max_body_size "";
client_max_body_size "";
I'm hoping to take a further look at this error soon but wanted to at least present these details in case they help.
cmrust
on 31 Jul 2018
This nginx parse error message seems to indicate a wider problem. I've verified it still happens as of 0.17.1
For example, the generated config also has unset variables in server blocks such as set $proxy_upstream_name ""; and unset upstreams proxy_next_upstream ;.
The real issue here is ingress-controller should not generate the configuration until it has gathered all the information it needs.
Mikulas
on 6 Aug 2018
I have the same issue, Did anyone find a workaround?
dunjoye4real
on 4 Sep 2018
Saw this on 0.15.0 and 0.18.0 as well. In the instances I observed, the problem coincided with the deletion of ingress objects. i.e. I saw the following sequence:
- ingress controller logs deletion event
- Log entry for error reading ingress annotation logged (see https://github.com/kubernetes/ingress-nginx/blob/nginx-0.18.0/internal/ingress/controller/controller.go#L384) due to missing annotation on ingress
- Reload of the nginx configuration fails due to
client_max_body_size "";on the location corresponding to the ingress object.
It looks like there can be an inconsistency in store's state around deletions of ingress objects.
viveklak
on 6 Sep 2018
I managed to work around this by increasing the readiness/liveness initial delay period (from 10s to 30s in my case). Don't know if this will work for other people, but there it is.
danielfm
on 10 Sep 2018
Same problem on 0.18.0 and 0.19.0
bernardoVale
on 25 Sep 2018
This is still an issue for us, but will definitely try out @danielfm solution above. It can't hurt.
mmerrill3
on 1 Oct 2018
We've fixed the issue by increasing the memory limit
bernardoVale
on 1 Oct 2018
On the documentation, this value is not a string, has any reason to be a string?
http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
I could get any error setting on this but still not working for me.
and the value is set like client_max_body_size "16m";
ggrocco
on 1 Oct 2018
There seems to be a race condition somewhere. Ingress controller fails to start consistently with not-enough CPU (I could reliably reproduce the error with as much as cpu: 100m for roughly 300 ingresses). With enough CPU everything starts properly; in our case, that was cpu: 250m (both as requests and limits to keep it in the Guaranteed QoS).
I set memory guarantee to 300Mi and it did not make any difference whatsoever, as long as it had enough to actually start. I've measured that our instances peak at roughly 180Mi.
I've tested this on 0.20.
Mikulas
on 9 Oct 2018
I have had similar experience on 0.20.0 and I can confirm that implementing the fixes as suggested by @danielfm and @Mikulas solved the problem for me.
It's been a day without incident (usually it takes only 3 - 4 hours for problems to appear).
LoudRR
on 10 Oct 2018
Related issues
natemurthy
路
3Comments
oilbeater
路
3Comments
geek876
路
3Comments
smeruelo
路
3Comments
sophaskins
路
3Comments
Most helpful comment
I have the same issue, Did anyone find a workaround?