Kubernetes-ingress: 413 Request Entity Too Large

Created on 25 Apr 2016  ·  24Comments  ·  Source: nginxinc/kubernetes-ingress

We've deployed a docker registry and created an ingress rule to it's kubernetes service whilst using the nginx ingress controller. When pushing larger images we quickly hit the nginx limits giving us the error below.

Error parsing HTTP response: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.9.14</center>\r\n</body>\r\n</html>\r\n"

I've forked the repo and hacked the nginx config adding a client_max_body_size attribute so we can push larger images. For a proper solution though, it might be nice to set a value in the kubernetes ingress rule and have that used when the nginx controller is updated?

picture rawlingsj
👍3

Most helpful comment

FYI, the annotation has changed and is now:

    nginx.ingress.kubernetes.io/proxy-body-size: 50m

Also, I had to restart the nginx pod for the effect to take place. It immediately started working after that.

picture lastcoolnameleft on 28 Jul 2018
👍89 🎉30 ❤8 🚀3

All 24 comments

We plan to add the ability to configure the global NGINX parameters through a ConfigMap.

To allow to customize the configuration per Ingress resource, we can leverage annotations. As an example,

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    nginx/client_max_body_size: 1m
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

Will that work for your case?

pleshakov picture pleshakov on 25 Apr 2016

@pleshakov yeah that would be great!

rawlingsj picture rawlingsj on 25 Apr 2016

This already works with ConfigMaps such as:

apiVersion: v1
kind: ConfigMap
data:
  body-size: "0"
metadata:
  name: nginx-conf

A value of "0" lifts the restriction on client_max_body_size.
Make sure to reference the ConfigMap in your ingress controller like so:

args:
        - /nginx-ingress-controller
        - --default-backend-service=default/default-http-backend
        - --nginx-configmap=default/nginx-conf

For more information on the available config parameters have a look at here.

styxlab picture styxlab on 1 Jun 2016
👍3

We're adding this features as well as other NGINX configuration parameters soon. Stay tuned.

pleshakov picture pleshakov on 2 Jun 2016

customization of NGINX configuration was added in https://github.com/nginxinc/kubernetes-ingress/pull/33

pleshakov picture pleshakov on 16 Jun 2016

@rawlingsj Please check the example on how to customize NGINX: https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/customization

pleshakov picture pleshakov on 16 Jun 2016

Not worked in gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2 and gcr.io/google_containers/nginx-ingress-controller:0.8.3 I set value like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/client-max-body-size: "100m"
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: test
          servicePort: 8888
Hronom picture Hronom on 23 Feb 2017

@Hronom the google_containers/nginx-ingress-controller images belong to a different project. Please see the documentation for that project available on this page https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md

The image for this project -- https://hub.docker.com/r/nginxdemos/nginx-ingress/tags/

pleshakov picture pleshakov on 23 Feb 2017

@pleshakov sorry guys, too many kubernetes ingresses over internet...

Hronom picture Hronom on 23 Feb 2017
😄3 👍2

@Hronom I'm setting it like that, but it doesn't work.don't know why??

misteruly picture misteruly on 28 Aug 2017

In case it helps, 413 solved with the ingress.kubernetes.io/proxy-body-size :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: services
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/proxy-body-size: 50m

Running quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17

benbonnet picture benbonnet on 27 Nov 2017
👍14

FYI, the annotation has changed and is now:

    nginx.ingress.kubernetes.io/proxy-body-size: 50m

Also, I had to restart the nginx pod for the effect to take place. It immediately started working after that.

lastcoolnameleft picture lastcoolnameleft on 28 Jul 2018
👍89 🎉30 ❤8 🚀3

Setting it to "0" makes the nginx post size unrestricted:

    nginx.ingress.kubernetes.io/proxy-body-size: "0"
ryanrca picture ryanrca on 2 Feb 2019
👍19

Looks like it changed to client-max-body-size for configmap.
client-max-body-size: "0"

I didn't try with annotations:
nginx.org/client-max-body-size: "0"

mehmetali picture mehmetali on 23 Jul 2019

For apiVersion: extensions/v1beta1
nginx.ingress.kubernetes.io/proxy-body-size: 50m annotation didn't solve 413 error for me.
When I've added one more annotation
nginx.org/client-max-body-size: "50m" it did the job

AntonioArts picture AntonioArts on 14 Aug 2019
👍4

I came here to configure the Docker registry helm chart and found: 

    nginx.ingress.kubernetes.io/proxy-body-size: 50m

I added it to the Ingress for the registry and it did the trick. 👌

alexellis picture alexellis on 14 Aug 2019
👍12 😄1

This solved it for me:
nginx.ingress.kubernetes.io/proxy-body-size: 50m

theguyver picture theguyver on 17 Nov 2019

FYI, the annotation has changed and is now:

    nginx.ingress.kubernetes.io/proxy-body-size: 50m

Also, I had to restart the Nginx pod for the effect to take place. It immediately started working after that.

Hi,

I know this is a very old post, we have the same issue and I've updated nginx.ingress.kubernetes.io/proxy-body-size: 50m in the file, now I have to restart Nginx POD to changes to effect, can you please let me know how to restart K8 Nginix POD ..?

I know this way, we can restart PODs,

kubectl scale deployment name--replicas=0 -n service
kubectl scale deployment name --replicas=1 -n service

any other way i can restart Nginx PODs to reload the configuration.

Ganeshkumar1023 picture Ganeshkumar1023 on 19 Nov 2019
👍5

@Ganeshkumar1023 in latest version of kubectl (1.17.0) they have added restart sub-command under rollout command. You can use this to restart the pods. so command would be:

kubectl rollout restart deployment/abc
kubectl rollout restart -f abc.yaml

techsaau picture techsaau on 13 Feb 2020

I have tried both proxy-body-size and client-max-body-size on the configmap and did a rolling restart of the nginx controller pods and when I grep the nginx.conf file in the pod it returns the default 1m. I am trying to do this within Azure Kubernetes Service (AKS). I'm working with someone from their support. They said its not on them since it appears to be a nginx config issue.

The weird hting is we had other clusters in Azure that this wasnt an issue on until we discovered this with some of the newer deployments. The initial fix they came up with is what is in this thread but it just refuses to change.

Below is my configmap:

apiVersion: v1
data:
  client-max-body-size: 0m
  proxy-connect-timeout: 10s
  proxy-read-timeout: 10s
kind: ConfigMap
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"nginx-nginx-ingress-controller-7b9bff87b8-vxv8q","leaseDurationSeconds":30,"acquireTime":"2020-03-10T20:52:06Z","renewTime":"2020-03-10T20:53:21Z","leaderTransitions":1}'
  creationTimestamp: "2020-03-10T18:34:01Z"
  name: ingress-controller-leader-nginx
  namespace: ingress-nginx
  resourceVersion: "23928"
  selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx
  uid: b68a2143-62fd-11ea-ab45-d67902848a80

After issuing a rolling restart: kubectl rollout restart deployment/nginx-nginx-ingress-controller -n ingress-nginx

Grepping the nginx ingress controller pod to query the value now reveals:

kubectl exec -n ingress-nginx nginx-nginx-ingress-controller-7b9bff87b8-p4ppw cat nginx.conf | grep client_max_body_size
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    1m;
            client_max_body_size                    21m;

Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

Aech1977 picture Aech1977 on 11 Mar 2020
👍1

@Waterdrips cc

alexellis picture alexellis on 11 Mar 2020

think you

------------------ 原始邮件 ------------------
发件人: "Aech1977"<[email protected]>;
发送时间: 2020年3月11日(星期三) 晚上7:46
收件人: "nginxinc/kubernetes-ingress"<[email protected]>;
抄送: "蓝鹏"<[email protected]>;"Comment"<[email protected]>;
主题: Re: [nginxinc/kubernetes-ingress] 413 Request Entity Too Large (#21)

I have tried both proxy-body-size and client-max-body-size on the configmap and did a rolling restart of the nginx controller pods and when I grep the nginx.conf file in the pod it returns the default 1m. I am trying to do this within Azure Kubernetes Service (AKS). I'm working with someone from their support. They said its not on them since it appears to be a nginx config issue.

The weird hting is we had other clusters in Azure that this wasnt an issue on until we discovered this with some of the newer deployments. The initial fix they came up with is what is in this thread but it just refuses to change.

Below is my configmap:

Please edit the object below. Lines beginning with a '#' will be ignored,

and an empty file will abort the edit. If an error occurs while saving this file will be

reopened with the relevant failures.

apiVersion: v1
data:
client-max-body-size: 0m
proxy-connect-timeout: 10s
proxy-read-timeout: 10s
kind: ConfigMap
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"nginx-nginx-ingress-controller-7b9bff87b8-vxv8q","leaseDurationSeconds":30,"acquireTime":"2020-03-10T20:52:06Z","renewTime":"2020-03-10T20:53:21Z","leaderTransitions":1}'
creationTimestamp: "2020-03-10T18:34:01Z"
name: ingress-controller-leader-nginx
namespace: ingress-nginx
resourceVersion: "23928"
selfLink: /api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader-nginx
uid: b68a2143-62fd-11ea-ab45-d67902848a80
After issuing a rolling restart: kubectl rollout restart deployment/nginx-nginx-ingress-controller -n ingress-nginx

Grepping the nginx ingress controller pod to query the value now reveals:

kubectl exec -n ingress-nginx nginx-nginx-ingress-controller-7b9bff87b8-p4ppw cat nginx.conf | grep client_max_body_size
client_max_body_size 1m;
client_max_body_size 1m;
client_max_body_size 1m;
client_max_body_size 1m;
client_max_body_size 21m;
Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

misteruly picture misteruly on 12 Mar 2020

Doesnt matter where I try to change it. On the configmap for global or the Ingress route specifically.......this value above never changes.

@Aech1977 did you solve it?

jomel picture jomel on 19 Oct 2020

@Aech1977 this key is wrong client-max-body-size: 0m
the key for the configmap is proxy-body-size: 0m

jvirriel-per picture jvirriel-per on 28 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

martinlevesque picture martinlevesque  ·  3Comments
ghost picture ghost  ·  3Comments
jaytaylor picture jaytaylor  ·  4Comments
martinlevesque picture martinlevesque  ·  5Comments
ajpinedam picture ajpinedam  ·  4Comments