Ingress-nginx: Requests hitting rate limit are sent 503 rather than 429

[gaieges]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report

NGINX Ingress controller version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0

Kubernetes version (use kubectl version): v1.9.6

Environment:

• Cloud provider or hardware configuration: Azure AKS w/ Azure LB in front
• OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS
• Kernel (e.g. uname -a): Linux aks-nodepool1-8-0 4.13.0-1012-azure #15-Ubuntu SMP Thu Mar 8 10:47:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Generic AKS + common ingress service components for :0.14.0
• Others:

What happened: When enabling the rate limiter and setting to a very low value to simulate hitting that limit quickly, ended up getting responses of 503 vs 429 which is what I would expect for a rate limit situation:

Ingress config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/limit-connections: '1'
    nginx.ingress.kubernetes.io/limit-rps: '1'
  name: prod
spec:
...

Response

HTTP/1.1 503 Service Temporarily Unavailable
Server: nginx/1.13.12
...

What you expected to happen: Expected to see a 429 response, ideally with a header for how much longer to wait, or something along those lines.

How to reproduce it (as minimally and precisely as possible): I believe it's just using that version of the ingress, and setting the above config attributes and hitting the service as hard as possible. I used:

parallel -j100 -n0 "curl 'https://HOST/' -H Host:HOST -k -Is" ::: $(seq 1 5000)

Anything else we need to know:
Not sure if this is the right place, could be an nginx issue too, but figured I'd throw it this way to get some context.

[aledbf]

Closing. Please check https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#limit-req-status-code to change the status code

[aledbf]

I will open a PR to change the default code to 429 (this is the second time during the week with the same report)

[gaieges]

Thanks for the quick response @aledbf ! Sorry I missed that

[wutingbupt]

@gaieges,
Have u managed make this work? I have added the limit-req-status-code inside the configmap, but it still show 503 during the test.

[gaieges]

@wutingbupt haven't tried it yet. I'd suggest opening another ticket with more details if thats true

[shaikatz]

@wutingbupt , the configMap option limit-req-stauts-code is only for request limits and not for connection limits, this is probably why you still see 503s.

[annProg]

@aledbf can add an annotation to change limit_req_status？ each location can use different limit_req_status

Syntax: limit_req_status code;
Default:    limit_req_status 503;
Context:    http, server, location
This directive appeared in version 1.3.15.

[aledbf]

@annProg you can use the configuration-snippet annotation :)

[annProg]

@annProg you can use the configuration-snippet annotation :)

Yes. but configuration-snippet sometimes dangerous if someone config a synax error. I have a web interface to check annotations value, if use annotation to set limit_req_status, I can use regex to check if user input match such as /^403$|^429$|^502$|^503$/ , while configuration-snippet can't use regex. So may be better to add an annotation.

Error: exit status 1
2019/06/23 07:45:03 [emerg] 26321#26321: invalid number of arguments in "limit_req_status" directive in /tmp/nginx-cfg241016693:794
nginx: [emerg] invalid number of arguments in "limit_req_status" directive in /tmp/nginx-cfg241016693:794
nginx: configuration file /tmp/nginx-cfg241016693 test failed

[annProg]

@aledbf I did a test. Set configuration-snippet to limit_req_status 429 (Missing semicolon），then run docker restart ingress pod, then ingress can not run, If I can not notice this in time, when pod restart or machine restart, ingress will not run. So configuration-snippet is a Administrator only configuration in my environment. I hope to use a special configuration to change limit_req_status.

Error: exit status 1
2019/06/23 07:53:49 [emerg] 106#106: invalid number of arguments in "limit_req_status" directive in /tmp/nginx-cfg132769299:794
nginx: [emerg] invalid number of arguments in "limit_req_status" directive in /tmp/nginx-cfg132769299:794
nginx: configuration file /tmp/nginx-cfg132769299 test failed

-------------------------------------------------------------------------------
W0623 07:53:49.452586      10 queue.go:130] requeuing dev/default-http-backend, err 
-------------------------------------------------------------------------------
Error: exit status 1
2019/06/23 07:53:49 [emerg] 106#106: invalid number of arguments in "limit_req_status" directive in /tmp/nginx-cfg132769299:794
nginx: [emerg] invalid number of arguments in "limit_req_status" directive in /tmp/nginx-cfg132769299:794
nginx: configuration file /tmp/nginx-cfg132769299 test failed

-------------------------------------------------------------------------------
I0623 07:53:52.460450      10 main.go:163] Received SIGTERM, shutting down
I0623 07:53:52.460489      10 nginx.go:341] Shutting down controller queues
I0623 07:53:52.460512      10 status.go:119] updating status of Ingress rules (remove)