Ingress-nginx: Existing connections get reset after an ingress configuration

Created on 20 Oct 2017  路  2Comments  路  Source: kubernetes/ingress-nginx

BUG REPORT

NGINX Ingress controller version:
NGINX: 0.9.0-beta.15

Kubernetes version (use kubectl version):
1.7.8-00

Environment:
Can reproduce in both environment
Bare-metal VM. k8s is setup through kubeadm
Bare-metal VM, k8s is setup through coreos matchbox

  • Cloud provider or hardware configuration:
    Openstack VM

  • OS (e.g. from /etc/os-release):

  • ubuntu 16.04
  • coreos
  • Kernel (e.g. uname -a):
    Linux edge-0 4.4.0-96-generic
  • Install tools:
  • kubeadm
  • coreos matchbox
  • Others:

What happened:
Existing requests (connections) to backend gets reset after any ingress configuration.
Client (REST client) gets a connection reset error

What you expected to happen:
Existing requests should not be dropped or interrupted.

How to reproduce it (as minimally and precisely as possible):
Create a simple REST API service with python flask with a helloworld GET. This GET will sleep for 120 seconds then return a "Hello World" response.
Create corresponding service and ingress with a hostname "helloworld.example.com"

curl the endpoint with "curl -H "Host: helloworld.example.com" https://[node_ip]" Normally, I will get a timeout response or a proper response if I increase the timeout in ingress.
However, if I do any ingress configuration. (kubectl create ingress or kubectl delete ingresss) which triggers a nginx reload. I will get a connection reset error. In nginx-controller logs, it shows configuration reload then also a 599 error.

Anything else we need to know:
If I use nginxinc's nginx-ingress-controller, it works as expected.
I think it is related to configuration reload.

Most helpful comment

@aledbf Is there a setting to never close existing connections, like it was in versions 0.9.0-beta.14 and before? We have long-standing connections going through the ingress, that we don't want resetted on every config change.

All 2 comments

Closing. This works as expected.
If you want to avoid this behavior you can tune the setting worker-shutdown-timeout in the configuration configmap. This setting sets a timeout for Nginx to wait for worker to gracefully shutdown. The default is "10s".

@aledbf Is there a setting to never close existing connections, like it was in versions 0.9.0-beta.14 and before? We have long-standing connections going through the ingress, that we don't want resetted on every config change.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

yuyang0 picture yuyang0  路  3Comments

c-mccutcheon picture c-mccutcheon  路  3Comments

cxj110 picture cxj110  路  3Comments

kfox1111 picture kfox1111  路  3Comments

bashofmann picture bashofmann  路  3Comments