Consul-helm: Envoy sidecar shutting down too early causes requests to fail

Created on 17 Mar 2021 · 2Comments · Source: hashicorp/consul-helm

Hey guys,

during a rolling update (pod termination) we are getting 502 errors in nginx for upstream requests. This happens because the load balancer in k8s still sends requests to the terminating pod due to the fact that the endpoint deregistration happens asynchronously, see https://github.com/kubernetes/kubernetes/issues/43576.

For nginx itself, I was able to resolve this race condition issue by adding a simple sleep to the preStop hook of the container

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5 && /usr/sbin/nginx -s quit"]

but the problem is not fully resolved since the envoy sidecar is still shutting down too early which causes requests to fail.

Current behavior

Nginx returns 502 errors because the upstream requests are send through the envoy sidecar proxy which is already shutting down due to the SIGTERM send by k8s.

Expected behavior

The envoy sidecar proxy should handle requests of the proxied service as long as the service is still running, e.g. if the service does some cleanup and still needs to send data when terminating or in case of nginx if further requests are still being routed to the pod.

Suggestion

In the envoy preStop command a delay could be added to prevent instantly sending the SIGTERM signal to the container. Since a hardcoded sleep duration seems to be too static maybe it would make sense to add something like this

while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do sleep 1; done

It would delay the shutdown process of envoy until there are no more TCP listeners, i.e. the proxied service is no longer running and it is save to also shutdown envoy without causing further requests to fail.

Another option could be to allow the user of the helm chart to customize the envoy preStop hook.

Environment details

consul-k8s version: 0.24.0
consul-helm version: 0.30.0
consul version: 1.9.3
envoy version: 1.16.0

areconnect bug themupgrades

Source

nflaig

❤1 👍1

Most helpful comment

hey @nflaig !! Thanks so much for bringing this to our attention. Also, the attention to detail as well as the suggestions in the issue are really appreciated!!

The team is stretched a little thin at the moment and we might not be in a position to solve this in the next release but will try and prioritize this for the one after.

Will keep this issue updated with the status of the fix! Thanks again!!

thisisnotashwin on 17 Mar 2021

👍2

All 2 comments

hey @nflaig !! Thanks so much for bringing this to our attention. Also, the attention to detail as well as the suggestions in the issue are really appreciated!!

The team is stretched a little thin at the moment and we might not be in a position to solve this in the next release but will try and prioritize this for the one after.

Will keep this issue updated with the status of the fix! Thanks again!!

thisisnotashwin on 17 Mar 2021

👍2

Having the same problem. Maybe a more flexible solution would be calling something like curl http://127.0.0.1:19000/drain_listeners; sleep $SOME_CONFIGURABLE_VALUE in the containers preStop command?

Edit: using the latest helm chart 0.32.1