Ingress-nginx: lua udp socket read timed out, context: ngx.timer

[Scong]

NGINX Ingress controller version:
0.28.0
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-f459c0", GitCommit:"f459c0672169dd35e77af56c24556530a05e9ab1", GitTreeState:"clean", BuildDate:"2020-03-18T04:24:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Environment:

• Cloud provider or hardware configuration:
  AWS-EKS
• OS (e.g. from /etc/os-release): Linux
• Kernel (e.g. uname -a):Linux loadbalancer-controller-nginx-ingress-loadbalancer-controlvn82t 4.14.171-136.231.amzn2.x86_64 #1 SMP Thu Feb 27 20:22:48 UTC 2020 x86_64 Linux
• Install tools:
  Helm, Flux
• Others:
  

  Flux helm release

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: loadbalancer-controller
namespace: prod-engineering
annotations:
flux.weave.works/automated: "true"
spec:
releaseName: loadbalancer-controller
helmVersion: v3
chart:
repository: https://kubernetes-charts.storage.googleapis.com/
name: nginx-ingress
version: 1.29.6
values:
controller:
name: loadbalancer-controller
scope:
enabled: true
namespace: prod-engineering
service:
targetPorts:
http: http
https: http
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
enableHttp: true
defaultBackend:
enabled: false

kubectl describe

Name: loadbalancer-controller-nginx-ingress-loadbalancer-controlvn82t
Namespace:
Priority: 0
Node:
Start Time: Thu, 25 Jun 2020 09:47:36 -0400
Labels: app=nginx-ingress
component=loadbalancer-controller
pod-template-hash=78f8f9788f
release=loadbalancer-controller
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP:
IPs:
Controlled By: ReplicaSet/loadbalancer-controller-nginx-ingress-loadbalancer-controller-78f8f9788f
Containers:
nginx-ingress-loadbalancer-controller:
Container ID: docker://ff55446b88a85cd34b0193463eb97c032e07f38698aea3374850ac0ab86dc6ff
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
Image ID: docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:ca2eee26afd16dc052c7950f13df6c906be279de64d990e88383c9f123556e06
Ports: 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--ingress-class=nginx
--configmap=prod-engineering/loadbalancer-controller-nginx-ingress-loadbalancer-controller
--watch-namespace=prod-engineering
State: Running
Started: Thu, 25 Jun 2020 09:47:44 -0400
Ready: True
Restart Count: 0
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: loadbalancer-controller-nginx-ingress-loadbalancer-controlvn82t (v1:metadata.name)
POD_NAMESPACE: prod-engineering (v1:metadata.namespace)
NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME: rcg-prod-k8s
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME: (v1:spec.nodeName)
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME: prod-engineering (v1:metadata.namespace)
NEW_RELIC_METADATA_KUBERNETES_POD_NAME: loadbalancer-controller-nginx-ingress-loadbalancer-controlvn82t (v1:metadata.name)
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME: nginx-ingress-loadbalancer-controller
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME: loadbalancer-controller-nginx-ingress-loadbalancer-controller
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from loadbalancer-controller-nginx-ingress-token-6gnwr (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
loadbalancer-controller-nginx-ingress-token-6gnwr:
Type: Secret (a volume populated by a Secret)
SecretName: loadbalancer-controller-nginx-ingress-token-6gnwr
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:

What happened:
Around 1am last night load balancer controller began throwing lua udp socket read timed out, context: ngx.timer errors. It appears this has happened in the past and recovery occurred fairly quickly, in this case the errors continued until 9am.

This error prevented many pods from being accessible until the load balancers controller was restarted.

What you expected to happen:
The errors to go away / recover / pods that don't use UDP connections to be accessible?

Without much knowledge, I guess that the load balancer controller got hung up on udp socket and prevented other connections from occurring. I'm not sure how this could be the case though.

The alternative is something is miss configured?

How to reproduce it:
This is an intermittent error, and difficult to reproduce.

Anything else we need to know:
I have not found other people with the same error so filing here with hopes that someone knows a bit about it. This may be more of a help request than a bug.

/kind bug

[aledbf]

@Scong please update to 0.33.0. Between the two versions, the luajit dependency and nginx were updated.

[Scong]

Upgraded to 0.32.0 (corresponding with most recent helm chart available). Will close issue and reopen if it happens again.

[Hermsi1337]

Hey @Scong and @aledbf , we're currently facing the exact same issue.

But tbh we don't really know how to investigate this behaviour further.
Maybe you guys have any tipps?

Our ingress-controller-log is full of messages like this:

2020/07/30 08:01:21 [error] 479#479: *33267 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:21 [error] 514#514: *26066 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:21 [error] 446#446: *33284 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:21 [error] 314#314: *33288 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:22 [error] 480#480: *31940 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:22 [error] 331#331: *31946 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:22 [error] 447#447: *31945 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:22 [error] 512#512: *30948 lua udp socket read timed out, context: ngx.timer
2020/07/30 08:01:22 [error] 347#347: *30964 lua udp socket read timed out, context: ngx.timer

Our environment looks like this...

Kubernetes:

Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Ingress:

apiVersion: apps/v1
kind: DaemonSet
metadata:
#[...]
  labels:
    app: nginx-ingress
    app.kubernetes.io/component: controller
    chart: nginx-ingress-1.41.2
    heritage: Helm
    release: ingress-nginx-public
#[...]
spec:
#[...]
  template:
#[...]
    spec:
      containers:
#[...]
        image: us.gcr.io/k8s-artifacts-prod/ingress-nginx/controller:v0.34.1

Many thanks !

Regards