Ingress-nginx: arm64: unprotected error in call to Lua API (bad light userdata pointer)

Created on 18 Jul 2018 · 6Comments · Source: kubernetes/ingress-nginx

Is this a BUG REPORT:

NGINX Ingress controller version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm64:0.17.1

Kubernetes version (use kubectl version): v1.10.5

Environment:

Cloud provider or hardware configuration: rock64
OS (e.g. from /etc/os-release): Gentoo/Linux
Kernel (e.g. uname -a): Linux cerberus02 4.17.2-gentoo #4 SMP Sat Jun 23 01:04:06 EEST 2018 aarch64 GNU/Linux
Install tools:
Others:

What happened:

Jul 18 02:56:24 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: NGINX Ingress controller
Jul 18 02:56:24 cerberus02 8467cda57149[20943]:   Release:    0.17.1
Jul 18 02:56:24 cerberus02 8467cda57149[20943]:   Build:      git-12f7966
Jul 18 02:56:24 cerberus02 8467cda57149[20943]:   Repository: https://github.com/kubernetes/ingress-nginx.git
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: nginx version: nginx/1.13.12
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: W0717 23:56:24.116206       8 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: I0717 23:56:24.117206       8 main.go:191] Creating API client for https://10.96.0.1:443
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: I0717 23:56:24.332529       8 main.go:235] Running in Kubernetes cluster version v1.10 (v1.10.5) - git (clean) commit 32ac1c9073b132b8ba18aa830f46b77dcceb0723 - platform linux/arm64
Jul 18 02:56:24 cerberus02 8467cda57149[20943]: I0717 23:56:24.384943       8 main.go:100] Validated ingress-nginx/default-http-backend as the default backend.
Jul 18 02:56:31 cerberus02 8467cda57149[20943]: I0717 23:56:31.169042       8 nginx.go:255] Starting NGINX Ingress controller
Jul 18 02:56:31 cerberus02 8467cda57149[20943]: I0717 23:56:31.219234       8 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"36639499-8a1c-11e8-bc57-e602c5d7d22a", APIVersion:"v1", ResourceVersion:"3943", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
Jul 18 02:56:31 cerberus02 8467cda57149[20943]: I0717 23:56:31.236387       8 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"366c3fca-8a1c-11e8-bc57-e602c5d7d22a", APIVersion:"v1", ResourceVersion:"3948", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
Jul 18 02:56:31 cerberus02 8467cda57149[20943]: I0717 23:56:31.236574       8 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"3672e0fe-8a1c-11e8-bc57-e602c5d7d22a", APIVersion:"v1", ResourceVersion:"3949", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.283653       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test-ingress", UID:"b4589798-8a1c-11e8-bc57-e602c5d7d22a", APIVersion:"extensions/v1beta1", ResourceVersion:"4311", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/test-ingress
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.370490       8 nginx.go:276] Starting NGINX process
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.371058       8 leaderelection.go:185] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.378071       8 controller.go:926] Ingress "default/test-ingress" defines a backend but no rule. Using it to configure the catch-all server "_"
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.378539       8 controller.go:169] Configuration changes detected, backend reload required.
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.487013       8 leaderelection.go:194] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: I0717 23:56:32.487592       8 status.go:197] new leader elected: nginx-ingress-controller-6598dcf768-hx7l9
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: E0717 23:56:32.494595       8 controller.go:181] Unexpected failure reloading the backend:
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: W0717 23:56:32.494670       8 queue.go:130] requeuing initial-sync, err
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:32 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: I0717 23:56:35.711299       8 controller.go:926] Ingress "default/test-ingress" defines a backend but no rule. Using it to configure the catch-all server "_"
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: I0717 23:56:35.711612       8 controller.go:169] Configuration changes detected, backend reload required.
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: E0717 23:56:35.831007       8 controller.go:181] Unexpected failure reloading the backend:
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: W0717 23:56:35.831101       8 queue.go:130] requeuing kube-system/kube-controller-manager, err
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:35 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: I0717 23:56:39.044671       8 controller.go:926] Ingress "default/test-ingress" defines a backend but no rule. Using it to configure the catch-all server "_"
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: I0717 23:56:39.044988       8 controller.go:169] Configuration changes detected, backend reload required.
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: E0717 23:56:39.156152       8 controller.go:181] Unexpected failure reloading the backend:
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: W0717 23:56:39.156230       8 queue.go:130] requeuing kube-system/kube-dns, err
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:39 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: I0717 23:56:42.377723       8 controller.go:926] Ingress "default/test-ingress" defines a backend but no rule. Using it to configure the catch-all server "_"
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: I0717 23:56:42.377935       8 controller.go:169] Configuration changes detected, backend reload required.
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: E0717 23:56:42.512025       8 controller.go:181] Unexpected failure reloading the backend:
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: W0717 23:56:42.512152       8 queue.go:130] requeuing default/nginx, err
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: Error: exit status 1
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: 
Jul 18 02:56:42 cerberus02 8467cda57149[20943]: -------------------------------------------------------------------------------
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.629530       8 main.go:158] Received SIGTERM, shutting down
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.629624       8 nginx.go:340] Shutting down controller queues
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.629698       8 status.go:119] updating status of Ingress rules (remove)
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.657504       8 status.go:138] removing address from ingress status ([])
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.658124       8 nginx.go:348] Stopping NGINX process
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: PANIC: unprotected error in call to Lua API (bad light userdata pointer)
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.693864       8 main.go:162] Error during shutdown: exit status 1
Jul 18 02:56:44 cerberus02 8467cda57149[20943]: I0717 23:56:44.694053       8 main.go:166] Handled quit, awaiting Pod deletion
Jul 18 02:56:54 cerberus02 8467cda57149[20943]: I0717 23:56:54.694330       8 main.go:169] Exiting with 1

What you expected to happen:
The ingress controller starts

How to reproduce it (as minimally and precisely as possible):
Boot the ingress controller on arm64 architecture

Anything else we need to know:

Looks like the infamous light userdata issue LuaJIT/LuaJIT#230 https://groups.google.com/forum/#!topic/openresty-en/Gp0NGN9lMXk etc. is affecting ingress-nginx on arm64 platform.

Same issue was observer on some other projects like: https://github.com/neovim/neovim/issues/7879

kinbug

Source

brezerk

All 6 comments

https://github.com/LuaJIT/LuaJIT/pull/245

LuaJIT on 64 bit systems provides a limited range of 47 bits for the legacy lightuserdata data type. This is only relevant on x64 systems which use the negative part of the virtual address space in user mode, e.g. Solaris/x64, and on ARM64 systems configured with a 48 bit or 52 bit VA. Avoid using lightuserdata to hold pointers that may point outside of that range, e.g. variables on the stack. In general, avoid this data type for new code and replace it with (much more performant) FFI bindings. FFI cdata pointers can address the full 64 bit range.

brezerk on 18 Jul 2018

/kind bug

ping @ElvinEfendi

antoineco on 22 Jul 2018

lua-nginx-module currently does not support ARM64, there's a pending PR: https://github.com/openresty/lua-nginx-module/pull/1379. We will update as soon as it gets merged and released.

ElvinEfendi on 10 Oct 2018

👍1

@brezerk please use quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm64:dev
This image contains current master with the luajit fork from openresty with support for arm64