Kong: centOS RHEL 7 - Kong is broken here.

[arwinf]

Summary

I tried to install kong on centOS with from repository and by package manually.
Both installations are successful, but if I try to use any kong command it does even nothing; No ERROR. Instantly.

Steps To Reproduce

1. Install from repo or the package
2. execute "kong start" command
3. see nothing happens

Additional Details & Logs

With version 10.3 it works fine..

• Kong version ($ kong version): (cant call the command, but kong-community-edition-0.11.0-1.noarch)
• Kong debug-level startup logs ($ kong start --vv) -- no logs generated

• Kong error logs (<KONG_PREFIX>/logs/error.log) -- nothing here

• Operating System

  Operating System: CentOS Linux 7 (Core)
  CPE OS Name: cpe:/o:centos:centos:7
  Kernel: Linux 3.10.0-514.26.2.el7.x86_64
  Architecture: x86-64

[arwinf]

I've found something in

journalctl -ex

kernel: traps: nginx[3489] trap invalid opcode ip:7f5c778b4a0c sp:7ffec8e7b2a0 error:0 in libluajit-5.1.

What is this..

[p0pr0ck5]

This sounds like potentially a hardware or kernel issue? The message in question comes from the kernel.

Also, do you have any other LuaJIT or related packages/libraries install on that system? Are you able to reproduce in various hardware environments? The more detailed information you can pass on, the better.

[gszr]

Just added the RPM repo to a CentOS 7 (release 7.3.1611), installed Kong, and could not reproduce.

[arwinf]

Its running on EXSI as an virtual machine. But why the hell is the previous version 10.3 working like charme? :octocat:

[p0pr0ck5]

@arwinf it's possible that changes in compilation between 0.10.x and 0.11 uncovered a bug or issue on your system. Have you been able to replicate this anywhere else, on any other hardware or such?

But why the hell is the previous version 10.3 working like charme?

No need to swear :)

[p0pr0ck5]

@arwinf any update?

[JnMik]

Same problem, kong running inside a container

Edit: Could this be related ?
https://github.com/openresty/docker-openresty/issues/39
https://github.com/openresty/openresty/issues/256
They're talking about LuaJIT SSE 4.2 instructions not compatible with old CPUs

Edit2: And I see Kong 0.11 have this in the changelogs :

Kong now requires OpenResty 1.11.2.4. OpenResty's LuaJIT can now be built with Lua 5.2 compatibility. #2489 #2790

Edit3: I've run some other tests since I have many nodes with different CPUs at my disposal, and this could pretty much be CPU related in my case. Guess I'll have to use swarm labels now.

[sido420]

Found similar issue in Kong (latest) on a VM running Ubuntu 16.x running on Virtualbox. I used the Ubuntu package of Kong.

tail of dmesg (msgs only get printed when I run kong).

[37390.606338] traps: nginx[8774] trap invalid opcode ip:7fe2b8e90373 sp:7ffe0cec0d50 error:0 in libluajit-5.1.so.2.1.0[7fe2b8e88000+78000]
[37394.105187] traps: nginx[8776] trap invalid opcode ip:7f061a9b7373 sp:7ffc97703120 error:0 in libluajit-5.1.so.2.1.0[7f061a9af000+78000]
[37547.285868] traps: nginx[8789] trap invalid opcode ip:7ff6efe31373 sp:7ffd7092cc50 error:0 in libluajit-5.1.so.2.1.0[7ff6efe29000+78000]
[37606.732810] traps: nginx[8797] trap invalid opcode ip:7fab621da373 sp:7ffc82d51880 error:0 in libluajit-5.1.so.2.1.0[7fab621d2000+78000]

No messages whatsoever printed on console and Kong is not starting.

[johndotpub]

We've confirmed with our local testing hardware that any CPU that does not support the flag for _sse4_2_ will not launch the v0.11 Kong due to the LuaJIT instruction requirements.

cat /proc/cpuinfo | grep -i sse | head -n1 and if you don't see _sse4_2_ you will hit this issue.

Great catch @JnMik .... you saved us a load of trouble. Currently replacing some devboxes now. Swarm label needs just got real. ;)

[JnMik]

Pleasure @z3usy :)
And indeed, swarm labels just got real ahaha

[p0pr0ck5]

Thank you all for the discussion! We appreciate the community feedback here. I will add a note to our docs discussing this case.

[jgava]

I have two guests CentOS 7 in HyperV from different hypervisors.

1) guest01
cat /proc/cpuinfo | egrep 'model|vendor|flags'
vendor_id : GenuineIntel
model : 45
model name : Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl pni cx16 hypervisor lahf_lm

2) guest02
cat /proc/cpuinfo | egrep 'model|vendor|flags'
vendor_id : AuthenticAMD
model : 9
model name : AMD Opteron(tm) Processor 6174
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow art rep_good nopl extd_apicid pni cx16 popcnt hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw

guest01 kong works, and its not available sse4_2 as you can see above.
guest02 kong dont start, the output: kernel: traps: nginx[10017] trap invalid opcode ip:7f647169ea5e sp:7ffca41242f0 error:0 in libluajit-5.1.so.2.1.0[7f6471696000+77000]

both use same kernel 3.10.0-514.2.2.el7.x86_64

So, I may be wrong but it does not seem to be the processor flag the root cause of the incompatibility.

[thibaultcha]

As pointed out in previous comments in this thread, this is, one way or another, very much likely the effect of an OpenResty (and subsequently, LuaJIT) upgrade between Kong 0.10.3 and Kong 0.11.0 (1.11.2.2 to 1.11.2.4). I'll be closing this as this undeniably becomes an issue with our essential, underlying dependency which is OpenResty itself...

Thank you everybody who jumped in and shared their findings!

[minusdavid]

I reckon Kong Inc should include this issue on their installation pages.

I've been sitting here trying out different installation methods thinking that Kong, Inc aren't maintaining their documentation or software, when the real problem seems to be my hardware (and a lack of warnings about a known issue).

[coopr]

@minusdavid we welcome PRs to improve Kong's documentation - I encourage you to add the warning you seek - see https://github.com/Kong/docs.konghq.com/blob/master/CONTRIBUTING.md to get started.

[minusdavid]

Thanks @coopr. Someone else suggested the same thing on a different issue, but I found https://github.com/Kong/docs.konghq.com/blob/master/CONTRIBUTING.md to be a bit too dense and labourious for a 1 line change. It's probably the only Kong contribution I'll ever make, so I'm not interested in setting up a dev environment and then strictly following a process to add a warning.

I get that the process is there for good reason, but the barrier to entry is just a bit to high for me I think. I have way too many other priorities at the moment.

[coopr]

Sure makes sense @minusdavid - we are working to make contributing more streamlined. In the mean time, I'd be happy to make the change on your behalf - I just need to know what needs to be added where.

[minusdavid]

I actually expected to get harshed out by you, so your reply is very welcome! Thanks for that.

This is the patch that I wrote: https://github.com/minusdavid/docs.konghq.com/commit/96b5de94c9d8f4ed6002696daff44ace2cb919ba

Happy for the wording to be changed, but I figure that described the problem and was in a good place in the documentation.

[coopr]

Oh nice! If you want to PR that you are welcome to do so, no need to run the change locally. If you've done as much as you plan to do, no worries, I'll copy/paste your change into a PR - just LMK your preference.

[minusdavid]

I don't really know what I'm doing all the text content with the Github PR but I've made this PR anyway: https://github.com/Kong/docs.konghq.com/pull/1028.