Kibana: [ARM] Kibana fatal error on startup

[liza-mae]

Kibana version: 8.0.0-SNAPSHOT, 7.11-0-SNAPSHOT default distribution

Elasticsearch version: 8.0.0-SNAPSHOT, 7.11-0-SNAPSHOT default distribution

Server OS version: Linux aarch64

Original install method (e.g. download page, yum, from source, etc.):
snapshots

Description of the problem including expected versus actual behavior:
Linux arm build throws fatal error on startup, only happening on default distro, the oss one starts up fine.

Steps to reproduce:

1. Install ES and Kibana
2. Start Kibana, notice fatal error occurs:
   node_modules/re2/build/Release/re2.node: cannot open shared object file: No such file or directory

Provide logs and/or server output (if relevant):

06:58:21      │ proc [kibana]   log   [13:58:21.929] [fatal][root] Error: /var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/re2/build/Release/re2.node: cannot open shared object file: No such file or directory
06:58:21      │ proc [kibana]     at Object.Module._extensions..node (internal/modules/cjs/loader.js:807:18)
06:58:21      │ proc [kibana]     at Module.load (internal/modules/cjs/loader.js:653:32)
06:58:21      │ proc [kibana]     at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
06:58:21      │ proc [kibana]     at Function.Module._load (internal/modules/cjs/loader.js:585:3)
06:58:21      │ proc [kibana]     at Module.require (internal/modules/cjs/loader.js:692:17)
06:58:21      │ proc [kibana]     at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/require-in-the-middle/index.js:80:39)
06:58:21      │ proc [kibana]     at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/require-in-the-middle/index.js:80:39)
06:58:21      │ proc [kibana]     at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/require-in-the-middle/index.js:80:39)
06:58:21      │ proc [kibana]     at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/require-in-the-middle/index.js:80:39)
06:58:21      │ proc [kibana]     at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/require-in-the-middle/index.js:80:39)
06:58:21      │ proc [kibana]     at require (internal/modules/cjs/helpers.js:25:18)
06:58:21      │ proc [kibana]     at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/re2/re2.js:3:13)
06:58:21      │ proc [kibana]     at Module._compile (internal/modules/cjs/loader.js:778:30)
06:58:21      │ proc [kibana]     at Module._compile (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana/node_modules/pirates/lib/index.js:99:24)
06:58:21      │ proc [kibana]     at Module._extensions..js (internal/modules/cjs/loader.js:789:10)
06:58:21      │ proc [kibana]     at Object.newLoader [as .js] (/var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana/node_modules/pirates/lib/index.js:104:7)
06:58:21      │ proc [kibana] 
06:58:21      │ proc [kibana]  FATAL  Error: /var/lib/jenkins/workspace/elastic+estf-kibana+master-snapshot+multijob-linux-arm/JOB/xpackGrp1/node/estf/kibana-build/kibana-8.0.0-SNAPSHOT-linux-aarch64/node_modules/re2/build/Release/re2.node: cannot open shared object file: No such file or directory

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[liza-mae]

This is blocking testing. (May impact #83094 too)

cc: @spalger

[smith]

This could be related to the Node 12 upgrade. I ran into the same problem (on Intel macOS) and did yarn kbn clean && yarn kbn bootstrap and it started working.

[liza-mae]

Thanks @smith - I am running on CI so it is a fresh install each time and it is still broken on master and 7.11 branches.

[liza-mae]

@tylersmalley we should get this fixed as soon as possible it will block 7.11 release and we should be testing continuously, can you take a look as to what might be the root cause here?

[tylersmalley]

@liza-mae ack, I will look into this today.

[tylersmalley]

@kobelb, it appears re2 doesn't have an ARM build: https://github.com/uhop/node-re2/releases. What was the reason the RE2 needed to be used over RegExp? Is it still applicable today?

[kobelb]

@tylersmalley we're using node-re2 so that we can safely use the regular expressions specified by end-users in Timelion: https://cve.circl.lu/cve/CVE-2020-7016

[tylersmalley]

I have a build of node-re2 on ARM working.

Next is to host that build and update the PatchNativeModules task to be arch aware to reference that build. This should resolve this issue, but we will follow up to automate that build or to see if the upstream project would be interested in adding to their pipeline.

[tylersmalley]

I have a PR to resolve this issue here: https://github.com/elastic/kibana/pull/84163

[tylersmalley]

@liza-mae, the fix has been merged, would you mind confirming it's resolved your issue?

[tylersmalley]

@liza-mae, have you been able to verify this was resolved?

[liza-mae]

@tylersmalley yes it is resolved, thank you.