Rocket.chat: Unable to start Rocket.Chat 0.72 on CentOS 7.5.1804

[gerricom]

Description:

I tried to update Rocket.Chat from 0.71.1 to 0.72.0 on our CentOS 7.5.1804 machine. It is installed there as described in the official documentation.

The server isn't starting because of missingZLIB_1.2.9 in the sharp-module. Issuing the prepacked libz.so in the LD_PRELOAD environment variable, as some google results suggest, is not working out.

Steps to reproduce:

1. Install Rocket.Chat 0.72.0 on the latest CentOS as described in the official documentation.

Expected behavior:

Rocket.Chat should launch.

Actual behavior:

Rocket.Chat dies with an Exception.

Server Setup Information:

• Version of Rocket.Chat Server: 0.72.0
• Operating System: CentOS 7.5.1804
• Deployment Method: Manual
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version: 8..11.3
• MongoDB Version: 3.6

Relevant logs:

Startup-log

Nov 30 10:52:37 klopstock systemd[1]: Stopping The Rocket.Chat server...
Nov 30 10:52:39 klopstock systemd[1]: Stopped The Rocket.Chat server.
Nov 30 10:52:59 klopstock systemd[1]: Started The Rocket.Chat server.
Nov 30 10:52:59 klopstock systemd[1]: Starting The Rocket.Chat server...
Nov 30 10:53:15 klopstock rocketchat[31356]: Updating process.env.MAIL_URL
Nov 30 10:53:15 klopstock rocketchat[31356]: Starting Email Intercepter...
Nov 30 10:53:20 klopstock rocketchat[31356]: Exception in callback of async function: Error: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by /opt/Rocket.Chat/programs/server/npm/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Object.Module._extensions..node (module.js:681:18)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.load (module.js:565:32)
Nov 30 10:53:20 klopstock rocketchat[31356]: at tryModuleLoad (module.js:505:12)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Function.Module._load (module.js:497:3)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.require (module.js:596:17)
Nov 30 10:53:20 klopstock rocketchat[31356]: at require (internal/module.js:11:18)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/sharp/lib/constructor.js:10:15)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module._compile (module.js:652:30)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Object.Module._extensions..js (module.js:663:10)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.load (module.js:565:32)
Nov 30 10:53:20 klopstock rocketchat[31356]: at tryModuleLoad (module.js:505:12)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Function.Module._load (module.js:497:3)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.require (module.js:596:17)
Nov 30 10:53:20 klopstock rocketchat[31356]: at require (internal/module.js:11:18)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/sharp/lib/index.js:3:15)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module._compile (module.js:652:30)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Object.Module._extensions..js (module.js:663:10)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.load (module.js:565:32)
Nov 30 10:53:20 klopstock rocketchat[31356]: at tryModuleLoad (module.js:505:12)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Function.Module._load (module.js:497:3)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.require (module.js:596:17)
Nov 30 10:53:20 klopstock rocketchat[31356]: at require (internal/module.js:11:18)
Nov 30 10:53:20 klopstock rocketchat[31356]: at npmRequire (/opt/Rocket.Chat/programs/server/npm-require.js:133:10)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.useNode (packages/modules-runtime.js:664:18)
Nov 30 10:53:20 klopstock rocketchat[31356]: at index.js (packages/modules.js:578:8)
Nov 30 10:53:20 klopstock rocketchat[31356]: at fileEvaluate (packages/modules-runtime.js:336:7)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.require (packages/modules-runtime.js:238:14)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/reify/lib/runtime/index.js:38:38)
Nov 30 10:53:20 klopstock rocketchat[31356]: at assets.js (packages/rocketchat:assets/server/assets.js:1:748)
Nov 30 10:53:20 klopstock rocketchat[31356]: at fileEvaluate (packages/modules-runtime.js:336:7)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.require (packages/modules-runtime.js:238:14)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/reify/lib/runtime/index.js:38:38)
Nov 30 10:53:20 klopstock rocketchat[31356]: at index.js (packages/rocketchat:assets/server/index.js:1:8)
Nov 30 10:53:20 klopstock rocketchat[31356]: at fileEvaluate (packages/modules-runtime.js:336:7)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Module.require (packages/modules-runtime.js:238:14)
Nov 30 10:53:20 klopstock rocketchat[31356]: at require (packages/modules-runtime.js:258:21)
Nov 30 10:53:20 klopstock rocketchat[31356]: at /opt/Rocket.Chat/programs/server/packages/rocketchat_assets.js:643:15
Nov 30 10:53:20 klopstock rocketchat[31356]: at /opt/Rocket.Chat/programs/server/packages/rocketchat_assets.js:648:3
Nov 30 10:53:20 klopstock rocketchat[31356]: at /opt/Rocket.Chat/programs/server/boot.js:411:36
Nov 30 10:53:20 klopstock rocketchat[31356]: at Array.forEach (<anonymous>)
Nov 30 10:53:20 klopstock rocketchat[31356]: at /opt/Rocket.Chat/programs/server/boot.js:220:19
Nov 30 10:53:20 klopstock rocketchat[31356]: at /opt/Rocket.Chat/programs/server/boot.js:471:5
Nov 30 10:53:20 klopstock rocketchat[31356]: at Function.run (/opt/Rocket.Chat/programs/server/profile.js:510:12)
Nov 30 10:53:20 klopstock rocketchat[31356]: at /opt/Rocket.Chat/programs/server/boot.js:470:11

[reetp]

This is the error:

Exception in callback of async function: Error: /lib64/libz.so.1: version `ZLIB_1.2.9' not found
(required by /opt/Rocket.Chat/programs/server/npm/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)
Nov 30 10:53:20 klopstock rocketchat[31356]: at Object.Module._extensions..node (module.js:681:18)

Have a read here:
https://bugs.centos.org/view.php?id=9686
I think for whatever good reason you do not have a high enough version of libz.so.1

[gerricom]

I think for whatever good reason you do not have a high enough version of libz.so.1

Well yes, that's because CentOS only ships with 1.2.7. Manually upgrading libz isn't recommended as this might brick the installation, as you can read for example here: https://stackoverflow.com/questions/48306849/lib-x86-64-linux-gnu-libz-so-1-version-zlib-1-2-9-not-found#comment92303162_48579743

It might be that Rocket.Chat will no longer work out of the box with CentOS as suggested in the documentation.

[reetp]

Could well be right.

Does this show you the ZLIB version?

strings /lib64/libz.so.1 | grep ZLIB

As a note this may be kind of related:
https://github.com/RocketChat/Rocket.Chat/issues/9829

[gerricom]

Here is the output

# strings /lib64/libz.so.1 | grep ZLIB
ZLIB_1.2.0
ZLIB_1.2.0.2
ZLIB_1.2.0.8
ZLIB_1.2.2
ZLIB_1.2.2.3
ZLIB_1.2.2.4
ZLIB_1.2.3.3
ZLIB_1.2.3.4
ZLIB_1.2.3.5
ZLIB_1.2.5.1
ZLIB_1.2.5.2

[reetp]

Your version:

ZLIB_1.2.5.2

So hence:

version ZLIB_1.2.9 not found

Going to need a dev to consider this one.

[tassoevan]

@gerricom Could you check if this works? https://github.com/RocketChat/Rocket.Chat/issues/9829#issuecomment-371487890

[mexashaggy]

I'm in the same boat as @gerricom. I tried the export LD_PRELOAD suggestion, and it did not work. I didn't know if I needed to rebuild after doing that, so I did a npm rebuild sharp after the export, and it still did not work. Any other suggestions, or could someone point me to where I can download 0.71.1?

[geekgonecrazy]

Try deleting bundle, npm modules and everything and downloading the bundle again and reinstalling npm modules in case some npm module is changed from manual recompile. Also make sure you check the path you set in LD_PRELOAD and see if you are actually pointing to the libz.so file.

releases.rocket.chat/0.71.1/download should get you 0.71.1 release. Though 0.72.0 is now out

FWIW LD_PRELOAD=/opt/Rocket.Chat/programs/server/npm/node_modules/sharp/vendor/lib/libz.so is likely what it would be

https://github.com/RocketChat/Rocket.Chat/blob/develop/.docker/Dockerfile.rhel -might be worth taking a look how it’s done on RHEL7

[gerricom]

I didn't have the time to test this during the weekend, so I gave it a fresh shout this morning. I'm pretty sure, I tried this before, but after all, with this systemd script, Rocket.Chat starts with 0.72:

[Unit]
Description=The Rocket.Chat server
After=network.target remote-fs.target nss-lookup.target nginx.target mongod.target
[Service]
ExecStart=/usr/local/bin/node /opt/Rocket.Chat/main.js
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=rocketchat
User=root
Environment=LD_PRELOAD=/opt/Rocket.Chat/programs/server/npm/node_modules/sharp/vendor/lib/libz.so NODE_ENV=production MONGO_URL=mongodb://localhost:27017/rocketchat ROOT_URL=xxx PORT=3000
[Install]
WantedBy=multi-user.target

[geekgonecrazy]

Going to go ahead and close this as the fix is now added to the documentation thanks to @gerricom

link for reference: https://rocket.chat/docs/installation/manual-installation/centos/#configure-the-rocketchat-service

[janzhanal]

Maybe its old already but LD_PRELOAD is missing at this moment in docs:
https://rocket.chat/docs/installation/manual-installation/centos/#configure-the-rocketchat-service

[reetp]

Maybe its old already but LD_PRELOAD is missing at this moment in docs:
https://rocket.chat/docs/installation/manual-installation/centos/#configure-the-rocketchat-service

Read the document.

This installation guide was tested in the following environment:
-    Rocket.Chat 1.0.2

You should be using Rocket.Chat v1.x

[janzhanal]

yes, I'm idiot, I didn't read it till the end.. sorry.