Sharp: Electron beta appears to use system zlib, can conflict with sharp

[lovell]

https://github.com/lovell/sharp/issues/843#issuecomment-318771819 reported by @Apophenia

ELECTRON_ASAR.js:173 Uncaught Error: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /[path]/electron-quick-start/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)
    at process.module.(anonymous function) [as dlopen] (ELECTRON_ASAR.js:173:20)
    at Object.Module._extensions..node (module.js:598:18)
    at Object.module.(anonymous function) [as .node] (ELECTRON_ASAR.js:173:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> ([path]/electron-quick-start/node_modules/sharp/lib/constructor.js:9:15)
    at Object.<anonymous> ([path]/electron-quick-start/node_modules/sharp/lib/constructor.js:234:3)

Electron v1.7.5 beta

[lovell]

Possible workaround for systems that still provide only zlib 1.2.8 would be to use something like:

LD_PRELOAD=/path/to/node_modules/sharp/vendor/lib/libz.so electron ...

[jczaplew]

I am encountering this same issue with node v6.11.2 and sharp 0.18.2 on Ubuntu 14.04, but specifying the LD_PRELOAD path resolved the error. My full error is

````
module.js:597
return process.dlopen(module, path._makeLong(filename));
^

Error: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /path/to/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)
at Error (native)
at Object.Module._extensions..node (module.js:597:18)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at Object. (/path/to/node_modules/sharp/lib/constructor.js:9:15)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
````

Additionally, I noticed in another project that I was using sharp 0.11, and downgrading to that also resolved this error.

[Apophenia]

The LD_PRELOAD fix did not prevent the issue in Kubuntu 16.04, unless I was not using it correctly within the context of the Electron chain. :( I substituted similar functionality from another package.

[kylemacfarlane]

I ran into this same error on a forked project: https://github.com/stechstudio/libvips-lambda/issues/1

LD_PRELOAD can fix the error on EC2 but for some reason has no affect on Lambda.

I don't think it's a problem with node-canvas or Electron or anything because I'm using Python. I think it's due to how sharp builds libpng and zlib. You can see the problem merely by checking libpng.so with ldd:

> wget https://dl.bintray.com/lovell/sharp/libvips-8.5.5-linux-x64.tar.gz
> tar xzf libvips-8.5.5-linux-x64.tar.gz
> ldd lib/libpng.so
lib/libpng.so: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by lib/libpng.so)
        linux-vdso.so.1 =>  (0x00007ffd88afc000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fba554d8000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fba551d6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fba54e11000)
        /lib64/ld-linux-x86-64.so.2 (0x000055a0a04ff000)

This seems to happen because sharp builds libz.so 1.2.11 for distribution but has actually built libpng.so against 1.2.9. Meanwhile the system libz.so on the Amazon AMI is 1.2.8. Thus it doesn't work.

[kylemacfarlane]

For the question as to why libpng doesn't just use libz 1.2.11 when all it needs is 1.2.9 then it's probably because of stuff like this:

https://github.com/glennrp/libpng/search?utf8=%E2%9C%93&q=ZLIB_VERNUM+&type=

[lovell]

@kylemacfarlane Great research, thank you, it looks like the suggestion to add the --zprefix flag when compiling zlib will help us.

[kylemacfarlane]

--zprefix result in this while building:

curl: symbol lookup error: /usr/lib64/libcurl.so.4: undefined symbol: zlibVersion

But I would have thought that --with-zlib-prefix when building libpng is what we want anyway. However even that doesn't work.

Building shared library libz.so.1.2.11 with gcc.

... 5 mins later ...

-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.8") <-- why doesn't --with-zlib-prefix prevent this?

And yet:

lib/libpng.so: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by lib/libpng.so)

If the build scripts download 1.2.11 and the system only has 1.2.8 where is it even getting 1.2.9 from?

[kylemacfarlane]

Also when I built a lightweight version of libvips with only jpeg and png support over in https://github.com/stechstudio/libvips-lambda/issues/1#issuecomment-329893244 I didn't get any zlib problems. So is one of the other dependencies clobbering libpng/zlib?

[lovell]

@kylemacfarlane Sadly the --zprefix flag won't help as other dependencies that need zlib such as libgsf don't support it.

sharp's binding.gyp sets rpath on the sharp.node shared library so the runtime linker will search for shared library dependencies relative to it.

-Wl,--disable-new-dtags -Wl,-rpath=\'$${ORIGIN}/../../vendor/lib\'

Here's what the current sharp.node looks like via ldd/readelf:

$ readelf -d build/Release/sharp.node | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [${ORIGIN}/../../vendor/lib]
$ ldd build/Release/sharp.node
linux-vdso.so.1 =>  (0x00007ffeb5d45000)
libvips-cpp.so.42 => /tmp/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42 (0x00007f8217a8b000)
libvips.so.42 => /tmp/sharp/build/Release/../../vendor/lib/libvips.so.42 (0x00007f8217470000)
libglib-2.0.so.0 => /tmp/sharp/build/Release/../../vendor/lib/libglib-2.0.so.0 (0x00007f82170fc000)
libgobject-2.0.so.0 => /tmp/sharp/build/Release/../../vendor/lib/libgobject-2.0.so.0 (0x00007f8216ea7000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8216b25000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f821681c000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8216605000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f821623b000)
libpng16.so.16 => /tmp/sharp/build/Release/../../vendor/lib/libpng16.so.16 (0x00007f8216002000)
libz.so.1 => /tmp/sharp/build/Release/../../vendor/lib/libz.so.1 (0x00007f8215de9000)
...

I think the next step is to look at how Electron is loading shared libraries.

Electron accesses Node as a shared library, let's call it node.so. My best guess would be that something goes wrong with rpath or ORIGIN when the node.so shared library attempts to load the sharp.node shared library.

[kylemacfarlane]

I managed to fix it for my use case in about 10 mins this morning (plus like 60 mins of building).

1. The Found ZLIB: /usr/lib64/libz.so from above was coming from openjpeg which builds its own internal version of zlib. openjpeg isn't in the sharp builds.
2. I'm pretty sure that version `ZLIB_1.2.9' not found does mean it can't find >= 1.2.9.
3. Doing print(sys.modules.keys()) on Lambda shows that a lot of non-std modules have already been imported including zlib.
4. I think environment variables on Lambda must be set late for security reasons. So we can't use LD_PRELOAD/ LD_LIBRARY_PATH to affect what version of zlib Lamda loads on startup. I don't understand why they don't work for @Apophenia on a regular Kubuntu box.

Then the fix:

1. Rebuild with VERSION_ZLIB=1.2.8.
2. Make sure that the path to your libs is prepended to LD_LIBRARY_PATH. This will prevent similar problems with any other libs.
3. ldd on EC2 might still complain but just upload the libs to lambda and they will work. I wish I'd have ignored ldd as I actually rebuilt this way two days ago but never uploaded the files to Lambda because my upload speed sucks and it takes a couple minutes each time.

[lovell]

The following change to the Linux packaging script appears to fix this also.

-export LDFLAGS="-L${TARGET}/lib"
+export LDFLAGS="-L${TARGET}/lib -Wl,-rpath,'\$\$ORIGIN'"

...which produces:

$ readelf -d vendor/lib/libpng.so | grep rpath
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN]
$ ldd vendor/lib/libpng.so
linux-vdso.so.1 =>  (0x00007ffdb2460000)
libz.so.1 => /tmp/sharp/vendor/lib/libz.so.1 (0x00007f7edf43d000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7edf134000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7eded6a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7edf88f000)

The pre-compiled binaries for the future libvips v8.6.0 to be used by the future sharp v0.19.0 will have this change.

Thanks for everyone's help in reporting/debugging this.

[lovell]

@kylemacfarlane zlib v1.2.8 has multiple security vulnerabilities e.g. CVE-2016-9841, CVE-2016-9843, so watch out if using it with untrusted input.

[kylemacfarlane]

Unfortunately I can't find any other solution. I tried some crazy hacks to reload libz while Python was running but nothing would work for me.

I checked the release candidate for Amazon AMI 2017.09 and it's still using 1.2.8. This is probably because Red Hat has marked all the CVEs as "will not fix".

This does work though:

def lambda_handler(event, context):
    print(subprocess.check_output(['python3', 'lambda_function.py']))
    return 'Hello from Lambda'


if __name__ == '__main__':
    print('inside main')
    import pyvips
    pyvips.Image.black(100, 100)

It has a huge overhead and takes 300-400ms to run but at least it seems to confirm that the problem is the zlib import done by Lambda at launch.

[christophemarois]

Running on Heroku, I fixed the error by setting the env variable:
LD_PRELOAD="/app/node_modules/sharp/vendor/lib/libz.so"

And requiring sharp before any other dependency in the entry js file

[england9911]

@christophemarois Thanks, that solved the issue for me.

[manikmi]

I am running on my local ubuntu 14.04 VM. Setting up the environment variable using the following command did NOT work for me.
LD_PRELOAD="/app/node_modules/sharp/vendor/lib/libz.so"
However, copying the libz.so* to /lib/x86_64-linux-gnu/ worked for me
cp /app/node_modules/sharp/vendor/lib/libz.so* /lib/x86_64-linux-gnu/

[lovell]

@manikmi That command will replace the system libz so should come with a "please don't try this at home" warning.

[manikmi]

@lovell I don't know why setting the environment variable did not work. Any suggestions???

[sr9yar]

The Env method did not workfor me. So I replaced the system lib. This may not be the best way, but it works for sure.

[lovell]

sharp v0.19.0 with libvips v8.6.1 compiled using the approach mentioned in https://github.com/lovell/sharp/issues/892#issuecomment-329981594 is now available.

[bendrick92]

I have installed and confirmed Sharp v0.19.0, but am still seeing the 'ZLIB_1.2.9' error.

[lovell]

@bendrick92 Assuming Linux (your comment doesn't mention a platform) what output does the following generate for v0.19.0?

$ ldd node_modules/sharp/vendor/lib/libpng.so

On Ubuntu 16.04 I see:

 linux-vdso.so.1 =>  (0x00007ffea1177000)
 libz.so.1 => [...]/node_modules/sharp/vendor/lib/libz.so.1 (0x00007fc80294e000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc802645000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc80227b000)
 /lib64/ld-linux-x86-64.so.2 (0x00007fc802da0000)

Are there dependencies other than sharp that might be trying to include a system zlib first? Does adding require('sharp'); at the start of your script/app help?

[bendrick92]

@lovell, yes - I'm running Linux:

Distributor ID: Ubuntu
Description:    GalliumOS 2.1
Release:    16.04
Codename:   xenial

Running the command I get:

linux-vdso.so.1 =>  (0x00007ffc6ddfc000)
libz.so.1 => [...]/node_modules/sharp/vendor/lib/libz.so.1 (0x00007f667ed1c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f667e9fe000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f667e633000)
/lib64/ld-linux-x86-64.so.2 (0x000056310af02000)

I have tried moving the sharp require to the top of my script (it's an Electron app).

Please feel free to let me know if there's other info I can provide that will help you troubleshoot!

[lovell]

@bendrick92 Thanks, this confirms that if sharp's libpng.so is loaded first, then it will load its own libz.so before the system version.

Is the error you're seeing the same as the original report, namely ELECTRON_ASAR.js:173 Uncaught Error: /lib/x86_64-linux-gnu/libz.so.1: version 'ZLIB_1.2.9' not found (required by [...]/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)? Does it explicitly mention libpng16.so.16?

[bendrick92]

@lovell Here's the full error I'm seeing - looks to be referencing the same module:

Uncaught Error: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by [...]/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)
    at process.module.(anonymous function) [as dlopen] (ELECTRON_ASAR.js:173:20)
    at Object.Module._extensions..node (module.js:598:18)
    at Object.module.(anonymous function) [as .node] (ELECTRON_ASAR.js:173:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> ([...]/node_modules/sharp/lib/constructor.js:10:15)
    at Object.<anonymous> ([...]/hyde/node_modules/sharp/lib/constructor.js:253:3)

[lovell]

I don't know that much about Electron, sorry. If someone can create a repo with minimal dependencies that consistency reproduces this error on Linux then I'm happy to take a deeper look.

[bendrick92]

@lovell I set up a very basic Electron repo using Sharp (the only Node package used) here: https://github.com/bendrick92/electron-sharp-test

Simply launching the app npm start produces the error.

[christophemarois]

Just in case this could help: I have successfully set up sharp in electron with electron-builder by adding this to package.json:

{
  ...
  "build": {
    ...
    "asar": true,
    "asarUnpack": [
      "**/node_modules/sharp/**/*"
    ],
  }
}

[lovell]

The electron binary distributed by the electron package has a runtime dependency on the system libz:

$ ldd node_modules/electron/dist/electron | grep libz
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9f2074d000)

so there's nothing sharp can do here. Two possible options are:

1. Upgrade the system libz from the vulnerable v1.2.8 to the latest v1.2.11
2. Use LD_PRELOAD=/path/to/node_modules/sharp/vendor/lib/libz.so electron ... so the system will use the sharp-provided v1.2.11 instead of its own v1.2.8.

You might want to make an upstream request that the electron binaries link and ship with the latest libz, which will involve a change somewhere around here https://github.com/electron/electron/blob/fce84fbe99b3ea726297bb6de0d811173ffb6dea/electron.gyp#L211

Thanks @bendrick92 for the repo, which helped me discover the above. Good luck!

[bendrick92]

@lovell, thanks so much for looking into this. I'm in uncharted territory for my technical understanding here, but after some research it appears that I'm at the latest version for my distro's zlib:

zlib1g is already the newest version (1:1.2.8.dfsg-2ubuntu4.1).

For reference for others, as you noted there's a few options:

1. Open a request that Electron update their zlib to the newest version (request for update already opened for Electron here)
2. Modify my environment variables to allow Sharp's zlib to take precedence (seems like a temporary fix?)
3. Update my local zlib by manually downloading the newest version (more details on this option outlined here)
4. Wait for GalliumOS to update their distro to zesty (which appears to include zlib 1.2.11)

[bendrick92]

Success! I was able to get Sharp working with Electron by downloading and compiling the latest version of zlib using the following method (reference):

wget http://www.zlib.net/zlib-1.2.11.tar.gz
tar -xvzf zlib-1.2.11.tar.gz
cd zlib-1.2.11
./configure --prefix=/usr/local/zlib
make
sudo make install

After compiling and installing the updated zlib package, I had to rebuild Electron using electron-rebuild.

[jardakotesovec]

@lovell I also run into this issue. I am not using electron but getting same error when running tests via jest.. don't know what is loading libz before, but don't see much reason to investigate that.

Wanted to ask if this 1.2.9 is requirement by sharp and is it just because of the security issues?

Tricky is that some conservative distros like debian (which are often used in docker images) tends to backport the security fixes rather than updating version for long term stability. So it might take ages before newer libz shows up there. At the moment I am using that LD_PRELOAD workaround, but don't see how to properly solve it. Manually installing zlib does not make much sense in this case.

https://security-tracker.debian.org/tracker/source-package/zlib

[lovell]

@jardakotesovec zlib v1.2.8 is vulnerable to the "9.8 Critical" CVE-2016-9841, which is why the prebuilt binaries provided by sharp use a more recent version.

It looks like Debian have yet to backport zlib v1.2.9+ to Wheezy or Jessie so you could ask upstream if the reported severity of 9.8 might now justify doing so.

[jardakotesovec]

@lovell Wheeze and Jessie are older than current debian stable, so they are not maintained/updated anymore. Sid is the newest and its still on 1.2.8.. so it will probably take ages before any newer version gets to stable/testing version.

On the other hand if I look to ubuntu - they basically ignore those. So they have vulnerable libz in some of the current LTS versions, which is unfortunate. So in that scenario it can encourage ubuntu users complaint more if sharp is forcing 1.2.9+. So there is probably not win for all solution.

[jardakotesovec]

Now I see that jessie is still under LTS support, so I take that back. Thats odd that they did not address it there..

[kylemacfarlane]

@jardakotesovec I had the same problem with the Amazon AMI.

From what I remember the CVEs aren't actually exploits (it's more obvious if you read the audit that lead to the CVEs). They are "undefined behaviour" which depends on the compiler.

Red Hat, Ubuntu, etc have marked them as "won't fix" because they've confirmed the undefined behaviour doesn't exist with how they built zlib with gcc. Presumably sharp could also build a safe version of 1.2.8.

[lovell]

Should sharp provide zlib v1.2.8 then the reverse problem, where a dependency is expecting v1.2.11, can also occur.

The pre-compiled binaries of libvips and its dependencies are provided to make things significantly easier for most of the people most of the time. All the code to build these is open source - you can make your own bundle via the scripts at https://github.com/lovell/sharp-libvips/

In addition, sharp willl always look for and prefer a globally-installed version of libvips.