Sharp: Doesn't work with worker_threads

[Ugzuzg]

When running with worker threads, terminating a previous thread and trying to require('sharp') in a new worker thread again I'm getting the following output:

(sharp:47348): GLib-GObject-WARNING **: 13:22:02.604: cannot register existing type 'VipsObject'

(sharp:47348): GLib-CRITICAL **: 13:22:02.605: g_once_init_leave: assertion 'result != 0' failed

(sharp:47348): GLib-GObject-CRITICAL **: 13:22:02.605: g_type_register_static: assertion 'parent_type > 0' failed

(sharp:47348): GLib-CRITICAL **: 13:22:02.605: g_once_init_leave: assertion 'result != 0' failed

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?
Yes

What are the steps to reproduce?
https://github.com/Ugzuzg/sharp-worker-threads
Run node index.js

What is the expected behaviour?
sharp works with worker_threads.

Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?
https://github.com/Ugzuzg/sharp-worker-threads
It has a dependency on threads which uses worker_threads internally.

Are you able to provide a sample image that helps explain the problem?
No image required.

What is the output of running npx envinfo --binaries --system?

  System:
    OS: Linux 5.7 Arch Linux
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Memory: 2.43 GB / 15.28 GB
    Container: Yes
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.3.0 - /tmp/fnm-shell-4410774/bin/node
    Yarn: 1.22.4 - /usr/bin/yarn
    npm: 6.14.5 - /tmp/fnm-shell-4410774/bin/npm

[lovell]

Hi, you'll need to ensure you require('sharp') in the main thread as well as worker threads.

Shared libraries are opened via dlopen and closed via dlclose on a reference counting basis. The use of terminate on a thread can result in dlclose being called on sharp, libvips and all of its dependencies. However the glib dependency does not respond to dlclose, so when dlopen is called a second time for sharp, libvips and all of its dependencies, they try to initialise themselves but it will fall over.

By requiring sharp in your main thread, you ensure the reference counter for all of its dependencies will always be at least 1, ensuring dlclose is only called after all work is complete.

[lovell]

You can watch shared libraries opening "init" and closing "fini" as a program runs using:

LD_DEBUG=files G_DEBUG=fatal-warnings node index.js 2>&1 | egrep "calling (init|fini)"

[Ugzuzg]

Thanks, requiring sharp in the main thread does delay the closing to the end of execution, as you say. Any benefit of adding this information to the documentation?

I noticed that there is no such problem on Alpine Linux (using musl libc).

[lovell]

Docs updated via commit c91373f, thanks for the suggestion.

The dlclose function is a "no-op" in musl libc, which is rather sensible, hence Alpine won't exhibit this behaviour.

https://wiki.musl-libc.org/functional-differences-from-glibc.html#Unloading_libraries

[ericman314]

This suggestion is working for me too, _unless_ the worker is terminated while converting an svg to a png and saving the png:

await sharp(Buffer.from(svgString))
  .toFile(fnPng)

If the main thread terminates the worker while the worker is executing the line above, then the entire main thread crashes with this:

terminate called after throwing an instance of 'Napi::Error'
  what():  
Aborted (core dumped)

I'm hoping you might recognize this error and know what I'm doing wrong. If you need me to provide code to reproduce this, or open a new issue, I would be happy to do that.

[lovell]

@ericman314 You should not terminate in-flight worker threads where native modules are involved. See https://github.com/nodejs/node/issues/34567 for more context, which includes the recent addition of experimental (N-)APIs to Node.js that we won't be able to use until at least Node.js 10 reaches EOL, possibly later.