Sharp: Possible libgsf race condition with concurrent tile-based output

[BiancoA]

following from this issue https://github.com/lovell/sharp/issues/678

I set:

"build": {
    "asar": true,
    "asarUnpack": ["**/node_modules/sharp/**/*"]
  }

in my package.json. I packaged the app with yarn dist. This seems to solve something but now my App is randomly crashing (white screen) and I get this error:

(sharp:24414): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfile'

(sharp:24414): GLib-GObject-CRITICAL **: g_type_register_static: assertion 'parent_type > 0' failed

(sharp:24414): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfile'

(sharp:24414): GLib-GObject-CRITICAL **: g_object_new_valist: assertion 'G_TYPE_IS_OBJECT (object_type)' failed

(sharp:24414): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfile'

(sharp:24414): GLib-GObject-WARNING **: invalid cast from 'GsfOutfileStdio' to '<invalid>'

my set up is:

$ yarn --version
1.5.1
     "sharp": "0.19.1"
    "electron": "1.8.3",
    "electron-builder": "20.3.1",

Final note: this does not happen in dev mode

[BiancoA]

Update: with sharp version 0.20.0 I don't neet to set "asarUnpack", but the app still randomly crashes with:

(sharp:36854): GLib-GObject-WARNING **: cannot register existing type 'GsfOutput'

(sharp:36854): GLib-GObject-CRITICAL **: g_type_register_static: assertion 'parent_type > 0' failed

(sharp:36854): GLib-GObject-CRITICAL **: g_type_register_static: assertion 'parent_type > 0' failed

(sharp:36854): GLib-GObject-CRITICAL **: g_object_new_valist: assertion 'G_TYPE_IS_OBJECT (object_type)' failed

the same happens setting "asarUnpack": ["**/node_modules/sharp/**/*"]

[lovell]

Hello, are there any other native dependencies involved that might rely on glib e.g. canvas?

[BiancoA]

Hello,

canvas is not in the node_modules directory, It may be some other cross-dependency that I don't know about, a quick search returns: detect-libc that is used by node-pre-gyp, that should not count and sharp of course

[lovell]

What does npm ls nan and npm ls node-pre-gyp display? Those should list any native dependencies and their tree.

[BiancoA]

├─┬ [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └─┬ [email protected]
│       └── [email protected] 
└── [email protected]  extraneous

├─┬ [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └─┬ [email protected]
│       └── [email protected]  deduped
└─┬ [email protected]
  └── [email protected]  deduped

[lovell]

Thank you, no obvious conflicts there. My best guess would be that the electron binary itself is dependent on a older glib shared library that is being loaded before sharp. What does ldd node_modules/electron/dist/electron | grep libg say?

[BiancoA]

I am under Mac OS X so i guess the equivalent would be:

$ otool -L node_modules/electron/dist/Electron.app/Contents/MacOS/Electron 
node_modules/electron/dist/Electron.app/Contents/MacOS/Electron:
    @rpath/Electron Framework.framework/Electron Framework (compatibility version 0.0.0, current version 0.0.0)
    /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 157.0.0)
    /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.10.0)
    /System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 57031.20.26)
    /System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface (compatibility version 1.0.0, current version 55058.0.0)
    @rpath/Squirrel.framework/Squirrel (compatibility version 1.0.0, current version 1.0.0)
    @rpath/ReactiveCocoa.framework/ReactiveCocoa (compatibility version 1.0.0, current version 1.0.0)
    @rpath/Mantle.framework/Mantle (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

no occurrences found

[lovell]

Thanks, nothing obviously wrong there either.

Do all the glib-related results from running otool -L node_modules/sharp/build/Release/sharp.node reference files in the sharp/vendor/lib directory?

[BiancoA]

Thanks for the help debugging!

it seems so:

node_modules/sharp/build/Release/sharp.node:
    @rpath/libvips-cpp.42.dylib (compatibility version 51.0.0, current version 51.1.0)
    @rpath/libvips.42.dylib (compatibility version 51.0.0, current version 51.1.0)
    @rpath/libglib-2.0.0.dylib (compatibility version 5401.0.0, current version 5401.3.0)
    @rpath/libgobject-2.0.0.dylib (compatibility version 5401.0.0, current version 5401.3.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

[lovell]

Please can you try setting the DYLD_PRINT_LIBRARIES and/or DYLD_PRINT_LIBRARIES_POST_LAUNCH environment variables to list all the shared library files as they are opened.

https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/LoggingDynamicLoaderEvents.html

[BiancoA]

it is really a lot of text, so i used pastebin:

https://pastebin.com/vnpbHYch

[BiancoA]

I had an idea, somewhere in my code I am using something like:

    console.log("tiling with sharp");
    for (var i = 0; i < nOfImages; i++) {
      arrayOfPromises.push(useSharpToCreateTiles(imagePath[i],options));
    }
  return Promise.all(arrayOfPromises);

where

function useSharpToCreateTiles(imagePath,options){
return sharp(inputImgPath)
    .tile({
      size: 512,
      layout: 'google'
    })
    .toFile(sharpDir)
    .then(info => {
      doSomethingElse();
      return Promise.resolve(info)
    });
}

This could be a problem if two sharp instances call at the same time gLib. I thought that this should not be an issue since javascript is not multithreaded, but maybe?

In this repository https://github.com/BiancoA/electron-webpack-quick-start I am doing some tests, as soon as I get something more specific I post it here

[BiancoA]

I was able to replicate the issue with the code here: https://github.com/BiancoA/electron-webpack-quick-start/blob/master/src/renderer/index.js

the problem seems to be in the

 for (var i = 0; i < 10; i++) {
          arrayOfPromises.push(sharp(inputImgPath)...
...

Am I misusing sharp? Should I wait for each promise to complete before initialising a new one?

regarding the problem with the .asar package: I think it may be an issue in electron-builder, so i created an issue here: https://github.com/electron-userland/electron-builder/issues/2679

[develar]

@lovell Am I right that sharp uses external libs (vendor/lib/*/.dylib)? And file sharp.node needs these files (not static)?

[lovell]

The invalid class cast from 'GsfOutput' to 'GsfOutput' error initially suggested there are conflicting versions of shared libraries involved. However I can't see any obvious conflicts in the provided logs, and given there are multiple writes occurring concurrently (sharp uses libuv-managed threads and libvips manages its own threads), this is starting to look like a thread safety problem.

Perhaps there is a race condition during library initialisation within libgsf itself? https://github.com/GNOME/libgsf/blob/master/BUGS#L19 suggests this is a possibility.

The next step is probably to remove the sharp/node layer and see if this problem can be recreated using libvips at the command line.

[lovell]

@develar Yes, sharp downloads platform-specific shared libraries (.dylib, .so, .dll) and stores them in node_modules/sharp/vendor at npm install time.

[jcupitt]

Hello, I tried looking inside libvips:

john@kiwi:~/pics$ gdb vips
Reading symbols from vips...done.
(gdb) break gsf_output_get_type
Function "gsf_output_get_type" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (gsf_output_get_type) pending.
(gdb) run dzsave k2.jpg x2
Starting program: /home/john/vips/bin/vips dzsave k2.jpg x2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, 0x00007ffff555d4a0 in gsf_output_get_type ()
   from /usr/lib/x86_64-linux-gnu/libgsf-1.so.114
(gdb) bt
#0  0x00007ffff555d4a0 in gsf_output_get_type ()
    at /usr/lib/x86_64-linux-gnu/libgsf-1.so.114
#1  0x00007ffff5561e35 in gsf_outfile_get_type ()
    at /usr/lib/x86_64-linux-gnu/libgsf-1.so.114
#2  0x00007ffff5563ff5 in gsf_outfile_stdio_get_type ()
    at /usr/lib/x86_64-linux-gnu/libgsf-1.so.114
#3  0x00007ffff55640d5 in gsf_outfile_stdio_new_valist ()
    at /usr/lib/x86_64-linux-gnu/libgsf-1.so.114
#4  0x00007ffff556423f in gsf_outfile_stdio_new_full ()
    at /usr/lib/x86_64-linux-gnu/libgsf-1.so.114
#5  0x00007ffff78e3981 in vips_foreign_save_dz_build (object=0x555555873000)
    at dzsave.c:1824

So the type is being created on demand by libvips the first time it tries to dzsave. This will obviously race if two threads try to dzsave at the same time.

Looking at libgsf, you'd think this would be created by gsf_init(), which libvips is careful to call single-threaded, but it's obviously not. I'll look a bit deeper.

[jcupitt]

I built libgsf 1.14.42 from source and it looks like it's fixed there. 1.14.41 and earlier seem to have this problem.

john@kiwi:~/pics$ gdb vips
Reading symbols from vips...done.
(gdb) break gsf_output_get_type
Function "gsf_output_get_type" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (gsf_output_get_type) pending.
(gdb) run dzsave k2.jpg x
Starting program: /home/john/vips/bin/vips dzsave k2.jpg x
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, gsf_output_get_type () at gsf-output.c:254
254 GSF_CLASS_ABSTRACT (GsfOutput, gsf_output,
(gdb) bt
#0  0x00007ffff5558170 in gsf_output_get_type () at gsf-output.c:254
#1  0x00007ffff5539d0d in gsf_init_dynamic (module=0x0) at gsf-utils.c:202
#2  0x00007ffff5539bfb in gsf_init () at gsf-utils.c:127
#3  0x00007ffff79586e8 in vips_init (argv0=0x7fffffffdf80 "/home/john/vips/bin/vips") at init.c:441
#4  0x0000555555558718 in main (argc=4, argv=0x7fffffffdc08) at vips.c:1095

So init is happening in init (as it should).

I've just uploaded some new Windows binaries for libvips 8.6.3 which (by happy coincidence!) are built against libgsf-1.14.42. Hopefully that'll fix it. Perhaps homebrew needs a nudge as well?

[lovell]

Great detective work, thank you @jcupitt.

Homebrew has provided libgsf v1.14.42 since 12th Nov last year according to https://github.com/Homebrew/homebrew-core/commit/ae4c957c12ae272e4582333e672734df9073365f

The pre-compiled binaries for sharp v0.19.x and v0.20.x also provide libgsf v1.14.42 - console.log(sharp.versions); can be used to confirm this.

This means the behaviour @BiancoA is experiencing may not be related, or at least this has now become even more confusing.

[jcupitt]

Well ... I ... erm, yes I'm baffled too.

Could another electron package be pulling in an older libgsf?

[BiancoA]

Could the problem be caused by the node version? electron ships with Node 8.2.1, that means node_module version 57.

[lovell]

@BiancoA I don't think so, but then again I'm really not sure what's going on here at all :(

[jcupitt]

Hello again, would it be possible to get a stack dump from the error? This should help to decide who is doing the second gsf_init().

You can make libvips warnings fatal like this:

$ G_ERROR=fatal-warnings gdb myapp

(I forget what the macOS debugger is called). This should make it stop on the first warning, and you should then be able to get a stack dump.

[lovell]

@BiancoA Were you able to make any progress with this?

[BiancoA]

Hi, sorry for the delay in answering. I didn't do much more investigation on the issue unfortunately. I hope to have some time in coming months. To avoid the problem I implemented something like:

for (var i = 0; i < fileNames.length; i++) {
        var v = await doTilingWithSharp(...);
      }

where

async function doTilingWithSharp(...){
  let tileOpt = {
    size: tileSize.w,
    layout: 'google'
  };
return sharp(inputImgPath)
    .tile(tileOpt)
    .toFile(outDir)
}

[lovell]

@BiancoA Thanks for the update, I'll add something to the docs about this.

[lovell]

I've added a comment to the docs about this in commit ddc2349

I'll close this issue for now but please feel free to re-open or comment further when anyone experiencing this can provide a stack trace of the problem occurring.

[dlong500]

Just for reference I've come across basically the same issue on Windows since I started using sharp. I don't have a stack trace yet but it is really odd because it only intermittently occurs.

[dlong500]

I'm definitely still getting messages like these on the latest versions with Windows. It's odd because even if I'm processing lots of images, it either seems to crash at the beginning or not at all, but it probably crashes about 60% of the time I try to start processing a set of images with output like so:

(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutput'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfile'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfileStdio'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutput'
(sharp:5540): GLib-GObject-WARNING **: cannot retrieve class for invalid (unclassed) type '<invalid>'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfile'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutput'
(sharp:5540): GLib-GObject-WARNING **: invalid class cast from 'GsfOutfileStdio' to '<invalid>'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutput'
(sharp:5540): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfile'
error Command failed with exit code 3221225477.

[dlong500]

@lovell I'm having difficulty trying to confirm what version of libgsf is being used on my system. console.log(sharp.versions) only outputs { vips: '8.6.1' } for me. Can you provide any more insight?

[lovell]

@dlong500 On Windows you'll probably be using libgsf v1.14.42. I've created https://github.com/lovell/sharp-libvips/issues/6 for the missing versions.json.

[dlong500]

I finally got around to making a small project to try and test this issue (as well as another #1232). Here's the repo: https://github.com/dlong500/sharp-glib-testing

There is some general info about the issues on the README, and I've also managed to get some backtrace data (on an Ubuntu 16.04 machine), but I'm not an expert at interpreting that data. I ran a concurrent deepzoom generation test (see the repo) with the following command:
G_DEBUG=fatal-warnings gdb --args /usr/bin/node entry.js deepzoom

And got the following backtrace data:

(sharp:6398): GLib-GObject-WARNING **: 23:43:46.199: cannot register existing type 'GsfOutput'
[New Thread 0x7fffcf47b700 (LWP 6411)]

Thread 9 "node" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 0x7fffde3fe700 (LWP 6409)]
0x00007fffe7ce8edc in g_logv () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libglib-2.0.so.0
(gdb) bt
#0  0x00007fffe7ce8edc in g_logv () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libglib-2.0.so.0
#1  0x00007fffe7ce9072 in g_log () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libglib-2.0.so.0
#2  0x00007fffe7a6ad65 in ?? () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgobject-2.0.so.0
#3  0x00007fffe7a6add4 in g_type_register_static ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgobject-2.0.so.0
#4  0x00007fffe6aa2c7d in gsf_output_get_type ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgsf-1.so.114
#5  0x00007fffe6aa7595 in gsf_outfile_get_type ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgsf-1.so.114
#6  0x00007fffe6aa9cd5 in gsf_outfile_stdio_get_type ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgsf-1.so.114
#7  0x00007fffe6aa9de5 in gsf_outfile_stdio_new_valist ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgsf-1.so.114
#8  0x00007fffe6aa9f72 in gsf_outfile_stdio_new_full ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libgsf-1.so.114
#9  0x00007ffff4221c7f in ?? () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#10 0x00007ffff421eeb1 in ?? () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#11 0x00007ffff425c748 in vips_object_build ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#12 0x00007ffff4268758 in vips_cache_operation_buildp ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#13 0x00007ffff46e835c in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42
#14 0x00007ffff46eeb6d in vips::VImage::dzsave(char*, vips::VOption*) ()
   from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42
#15 0x00007ffff4930ae5 in PipelineWorker::Execute() () from /opt/nodejs/sharp-glib-testing/node_modules/sharp/build/Release/sharp.node
#16 0x00000000009e3721 in ?? ()
#17 0x00007ffff6f146ba in start_thread (arg=0x7fffde3fe700) at pthread_create.c:333
#18 0x00007ffff6c4a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Is this helpful? I'm happy to do anything you'd like me to do to see if we can narrow down the problem.

[jcupitt]

Hello @dlong500, that's great! Thanks for working on this.

Could you be using an older libgsf? Before 1.14.42, libgsf didn't initialize all of its types on startup and so would race if used in parallel. On Ubuntu 16.04, you'll need to build the latest libgsf from source, and then rebuild libvips against that. Perhaps you've done this already.

If you see the problem on libgsf 1.14.42 or later, then sharp must (I think?) be calling vips functions before vips_init() has finished. The timeline ought to be:

main thread             worker thread
===========             =============

enter vips_init()
enter gsf_init()
gsf_output_get_type()
exit gsf_init()
exit vips_init()

                        call dzsave

But I think this race must mean that this is happening:

main thread             worker thread
===========             =============

enter vips_init()
enter gsf_init()        call dzsave
gsf_output_get_type()
exit gsf_init()
exit vips_init()

To prove or disprove this, we need to add logging to vips_init() to verify the order of invocation. Here's the start of vips_init():

https://github.com/jcupitt/libvips/blob/master/libvips/iofuncs/init.c#L286

Add something like:

        g_printf( "vips_init: starting\n" );
        g_printf( "vips_init: done\n" );

To the start and end of that function. Do the same for _save_dz_buiild():

https://github.com/jcupitt/libvips/blob/master/libvips/foreign/dzsave.c#L1615

Then run and look at the output to see the order of operations. If build ever starts before init has done, we've found the race.

If that's NOT happening, then libgsf must still be buggy somehow. I've read gsf_init() a couple of times and I don't see how GsfOutput can not be created :(

[dlong500]

Right now it's just using the pre-compiled binaries, as I don't have a globally installed version of libvips on this machine, so I think it is using libgsf 1.14.42 as shown with sharp.versions outputting:

{ cairo: '1.14.12',
  croco: '0.6.12',
  exif: '0.6.21',
  expat: '2.2.5',
  ffi: '3.2.1',
  fontconfig: '2.12.6',
  freetype: '2.9',
  gdkpixbuf: '2.36.11',
  gif: '5.1.4',
  glib: '2.55.1',
  gsf: '1.14.42',
  harfbuzz: '1.7.4',
  jpeg: '1.5.3',
  lcms: '2.9-',
  orc: '0.4.28',
  pango: '1.41.0',
  pixman: '0.34.0',
  png: '1.6.34',
  svg: '2.42.0',
  tiff: '4.0.9-cda4b06',
  vips: '8.6.1',
  webp: '0.6.1',
  xml: '2.9.7',
  zlib: '1.2.11' }

I'll have to compile libgsf and libvips with the statements to see the order of operations. Thanks for the help! I'll report back hopefully later today.

[dlong500]

Ok, I've done some preliminary testing, and I'm not sure quite what to make of the output yet. Should vips_init() be running hundreds/thousands of times even for just a small number of files?

I've pasted the output of a run that failed into a gist here. I added the diagnostic statements to libvips, but haven't added any diagnostic code to libgsf yet. Let me know if that gives you any insight or if you'd like me to take any additional specific steps.

[jcupitt]

Oh heh I'd forgotten, yes, it's called a lot. Just leave the done one.

That's baffling. init is being called correctly, but gsf is obviously not initing.

Could you add a printf to gsf_init_dynamic() to verify that that is being triggered? It should be in gsf-utils.c:

https://github.com/GNOME/libgsf/blob/master/gsf/gsf-utils.c#L176

Right at the start of that thing.

[dlong500]

Ok, I added a printf to gsf_init_dynamic() and it does appear to call it once at the very beginning. Here's the output:

to be executed: node entry.js deepzoom
gsf_init_dynamic: starting
vips_init: done
sharp.versions: { vips: '8.6.5' }
-> Removed existing output content (if present) in preparation for image processing
Attempting to generate deepzoom data for images in the input folder...
The resultant data will be saved in: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom
vips_foreign_save_dz_build: starting
vips_foreign_save_dz_build: starting
vips_foreign_save_dz_build: starting
vips_foreign_save_dz_build: starting
vips_foreign_save_dz_build: about to call gsf_outfile_stdio_new() for: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/339H-zoom-7LV3MZ
vips_foreign_save_dz_build: about to call gsf_outfile_stdio_new() for: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/379H-zoom-FMV3MZ

(sharp:12302): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfileStdio'

(sharp:12302): GLib-GObject-CRITICAL **: g_object_new_valist: assertion 'G_TYPE_IS_OBJECT (object_type)' failed
Segmentation fault
error Command failed with exit code 139.

It looks like it attempts to start generating zoom data for 4 images, and two temporary folders get created before the crash. That matches what I'm seeing in the output folder after the crash (two temporary folders).

[jcupitt]

Reading the libgsf sources, they have their own thinig to make new types:

https://github.com/GNOME/libgsf/blob/master/gsf/gsf-impl-utils.h#L49

And they are using G_UNLIKELY() to test for repeated invocation. Perhaps that doesn't work well between threads on some platforms?

Could you try removing that? Make it just:

        static GType type = 0;                                          \
        if (type == 0) {                                  \
                static GTypeInfo const object_info = {                  \

You need the backslashes. I inserted a print before the register as well:

                };                                                      \
                printf( #name ": init\n" ); \
                type = g_type_register_static (parent_type, #name,      \

On dzsave I see:

$ vips dzsave k2.jpg x
GsfInput: init
GsfInputGZip: init
GsfInputHTTP: init
...
GsfClipData: init
GsfDocMetaData: init
GsfDocPropVector: init
vips_init: done
$

ie. all the Gsf types are built in init, as you'd hope, and no types are built twice.

[dlong500]

Ah, so it looks like there _is_ something odd going on here.

The only type I see being built before vips_init: done is GsfDocPropVector. As you can see in the output below there are other types being constructed later, including a duplicate for GsfOutfileStdio (which, of course, explains the warning):

to be executed: node entry.js deepzoom
gsf_init_dynamic: starting
GsfDocPropVector: init
vips_init: done
sharp.versions: { vips: '8.6.5' }
-> Removed existing output content (if present) in preparation for image processing
Attempting to generate deepzoom data for images in the input folder...
The resultant data will be saved in: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom
vips_foreign_save_dz_build: starting: 343H-zoom
vips_foreign_save_dz_build: starting: 373H-zoom
vips_foreign_save_dz_build: starting: 339H-zoom
vips_foreign_save_dz_build: starting: 379H-zoom
vips_foreign_save_dz_build: about to call gsf_outfile_stdio_new() for: 373H-zoom
vips_foreign_save_dz_build: about to call gsf_outfile_stdio_new() for: 343H-zoom
gsf_outfile_stdio_new: tag: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/373H-zoom-PR29MZ
gsf_outfile_stdio_new_full: starting: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/373H-zoom-PR29MZ
gsf_outfile_stdio_new: tag: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/343H-zoom-FQ29MZ
gsf_outfile_stdio_new_full: starting: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/343H-zoom-FQ29MZ
gsf_outfile_stdio_new_valist: starting
gsf_outfile_stdio_new_valist: starting
gsf_outfile_stdio_new_valist: about to call g_object_new_valist(): /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/343H-zoom-FQ29MZ
GsfOutfileStdio: init
GsfOutfile: init
GsfOutput: init
gsf_outfile_stdio_new_valist: about to call g_object_new_valist(): /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/373H-zoom-PR29MZ
GsfOutfileStdio: init
vips_foreign_save_dz_build: about to call gsf_outfile_stdio_new() for: 339H-zoom
gsf_outfile_stdio_new: tag: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/339H-zoom-KF29MZ
gsf_outfile_stdio_new_full: starting: /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/339H-zoom-KF29MZ
gsf_outfile_stdio_new_valist: starting

(sharp:25490): GLib-GObject-WARNING **: cannot register existing type 'GsfOutfileStdio'

(sharp:25490): GLib-GObject-CRITICAL **: g_object_new_valist: assertion 'G_TYPE_IS_OBJECT (object_type)' failed
gsf_outfile_stdio_new_valist: about to call g_strdup(): /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/373H-zoom-PR29MZ
gsf_outfile_stdio_new_valist: about to call g_object_new_valist(): /opt/nodejs/sharp-glib-testing/testdata/output/deepzoom/339H-zoom-KF29MZ
GsfOutfileStdio: init
vips_foreign_save_dz_build: about to call gsf_outfile_stdio_new() for: 379H-zoom
Segmentation fault
error Command failed with exit code 139.

And just to make sure there wasn't something really strange going on with how sharp was interacting with vips, I called vips directly and still get the types being constructed later.

[jcupitt]

How completely bizarre. Well, that's certainly the race -- everything should be built before vips_init() finishes.

Could you try without the G_UNLIKELY()?

[dlong500]

That last output was with G_UNLIKELY() removed. I wish I could be more help without having to bug you; I'm just not sure what to check next.

[jcupitt]

Huh well that's even crazier. They are using a REGISTER macro to start types up, I'll try reading that.

[jcupitt]

Could it be an optimiser bug? It's generating code like this:

t = gsf_input_get_type();
...
t = gsf_doc_meta_data_get_type();
t = gsf_docprop_vector_get_type();
if (t == 0) { start up error ...; }

You are only seeing gsf_docprop_vector_get_type() being called, so perhaps the optimiser is throwing away all the earlier calls, since they are overwriting the value of t.

Could you try setting t = 0; before the first REGISTER in gsf_init_dynamic(), and then changing the definition of REGISTER from

#define REGISTER(prefix)        t = prefix ## _get_type()

To:

#define REGISTER(prefix)        t |= prefix ## _get_type()

ie. OR all the get types together. That would force the optimiser to keep all the inits.

[dlong500]

I think you may be onto something. Making that last change appears to correctly build all types at the beginning now. So, if it's a compiler optimization bug where do we go from here?

[jcupitt]

Wow! OK, that's great!

Try building the original libgsf, but with the optimiser turned off, so no -O flag to gcc or whatever compiler you are using.

[dlong500]

Ok, I do think that fixes the issue, though I'll need to do some more extensive testing because of the intermittent nature of the problem to be absolutely sure. But now the question is how do we get the pre-compiled binaries for all platforms fixed?

[dlong500]

That was a very fascinating process! I'm glad the time spent produced a good result. I certainly don't think I could have gotten there by myself, so thanks immensely for the help!

[jcupitt]

That's very exciting! Well done for persevering with this.

It works fine on current Ubuntu with gcc 7.3, so I guess it's just a problem in older versions. Do you know which gcc you are using?

[dlong500]

On my Ubuntu 16.04 machine gcc is at version 5.4.0:
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

But the pre-compiled version of sharp + libvips + other dependencies that is downloaded when installing sharp on a system that doesn't have a global version of libvips present still exhibits the issue. That is true on both my Ubuntu machine when I install sharp without a global libvips present as well as on Windows. Does this mean that something needs to be updated at lovell/sharp-libvips?

EDIT: or maybe it's just that the system @lovell uses to actually build the pre-compiled binaries needs to have a more recent version of gcc?

[jcupitt]

I make the main vips win binaries in Ubuntu 18.04, though you're right, I think the dockerised version is still on 16.04.

I'll try updating that.

[dlong500]

Would that resolve a Windows build too or do we need to separately test how Microsoft's MSVC compiler handles the optimizations?

[jcupitt]

You could try dropping these binaries into sharp:

https://github.com/jcupitt/libvips/releases/download/v8.6.5/vips-dev-w64-web-8.6.5.zip

That's built on 18.04 and should have this compiler fix.

[jcupitt]

sharp uses MSVC to build the libvips C++ API, but it relies on the gcc-built libvips.dll for the actual image processing code.

[dlong500]

Hmm, I'm not having a whole lot of luck trying to use those binaries (I think it messes with the sharp.node compiled module):

to be executed: "node entry.js deepzoom"
internal/modules/cjs/loader.js:718
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: The specified module could not be found.
\\?\C:\local\code\sharp-glib-testing\node_modules\sharp\build\Release\sharp.node

But I'm OK to wait and see if @lovell can offer insight or update the binaries he includes with sharp.
EDIT: How exactly are the binaries that come with sharp being generated? Are you producing them @jcupitt or is it @lovell ?

[jcupitt]

OK, I updated the builder to use latest stable ubuntu in the container, and fixed it up a bit. Lovell will need to update his builder as well.

[lovell]

Amazing detective work, thank you both.

The pre-compiled Linux x64 libvips binaries provided by sharp are currently created on Debian 7 and therefore gcc v4.7.2. This places a minimum glibc version of v2.13. Upgrading to Debian 8 would then depend on glibc v2.19 so exclude support for RHEL/Centos 7 still on v2.17, so we can't do that quite yet.

Luckily it looks like Debian 7 has a backport of gcc v4.9.2 so we can at least upgrade to that for the next major release. I've created https://github.com/lovell/sharp-libvips/issues/7 to track this.

The change John made in https://github.com/jcupitt/build-win64/commit/43595b9de4d03fcd90e2fd4f82f81b257d0d8f87 will be picked up by the sharp-libvips scripts when they package the forthcoming libvips v8.7.0.

[jcupitt]

OK, sounds good Lovell.

Summary: the fix until you can get to gcc 7 or later is to disable the optimiser for libgsf.

I'll see if libgsf are still using that REGISTER macro. They might be interested in this.

[jcupitt]

Yes, libgsf still have this code in master. I opened an issue:

https://gitlab.gnome.org/GNOME/libgsf/issues/10

[lovell]

Could setting the (undocumented?) _GSF_GTYPE_THREADING_FIXED define help?

https://github.com/GNOME/libgsf/blob/79e00674d47817778f233507e7e769cb1906b46e/gsf/gsf-utils.c#L157-L167

[jcupitt]

It won't compile with that flag enabled -- the GSList types is not defined anywhere.

I think optimiser off is probably the only fix for now.

[jcupitt]

... off for libgsf, I mean. It's fine to use elsewhere, of course.

[jcupitt]

The gsf maintainers have fixed it their end, so the next libgsf release should be fine, on this issue at least.

[kleisauke]

libgsf 1.14.44 has been released that should solve this issue.

Note for build-win64:
This patch should be included (see PR @ libgsf) otherwise it will cause a build error with MinGW.