Sdk: dart:ffi GC finalizers

[dcharkes]

Expose finalizers in Dart.

/// Return a pointer object that has a finalizer attached to it. When this
/// pointer object is collected by GC the given finalizer is invoked.
///
/// Note: the pointer object passed to the finalizer is not the same as 
/// the pointer object that is returned from [finalizable] - it points
/// to the same memory region but has different identity. 
Pointer<T> finalizable<T>(Pointer<T> p, void finalizer(Pointer<T> ptr)) {

}

Update 2020-06-26:
Finalizers are now available in native code (not yet exposed in Dart) on Dart master:

d.lookupFunction<Handle Function(Handle),
                 Object Function(Object)>("PassObjectToC");

static void RunFinalizer(void* isolate_callback_data,
                         Dart_WeakPersistentHandle handle,
                         void* peer) {
  printf("Running finalizer for weak handle.\n");
}

extern "c" Dart_Handle PassObjectToC(Dart_Handle h) {
  void* peer = 0x0;
  intptr_t size = 8;
  Dart_NewWeakPersistentHandle_DL(
      handle_2, peer, size, RunFinalizer);
}

Use dynamic linking (the _DL suffix) so that the symbols are available in Flutter. See documentation and samples on 7eac9f355ed54cb0aab018814f776c08d20cef44.

See more documentation in native_api.h and dart_api_dl.h.

Caveats:

• https://github.com/dart-lang/sdk/issues/42312 Trying to delete the finalizer or getting the object out of it while the object is unreachable will race the GC. See details in issue.
• The dart optimiser can inline method bodies and fields, which means objects can in some cases be GCed (and their finalizer run) before all methods are done executing. When we expose finalizers in Dart, we will have a solution for this. Workaround: do an ffi call with a Dart_Handle passing that object to native, that will keep it alive.

[dcharkes]

As discussed with @jonasfj and @mkustermann, maybe it would be better to provide a Pointer to a C function as finalizer, as the GC does currently not support running Dart code for finalization.

class Pointer {
  attachFinalizer(Pointer<NativeFunction<void Function(Pointer)>> finalizerPointer) {

  }
}

[jonasfj]

You might also want to include a sizeHint so we can signal an approximate size of the object to the GC.
Even if I don't know the exact size of the object I'm holding on to, I know if it's approximately 12 bytes, 12kb, or 12mb give or take a few orders of magnitude.

[sjindel-google]

We should implement this in the initial release since so many users seem to need it.

[sjindel-google]

I think we should estimate 3 weeks for this, given that we need:

• Support for attaching finalizers to Pointer (based on existing finalizer support).
• Maintaining a reference to the original from derived pointers.
• Maintaining a reference to the original from external typed data.

[yulingtianxia]

We should implement this in the initial release since so many users seem to need it.

Yeah, we really need it.

[ds84182]

Looking at https://dart-review.googlesource.com/c/sdk/+/123662, I don't think this solves all issues, especially with long lived structures or strings in a static method.

For example, libpng's png_create_read_struct needs the version string of libpng from link-time to ensure that ABI-incompatible changes haven't occurred (https://github.com/glennrp/libpng/blob/libpng16/png.h#L922). Yes, you could also wrap the function with another C library to forward the version correctly, but then you have to make sure that Dart, the auxiliary native library, and libpng are all in sync and are ABI compatible.

I'm just hoping that that CL isn't the final design for pointer finalizers.

[derolf]

When will this feature be available?

[dcharkes]

Somewhere in the next quarters, I cannot give you a precise date.

[VoLaNg]

Hi! How's it going? I wrote C interop wrapper for LevelDB, but I can't finish it, because I strongly depend on this issue.

[leecommamichael]

I'm very excited for this, as it gets us closer to using the Realm database with Flutter, which would make it far more production ready for my teams' skill sets.

[najibghadri]

+1

[jobweegink]

really need this for Realm Database with flutter!

[aureliopetrone]

+1

[yulingtianxia]

+1

[sharp3dges]

+1

[UmarBhutta]

+1

[mraleph]

Please do not post empty comments like +1 on this issue (or any other issues to that matter). If you want to highlight the interest just 👍 the top comment.

That said - we understand very well the importance of this issue and the work on it has been ongoing for some time. It is a complicated piece to design and implement properly that is why it is taking much longer than other components of the FFI.

[dcharkes]

dart:ffi now exposes finalizers in native code through dart_api.h, see details in the updated first post.

[leisim]

@dcharkes

Will it be possible to remove the finalizer from a finalizable pointer?

This would be needed if the memory the pointer points to becomes invalid for some reason before the pointer is GC'd. One example is a pointer to a raw Rust Box. When Rust takes back ownership, before the pointer is GC'd, Rust will take care of freeing the memory.

[dcharkes]

Yes, the current working design has that. (But the design has not been finalized yet.)

You can already do this with the finalizers in native right now, with Dart_DeleteWeakPersistentHandle_DL.

[darrinps]

Been over a quarter since the last update. Any news on this front? I believe Mongo, who acquired Realm, would be able to move forward with Realm support for Flutter which many of us want once this issue is resolved.

[drriguz]

What's the progress of that? I think this feature might be what I needed. https://github.com/simolus3/moor/issues/835#issuecomment-725489778

[mraleph]

@drriguz The status is given in the issue body itself. See the text after Update 2020-06-26.