Sdk: Native extensions are not "portable" with kernel

[grouma]

Copying an internal issue:

I created a minimal external example (attached) based off this example: https://www.dartlang.org/articles/dart-vm/native-extensions

My main file is as follows:

import 'dart-ext:sample_extension';

int systemRand() native "SystemRand";
bool systemSrand(int seed) native "SystemSrand";

void main() {
  print(systemRand());
}

I compiled the C code with the following commands:
g++ -fPIC -I/usr/local/google/home/grouma/dart-sdk -DDART_SHARED_LIB -c sample_extension.cc
gcc -shared -Wl,-soname,libsample_extension.so -o libsample_extension.so sample_extension.o

Now if I run from source it executes as expected, printing a "random" int:
dart main.dart -> 1804289383

I can also run from a kernel snapshot.

First compile:
dart --snapshot=main.dart.dill main.dart

Then run:
dart main.dart.dill -> 1804289383

However, if I move the compiled program (along with corresponding .so file) to a subfolder 'foo', it will no longer run from kernel.

Move program:
mkdir foo
mv libsample_extension.so foo
mv main.dart foo
mv main.dart.dill foo

Run from source (success):
dart main.dart -> 1804289383

Run with kernel:
dart main.dart.dill -> libsample_extension.so: cannot open shared object file: No such file or directoryerror: library handler failed

I dumped the contents of the kernel snapshot:
dart pkg/vm/bin/dump_kernel.dart ~/Projects/kernel_test/foo/main.dart.dill output.txt

The contents show an absolute path to the main.dart file which could be problematic?

main = main::main;
@_in::ExternalName::•("dart-ext:sample_extension")
library from "file:///usr/local/google/home/grouma/Projects/kernel_test/main.dart" as main {
  @_in::ExternalName::•("SystemRand")
  external static method systemRand() → core::int;
  @_in::ExternalName::•("SystemSrand")
  external static method systemSrand(core::int seed) → core::bool;
  static method main() → void {
    core::print(main::systemRand());
  }
}
constants  {
}

Internally we compile our modular kernel files in temporary directories, not the directory in which the final program will run. If the VM is relying on this embedded path for some reason then it will not work.

sample.tar.gz

[grouma]

cc @kmillikin

Any update on this?

[davidmorgan]

Any update on this, please?

We don't need this _now_, but at some point we are likely to be blocked on it, and it will suddenly become urgent. (The first time we need to be able to quickly load dart code with native extensions).

[zoechi]

Will #34452 replace native extensions?

[davidmorgan]

7053 looks unrelated?

[zoechi]

Sorry - fixed.

[thosakwe]

This issue seems to now be unavoidable when trying to work with native extensions (I guess the Dart sources are compiled to kernel first).

I have an extension lib/src/libfoo.dylib, which is referenced as dart-ext:foo from lib/src/foo.dart.

I get this error every single time:

dlopen(libfoo.dylib, 1): image not founderror: library handler failed

When I move libfoo.dylib to the working directory, it begins to work.

This makes native extensions basically unusable, especially if distributed via Pub, etc., because the VM seems to expect the library to be present in the working directory.

[sjindel-google]

It's a little more complicated actually.

When the Dart sources are compiled to Kernel, the dart-ext:foo import is associated with an absolute path, which is resolved from the path of the library part containing the import at Kernel-compile time.

This is the first path the VM searches when loading the library at runtime. However, if it's not found there, the VM searches a few other paths, including the working directly and the directory containing the VM executable.

Finally, if the library is not found in any of those paths, the VM delegates to dlopen, which searches the system's library paths. On Linux, this includes LD_LIBRARY_PATH and LD_PRELOAD.

If the library cannot be found with any of those methods, the library resolution will fail. If Pub wants to allow distributing native extensions, it could create a directory with symlinks to all native libraries from all downloaded packages and append it to LD_LIBRARY_PATH (cc @jonasfj)

[thosakwe]

If Pub wants to allow distributing native extensions, it could create a directory with symlinks to all native libraries from all downloaded packages and append it to LD_LIBRARY_PATH (cc @jonasfj)

I'm more than willing to help with a PR on this, as it would be very useful. Otherwise, it seems like native extensions will be difficult, if not impossible, to distribute on Pub.

[sjindel-google]

I think this issue is obsolete with the FFI. With the FFI, the path to the library being loaded can be selected at runtime, rather than being fixed in an import statement.

Thus it can factor in the script's path, current platform, etc. I think it should be the package's responsibility to decide how the library is shipped and where it should be loaded from.

/cc @mkustermann Do you object to closing this?

[natebosch]

@sjindel-google - do we have FFI working internally? Do we have docs on using it?

We should try migrating at least one internal use case for native extensions to FFI and validate it works with AOT builds before closing this.