wasmer and dynamic set of host functions

[pepyakin]

I am playing with embedding wasmer as a backend into substrate. While doing this, I discovered that the high-level API assumes that the set of imports is known statically - for my use-case this is not the case. At least I haven't seen any APIs that would (easily) allow to do so. I wasn't able to find any examples that would show how to proper achieve that as well.

I thought I would be able to workaround that basically by getting inspired by this example and calling the same Rust functions.

However, regardless that I am essentially doing the same I am observing some weird behavior.

So the question is, do you have an advice on how to integrate wasmer with a dynamic set of host functions?

[losfair]

The high-level API that allows to statically specify imports is a declarative macro imports!: https://github.com/wasmerio/wasmer/blob/73370b9de9acf9c1c0f33b1fb29dba0488a05dff/lib/runtime-core/src/macros.rs#L78

which means that there isn't too much magic in it and it's possible to do the same thing outside runtime-core.

The code to dynamically specify imports would be like:

let mut import_object = ImportObject::new();

let mut ns = Namespace::new();
ns.insert("func1", func1_impl);
ns.insert("func2", func2_impl);

import_object.register("my_namespace", ns);

// now we can use import_object

[pepyakin]

Yep, I checked that in the very beginning of my journey. The part which is unclear is how to actually obtain func1_impl and func2_impl given that the signatures of those are not known upfront. The func! macro takes a function declaration as a parameter and I don't have that at hand at the runtime.

[losfair]

Okay so the problem is with dynamically-typed imports. Yeah the current way to do that in the master branch is with the various Trampoline* APIs, which are a bit hard to use. We are working on a new API for that: https://github.com/wasmerio/wasmer/pull/1217

The new API would be like:

let func = ErasedFunc::new_polymorphic(
    Arc::new(FuncSig::new(vec![Type::I32], vec![Type::I32])),
    |ctx, params| -> Vec<Value> {
        match params[0] {
            Value::I32(x) => vec![Value::I32(x + 1)],
            _ => unreachable!()
        }
    }
);

[pepyakin]

I think this would be exactly what I am looking for!

[syrusakbary]

We just merged #1217 into master.

The final API looks very similar to what @losfair commented:

let func = DynamicFunc::new(
    Arc::new(FuncSig::new(vec![Type::I32], vec![Type::I32])),
    |ctx, params| -> Vec<Value> {
        match params[0] {
            Value::I32(x) => vec![Value::I32(x + 1)],
            _ => unreachable!()
        }
    }
);

We will be releasing a new version of Wasmer soon. Let us know if this fully fits your needs @pepyakin :)

[pepyakin]

Hey! Found some time to test that and got the following questions:

1. I assumed that the way how the closure (or rather, the function, but about this later) can generate trap is by panicking. However, if I panic I get this message: Call error: \"unknown trap at 0x0 - segmentation violation\" which made me doubt: is this the correct way to generate traps from host functions?
2. DynamicFunc has a lifetime 'a. I wasn't able to figure out why since I cannot see how exactly it gets bounded and to what. The docs don't mention anything about this.

And the third, the most important part. I wasn't able to figure out how to actually implement what I aimed to do. As a recap: I have a dynamic set of host functions that are not known in advance (number of them, names, signatures or their bodies). Therefore, I need to somehow create these functions so each individual dynamic function would know where to pass control after it was called.

First thing I tried is to pass some sort of context into the closure. However, I learned the hard way ("DynamicFunc with captured environment is not yet supported") that the passed function is not actually a closure but rather a function since it cannot close over any variables.

Then, I guessed that maybe the Ctx is the right way to pass the data in the dynamic function. But reading the docs, specifically the data field, I couldn't figure out how it would work, I am not exactly sure what the following means

Alternatively, per-function data can be used if the function in the [ImportObject] is a closure. This cannot duplicate data though, so if data may be shared if the [ImportObject] is reused.

I read that as if DynamicFunc should pass this data for me, but it doesn't. The other option seems to be to use the instance wide context, but it won't work for me since there is no way to invoke a function.

The only way I can see this could possible work atm, is to statically instantiate a lot of F closures/function to some maximum number of host functions using type nums and then map each static instantiation to a dynamic index. But that is very gross and not sure if we want to pursue that.

Maybe I am missing something? If not, are there are any plans to lift the limitation wrt closures?

[losfair]

With the default configuration the runtime will enforce the closure passed to DynamicFunc not having any environment bindings. But this is a soft limit that we turned on by default for now, in case of possible future API changes. Enabling the dynamicfunc-fat-closures feature on the runtime-core package will lift the limitation.

[losfair]

The lifetime on DynamicFunc is for casting from a Func that carries a non-static lifetime. For DynamicFuncs that are directly created using ::new(), the lifetime is 'static.

Panic handling is indeed missing - will fix it.

[pepyakin]

Ah, thanks for pointing out! I actually was looking for something like this in the docs source but couldn't find anything but it seems it is in master already. I think that can help me move forward. No need for a release I am able to prototype with master for now.

[pepyakin]

Was this supposed to be solved actually?

FWIW, I am still seeing unknown trap at 0x1a - segmentation violation\" and sometimes (signal: 6, SIGABRT: process abort signal) with the latest master.

[losfair]

Can you provide an example to reproduce the error? Panics from a DynamicFunc should be properly handled in latest master.

[pepyakin]

Yep, so my code is basically this:

    DynamicFunc::new(
        Arc::new(FuncSig::new(params, returns)),
        move |ctx, args| -> Vec<wasmer_runtime_core::types::Value> {
            dbg!();
            vec![]
        },
    )
    .to_export()

The debugger shows me the following backtrace (stripped)

* thread #2, stop reason = signal SIGABRT
  * frame #0: 0x00007fff719ad7fa libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff71a6abc1 libsystem_pthread.dylib`pthread_kill + 432
    frame #2: 0x00007fff71934a1c libsystem_c.dylib`abort + 120
    frame #3: 0x0000000100d9b019 sc_executor-3996472ec945691b`std::sys::unix::abort_internal::h6a4dbde647e9c071 at mod.rs:155:4 [opt]
    frame #4: 0x0000000100d93e10 sc_executor-3996472ec945691b`std::sys_common::util::abort::hadf0ead4dbab210e at util.rs:20:8 [opt]
    frame #5: 0x0000000100d95263 sc_executor-3996472ec945691b`rust_panic at panicking.rs:523:4 [opt]
    frame #6: 0x0000000100d9513d sc_executor-3996472ec945691b`std::panicking::rust_panic_with_hook::h497844b5b7d1708e at panicking.rs:491:4 [opt]
    frame #7: 0x0000000100d94bd9 sc_executor-3996472ec945691b`rust_begin_unwind at panicking.rs:375:4 [opt]
    frame #8: 0x0000000100dc850c sc_executor-3996472ec945691b`core::panicking::panic_fmt::h2877d31f4cece04d at panicking.rs:84:13 [opt]
    frame #9: 0x0000000100dc8464 sc_executor-3996472ec945691b`core::panicking::panic::hb29adf552f3b34a3 at panicking.rs:51:4 [opt]
    frame #10: 0x0000000100dacf0c sc_executor-3996472ec945691b`alloc::raw_vec::capacity_overflow::h865fa9022ef4665d at raw_vec.rs:777:4 [opt]
    frame #11: 0x000000010066f102 sc_executor-3996472ec945691b`alloc::raw_vec::RawVec$LT$T$C$A$GT$::reserve::h4b656c4233cb2891(self=0x0000700006bd3760, used_capacity=0, needed_extra_capacity=8529655384782105703) at raw_vec.rs:521:37
    frame #12: 0x00000001006ed268 sc_executor-3996472ec945691b`alloc::vec::Vec$LT$T$GT$::reserve::hf748aea90cd5ad3e(self=0x0000700006bd3760, additional=8529655384782105703) at vec.rs:505:8
    frame #13: 0x00000001006f03f6 sc_executor-3996472ec945691b`_$LT$alloc..vec..Vec$LT$T$GT$$u20$as$u20$alloc..vec..SpecExtend$LT$T$C$I$GT$$GT$::spec_extend::h7aa5775c1efd2258(self=0x0000700006bd3760, iterator=Map<core::iter::adapters::Enumerate<core::slice::Iter<wasmer_runtime_core::types::Type>>, closure-0> @ 0x0000700006bd3778) at vec.rs:2037:12
    frame #14: 0x00000001006f1a5c sc_executor-3996472ec945691b`_$LT$alloc..vec..Vec$LT$T$GT$$u20$as$u20$alloc..vec..SpecExtend$LT$T$C$I$GT$$GT$::from_iter::h55d3b7fe2acb1e49(iterator=Map<core::iter::adapters::Enumerate<core::slice::Iter<wasmer_runtime_core::types::Type>>, closure-0> @ 0x0000700006bd37d0) at vec.rs:2024:8
    frame #15: 0x00000001006f415f sc_executor-3996472ec945691b`_$LT$alloc..vec..Vec$LT$T$GT$$u20$as$u20$core..iter..traits..collect..FromIterator$LT$T$GT$$GT$::from_iter::h7cf1e3003abd442a(iter=<unavailable>) at vec.rs:1911:8
    frame #16: 0x00000001006cf757 sc_executor-3996472ec945691b`core::iter::traits::iterator::Iterator::collect::h82287c87a63e2227(self=<unavailable>) at iterator.rs:1494:8
    frame #17: 0x000000010068a973 sc_executor-3996472ec945691b`wasmer_runtime_core::typed_func::DynamicFunc::new::do_enter_host_polymorphic::h338d8827907f24fb(ctx=0x0000000102d08040, args=0x0000700006bd3a50) at typed_func.rs:320:35
    frame #18: 0x000000010068af31 sc_executor-3996472ec945691b`wasmer_runtime_core::typed_func::DynamicFunc::new::enter_host_polymorphic_i::h08bdb318a5324182(ctx=0x0000000102d08040, args=0x0000700006bd3a50) at typed_func.rs:350:23
    frame #19: 0x000000011c000038
    frame #20: 0x0000000105f20186
    frame #21: 0x0000000105f2a237
    frame #22: 0x0000000105f296f1
    frame #23: 0x00000001027e35d5
    frame #24: 0x000000010026c8e7 sc_executor-3996472ec945691b`_$LT$wasmer_clif_backend..signal..Caller$u20$as$u20$wasmer_runtime_core..backend..RunnableModule$GT$::get_trampoline::invoke::_$u7b$$u7b$closure$u7d$$u7d$::h63e0da218c7022e5 at mod.rs:74:16
    frame #25: 0x000000010026b8fc sc_executor-3996472ec945691b`wasmer_clif_backend::signal::unix::call_protected::h0f409d7bfba0035a(handler_data=0x0000000102d019d0, f=closure-0 @ 0x0000700006bd41f0) at unix.rs:131:22
    frame #26: 0x000000010026c79a sc_executor-3996472ec945691b`_$LT$wasmer_clif_backend..signal..Caller$u20$as$u20$wasmer_runtime_core..backend..RunnableModule$GT$::get_trampoline::invoke::hcb5a0199f50a4df8(trampoline=(0x00000001027e35b0), ctx=0x0000000102d051e0, func=NonNull<wasmer_runtime_core::vm::Func> @ 0x0000700006bd41b0, args=0x0000700006bd4890, rets=0x0000700006bd4b80, error_out=0x0000700006bd42e8, invoke_env=Option<core::ptr::non_null::NonNull<core::ffi::c_void>> @ 0x0000700006bd4260) at mod.rs:72:22
    frame #27: 0x00000001006b5235 sc_executor-3996472ec945691b`wasmer_runtime_core::instance::call_func_with_index_inner::_$u7b$$u7b$closure$u7d$$u7d$::h3c77e1d597959adc(result_space=0x0000700006bd4b80) at instance.rs:678:22
    frame #28: 0x00000001006b46af sc_executor-3996472ec945691b`wasmer_runtime_core::instance::call_func_with_index_inner::h61137afa68dbea59(ctx_ptr=0x0000000102d051e0, func_ptr=NonNull<wasmer_runtime_core::vm::Func> @ 0x0000700006bd45a8, signature=0x0000000103002480, wasm=Wasm @ 0x0000700006bd4f48, args=(data_ptr = 0x0000700006bd5700, length = 2), rets=0x0000700006bd53c0) at instance.rs:726:12
    frame #29: 0x00000001006b3e34 sc_executor-3996472ec945691b`wasmer_runtime_core::instance::call_func_with_index::he501b1a460855857(info=0x0000000102d01bf8, runnable=&RunnableModule @ 0x0000700006bd4ea8, import_backing=0x0000000103802480, local_ctx=0x0000000102d051e0, func_index=(__0 = 84), args=(data_ptr = 0x0000700006bd5700, length = 2), rets=0x0000700006bd53c0) at instance.rs:613:4
    frame #30: 0x00000001006b389c sc_executor-3996472ec945691b`wasmer_runtime_core::instance::Instance::call::hf8d2b5de9a40dd03(self=0x0000700006bd6098, name=(data_ptr = "test_data_inHello worldall ok!assertion failed: `(left == right)`\n  left: ``,\n right: ``client/executor/src/integration_tests/mod.rsinputbaz", length = 12), params=(data_ptr = 0x0000700006bd5700, length = 2)) at instance.rs:363:8
    frame #31: 0x00000001001ddbff sc_executor-3996472ec945691b`sc_executor_wasmer::call_method::_$u7b$$u7b$closure$u7d$$u7d$::h9beca28fb4578d59 at lib.rs:104:8

which looks like that the segfault happens before my code is ran.

[pepyakin]

I think it is right time to mention that I am running macOS 10.15.3 and building in the debug mode.

[losfair]

The problem seems to be in to_export. The returned Export value should be bound to the lifetime of the DynamicFunc (or more accurately its inner field), but it is not.

[pepyakin]

To clarify will it be fixed upstream or you say that this would be the solution?

[losfair]

We would like to fix this ASAP since it's a soundness bug, but it might be a breaking change (requires change to the public Export enum) and we are having some discussions to ensure the new API is future-proof.

[HaronK]

Hi! I'm also trying to dynamically specify imports as it was suggested above but getting error that compiler cannot find Namespace structure.
I do following:

use wasmer_runtime::*;

and it sees pretty good ImportObject structure but not Namespace.
What I'm doing wrong?

[MarkMcCaskey]

@HaronK it looks like it's not exposed in the wasmer_runtime API, you can implement the LikeNamespace trait for your own type if you want more control over it.

It's also available from wasmer_runtime_core (https://docs.rs/wasmer-runtime-core/0.17.0/wasmer_runtime_core/import/struct.Namespace.html) but those APIs are more likely to break in the future and we now generally don't advise people to use it if they can avoid it.

[HaronK]

Ah! Thanks! I'm experimenting right now so I think it's ok to use wasmer_runtime_core.

[syrusakbary]

We are working on a refactor of Wasmer. Once it lands in master, it should make super trivial to work with dynamic functions.

[pepyakin]

Thanks for the heads up!

[syrusakbary]

We just landed the refactor in master a few weeks ago, and the new API makes it trivial to work with dynamic functions via Function::new and Function::new_with_env.

Let us know if you need any help using them @pepyakin

Closing the issue