Wasmer: failed to find instruction for 7fd8fbada259

[webmaster128]

Describe the bug

I'm upgrading to Wasmer at 6041c031b9 (almost latest master). I have everything working fine locally and all tests passing.

Now there is a strange error that only happens in the CI:

thread 'handle_panic' panicked at 'failed to find instruction for 7fd8fbada259', /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/6041c03/lib/engine/src/trap/frame_info.rs:151:9

which refers to this line. The name "handle_panic" is my test name, which executes Wasm that contains a Rust panic.

Steps to reproduce

Unfortunately this only happens in the CI, not on my local mac machine and not on my local Linux. All run Rust in debug mode.

Expected behavior

Tests passt

Actual behavior

Tests error with the message above

Additional context

See https://github.com/CosmWasm/cosmwasm/pull/504

[MarkMcCaskey]

Now there is a strange error that only happens in the CI

It could be possibly be related to caching; does your CI preserve wasmer's cache data?

[syrusakbary]

It could be possibly be related to caching; does your CI preserve wasmer's cache data?

Doesn't seem so:

https://github.com/CosmWasm/cosmwasm/blob/upgrade-to-wasmer-reborn/.circleci/config.yml

[webmaster128]

Since we use Wasmer as a library, I am not aware of any caching that could happen. The only thing that is cached is cargo stuff.

[syrusakbary]

@webmaster128 I've investigated a bit the issue. Is it possible that you are testing in release mode in local (mac/linux) but in debug mode in CircleCI?

[webmaster128]

@syrusakbary thank you for looking into this. It debug in all cases: cd contracts/hackatom && cargo wasm && cargo integration-test with .cargo/config:

[alias]
wasm = "build --release --target wasm32-unknown-unknown"
wasm-debug = "build --target wasm32-unknown-unknown"
unit-test = "test --lib"
integration-test = "test --test integration"
schema = "run --example schema"

i.e. the Wasm is build in release and the tests are executed in debug.

---

But I found something different on my local machine. With a local copy of Wasmer and some debugging statements like this

diff --git a/lib/engine/src/trap/frame_info.rs b/lib/engine/src/trap/frame_info.rs
index 9d5585a03..5574c5ad4 100644
--- a/lib/engine/src/trap/frame_info.rs
+++ b/lib/engine/src/trap/frame_info.rs
@@ -108,8 +108,11 @@ impl GlobalFrameInfo {
     /// Returns an object if this `pc` is known to some previously registered
     /// module, or returns `None` if no information can be found.
     pub fn lookup_frame_info(&self, pc: usize) -> Option<FrameInfo> {
+        println!("In lookup_frame_info");
         let module = self.module_info(pc)?;
+        println!("Got module");
         let func = module.function_info(pc)?;
+        println!("Got func");

         // Use our relative position from the start of the function to find the
         // machine instruction that corresponds to `pc`, which then allows us to
@@ -148,6 +151,7 @@ impl GlobalFrameInfo {
         // the function, because otherwise something is buggy along the way and
         // not accounting for all the instructions. This isn't super critical
         // though so we can omit this check in release mode.
+        println!("Pos: {:?}", pos);
         debug_assert!(pos.is_some(), "failed to find instruction for {:x}", pc);

         let instr = match pos {

I only get the first log statement locally. "Got module" is never reached.

[webmaster128]

The behaviour is the same on the 1.0.0-beta1 tag. Here is a stacktrace to it:

thread 'handle_panic' panicked at 'failed to find instruction for 7f0e89427259', /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/engine/src/trap/frame_info.rs:151:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:483
   1: std::panicking::begin_panic_fmt
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:437
   2: wasmer_engine::trap::frame_info::GlobalFrameInfo::lookup_frame_info
             at /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/engine/src/trap/frame_info.rs:151
   3: wasmer_engine::trap::error::RuntimeError::new_with_trace::{{closure}}
             at /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/engine/src/trap/error.rs:172
   4: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:269
   5: core::iter::traits::iterator::Iterator::find_map::check::{{closure}}
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/iter/traits/iterator.rs:2257
   6: core::iter::traits::iterator::Iterator::try_fold
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/iter/traits/iterator.rs:1888
   7: core::iter::traits::iterator::Iterator::find_map
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/iter/traits/iterator.rs:2263
   8: <core::iter::adapters::FilterMap<I,F> as core::iter::traits::iterator::Iterator>::next
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/iter/adapters/mod.rs:1245
   9: <alloc::vec::Vec<T> as alloc::vec::SpecFromIterNested<T,I>>::from_iter
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/vec.rs:2125
  10: <alloc::vec::Vec<T> as alloc::vec::SpecFromIter<T,I>>::from_iter
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/vec.rs:2267
  11: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/vec.rs:2002
  12: core::iter::traits::iterator::Iterator::collect
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/iter/traits/iterator.rs:1670
  13: wasmer_engine::trap::error::RuntimeError::new_with_trace
             at /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/engine/src/trap/error.rs:170
  14: wasmer_engine::trap::error::RuntimeError::from_trap
             at /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/engine/src/trap/error.rs:102
  15: wasmer::externals::function::Function::call_wasm
             at /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/api/src/externals/function.rs:440
  16: wasmer::externals::function::Function::call
             at /usr/local/cargo/git/checkouts/wasmer-f11f30e62739aa29/f0a2860/lib/api/src/externals/function.rs:528
  17: cosmwasm_vm::instance::Instance<S,A,Q>::call_function
             at /root/cosmwasm/packages/vm/src/instance.rs:306
  18: cosmwasm_vm::calls::call_raw
             at /root/cosmwasm/packages/vm/src/calls.rs:155
  19: cosmwasm_vm::calls::call_handle_raw
             at /root/cosmwasm/packages/vm/src/calls.rs:117
  20: cosmwasm_vm::calls::call_handle
             at /root/cosmwasm/packages/vm/src/calls.rs:53
  21: integration::handle_panic
             at ./tests/integration.rs:293
  22: integration::handle_panic::{{closure}}
             at ./tests/integration.rs:282
  23: core::ops::function::FnOnce::call_once
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227
  24: core::ops::function::FnOnce::call_once
             at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227

[syrusakbary]

Great, thanks for the report. We'll take a closer look soon

[syrusakbary]

@nlewycky brought to our attention that this bug is probably related to:

https://github.com/wasmerio/wasmer/issues/1523

[syrusakbary]

@webmaster128 after some research we found the issue:

1. macOS doesn't map properly the errors for the module (so it exits early and the issue never appears)
2. singlepass in Linux was not emitting the expected addressmap for the traps. This is not strictly required, but it's a bit nicer than removing the debug assert.

The solution for 2. is available in the PR #1870. If you could test it and output results would be great!

[webmaster128]

Great, thank you. At 6eb897d (from #1870)

• Everything still passes on the Mac because of 1.
• The error in the CI we experienced before disappears (https://github.com/CosmWasm/cosmwasm/pull/646) 🥳

The local Linux machine I was referring to was on the wrong branch 🤦‍♂️. I can now reproduce the CI's behaviour there. So only two cases left: Mac+singlepass and Linux+singlepass.