Wasmer: Unrecoverable deserialization panics

[robinvanemden]

When stress testing a Wasmer based service that includes the following code:

                ...
        serializedModule, err := readStore(cfid)
        if err != nil {
            return err
        }
        module, err = wasm.DeserializeModule(serializedModule)
        if err != nil {
            return err
        }
        defer module.Close()
                ...

... we encounter the following two types of panic at irregular intervals (about once every 10.000 to 20.000 calls, on a heavily taxed server):

Deserialization panic 1

thread '<unnamed>' panicked at 'index 1979 out of range for slice of length 0', src/libcore/slice/mod.rs:2674:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
fatal runtime error: failed to initiate panic, error 5
SIGABRT: abort
PC=0x7f5c28582e97 m=21 sigcode=18446744073709551610

goroutine 3822 [runnable]:
github.com/wasmerio/go-ext-wasm/wasmer._Cfunc_wasmer_module_deserialize(0xc00010c090, 0x7f5bfc9f81a0, 0x1)
        _cgo_gotypes.go:946 +0x4d
github.com/wasmerio/go-ext-wasm/wasmer.cWasmerModuleDeserialize.func1(0xc00010c090, 0x7f5bfc9f81a0, 0xc00010c090)
        /home/robin/go/pkg/mod/github.com/wasmerio/[email protected]/wasmer/bridge.go:622 +0x60
github.com/wasmerio/go-ext-wasm/wasmer.cWasmerModuleDeserialize(0xc00010c090, 0x7f5bfc9f81a0, 0x78938)
        /home/robin/go/pkg/mod/github.com/wasmerio/[email protected]/wasmer/bridge.go:622 +0x35
github.com/wasmerio/go-ext-wasm/wasmer.DeserializeModule(0xc000958000, 0x78938, 0x7fe00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/robin/go/pkg/mod/github.com/wasmerio/[email protected]/wasmer/module.go:268 +0x152
github.com/scailable/sclbl-computenode/server.runWasm(0xc000532000, 0xfd, 0xc00001a9a4, 0x4, 0xc0002d8060, 0x24, 0xc000216000, 0x44, 0x0, 0x0, ...)
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/run_task.go:121 +0x155
github.com/scailable/sclbl-computenode/server.runTask(0xc000016440, 0x16, 0xc000216000, 0x44, 0xc000338088, 0x4, 0xc0003380b0, 0x8, 0xc000532000, 0xfd, ...)
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/run_task.go:39 +0xc6
github.com/scailable/sclbl-computenode/server.startTask(0xc000016440, 0x16, 0xc000216000, 0x44, 0xc000338088, 0x4, 0xc0003380b0, 0x8, 0xc000532000, 0xfd, ...)
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/handle_node.go:112 +0x60
created by github.com/scailable/sclbl-computenode/server.(*server).handleTask.func1
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/handle_node.go:96 +0x36e

Deserialization panic 2

thread '<unnamed>' panicked at 'Could not protect memory as its original protection: ProtectionFailed(140665029545984, 159744, "Cannot allocate memory")', src/libcore/result.rs:1188:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
fatal runtime error: failed to initiate panic, error 5
SIGABRT: abort
PC=0x7fef592d5e97 m=0 sigcode=18446744073709551610

goroutine 10141 [syscall]:
runtime.cgocall(0x6ca500, 0xc0004b1878, 0xc0004b1870)
        /usr/lib/go-1.13/src/runtime/cgocall.go:128 +0x5b fp=0xc0004b1848 sp=0xc0004b1810 pc=0x406eeb
github.com/wasmerio/go-ext-wasm/wasmer._Cfunc_wasmer_module_deserialize(0xc00016a018, 0x7f1c006bf720, 0x0)
        _cgo_gotypes.go:946 +0x4d fp=0xc0004b1878 sp=0xc0004b1848 pc=0x69c96d
github.com/wasmerio/go-ext-wasm/wasmer.cWasmerModuleDeserialize.func1(0xc00016a018, 0x7f1c006bf720, 0xc00016a018)
        /home/robin/go/pkg/mod/github.com/wasmerio/[email protected]/wasmer/bridge.go:622 +0x60 fp=0xc0004b18b0 sp=0xc0004b1878 pc=0x69fa00
github.com/wasmerio/go-ext-wasm/wasmer.cWasmerModuleDeserialize(0xc00016a018, 0x7f1c006bf720, 0x78938)
        /home/robin/go/pkg/mod/github.com/wasmerio/[email protected]/wasmer/bridge.go:622 +0x35 fp=0xc0004b18d8 sp=0xc0004b18b0 pc=0x69d805
github.com/wasmerio/go-ext-wasm/wasmer.DeserializeModule(0xc000a90000, 0x78938, 0x7fe00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/robin/go/pkg/mod/github.com/wasmerio/[email protected]/wasmer/module.go:268 +0x152 fp=0xc0004b19e0 sp=0xc0004b18d8 pc=0x69a972
github.com/scailable/sclbl-computenode/server.runWasm(0xc00027c200, 0xfd, 0xc00001a9b0, 0x4, 0xc000160090, 0x24, 0xc0000dc0f0, 0x44, 0x0, 0x0, ...)
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/run_task.go:121 +0x155 fp=0xc0004b1d18 sp=0xc0004b19e0 pc=0x6a58e5
github.com/scailable/sclbl-computenode/server.runTask(0xc0000165e0, 0x16, 0xc0000dc0f0, 0x44, 0xc0001541b0, 0x4, 0xc0001541b8, 0x8, 0xc00027c200, 0xfd, ...)
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/run_task.go:39 +0xc6 fp=0xc0004b1e40 sp=0xc0004b1d18 pc=0x6a54d6
github.com/scailable/sclbl-computenode/server.startTask(0xc0000165e0, 0x16, 0xc0000dc0f0, 0x44, 0xc0001541b0, 0x4, 0xc0001541b8, 0x8, 0xc00027c200, 0xfd, ...)
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/handle_node.go:112 +0x60 fp=0xc0004b1f28 sp=0xc0004b1e40 pc=0x6a5030
runtime.goexit()
        /usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004b1f30 sp=0xc0004b1f28 pc=0x45dc71
created by github.com/scailable/sclbl-computenode/server.(*server).handleTask.func1
        /mnt/c/Users/robin/Documents/GitHub/sclbl-computenode/server/handle_node.go:96 +0x36e

Though the issues are hard to replicate, maybe the above stack traces can help pinpoint the root cause?

[Hywan]

I've transfered the issue from https://github.com/wasmerio/go-ext-wasm. I think it belongs to the runtime itself.

[robinvanemden]

@Hywan thanks!

[MarkMcCaskey]

Hmm this also raises the issue that we should be catching panics at the FFI boundary and passing an error value back... that seems like it could be a lot of work but also it seems like something a proc macro could do for us

[robinvanemden]

@MarkMcCaskey catching panics would certainly be a nice to have for us. Currently, we are running several WASM services. When one of these services crashes with an unrecoverable panic, we now just restart that particular service through systemd. Still, would be nice to resolve the panics in a more elegant way :)

[syrusakbary]

We greatly improved serialization/deserialization in the wasmer refactor.
Closing the issue