Wasmer: Can't run Golang with large binary

[lucklove]

Describe the bug

wasmer hang when run large .wasm file compiled from Golang

echo "`wasmer -V` | `rustc -V` | `uname -m`"

wasmer 0.9.0 | | x86_64

Steps to reproduce

main.go:

package main

import (
        "fmt"
)

func main() {
        fmt.Println(len(xs))
}

var xs []int = []int{
        0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,
        0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,
        0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,
        // many lines copy and paste from above
}

With go/wasi implementation, compile with
GOOS=wasi GOARCH=wasm go build -o main.wasm

Run with wasmer run main.wasm

When the xs is small, it's ok, but it take longer time when xs grows. And when the main.wasm is large (abount 50MB), the wamser will hang

Expected behavior

Actual behavior

Additional context

[syrusakbary]

This means that the Clif backend is actually taking a lot of time compiling.

I think things should be faster if you run with the singlepass backend: wasmer run main.wasm --backend=singlepass. Let me know if that fixes your issue for now @lucklove

As a way to solve this generically, perhaps we can make singlepass the default backend when the provided binary file is larger than 10Mb... thoughts @bjfish @losfair @nlewycky ?

[lucklove]

@syrusakbary Thanks for help, with --backend=singlepass, it can run, but with another error:

$ wasmer run main.wasm --backend=singlepass
Error: Can't instantiate WASI module: LinkError([Generic { message: "data segment does not fit" }])

I don't know if it's related to --backend=singlepass or the Golang compiler

[syrusakbary]

Can you post here the wasm file? (you can compress it in a zip and send it in a comment). I'd like to debug a bit further

[lucklove]

main.wasm.zip
thanks for help

[Hywan]

Why are you closing the issue? The problem doesn't seem to be solved yet :-).

[lucklove]

@Hywan Oh, sorry, I thought nobody cares about it.
Maybe it's the same problem with https://github.com/pingcap/tidb/issues/13236

[syrusakbary]

@lucklove we debugged it further and it seem to be a bug on the go/wasi WebAssembly file.

The file generated is have an invalid data segment length. It should be an easy fix, but it should be done in the https://github.com/neelance/go/tree/wasi repo

Pinging @neelance, to see if he can help a bit on that!

[neelance]

I believe this issue is already fixed on Go master. I need to rebase my branch. Will do so later.

[lucklove]

I cherry-pick these commits and it works now:

142c002ee
1c50fcf85
9c384cc57
ecc7dd546
57662b157
30521d512
1c8e6077f
ecba83520
6dc740f09
d0f10a6e6
2686e7494
03bb3e9ad
e9c994954
ca70ada28
763d3ac75
e3c7ffcd9

[syrusakbary]

That's awesome @lucklove. We just tested tidb on wasmer locally and it works great.

It seems there is an error on WebAssembly.sh that we are trying to debug now, once it's solved... you will be able to run tidb on the online WebAssembly browser shell as well!

[syrusakbary]

Closing the issue as this appears to be an issue solved in the go/wasi repo.

We also have the PR #1004 that will use automatically the singlepass backend (when available) when the provided file is large (as both cranelift and llvm will take a lot of time compiling).

[neelance]

I've now rebased my fork for wasi at https://github.com/neelance/go/tree/wasi.