Wasmtime: Rust doesn't work by default on Windows

[alexcrichton]

I ran across this uptream in cargo-wasi, but this fails to work in this repository, but only on Windows:

$ echo "#[test] fn foo() {}" >> foo.rs
$ rustc foo.rs --target wasm32-wasi --test
$ cargo run foo.wasm
    Finished dev [unoptimized + debuginfo] target(s) in 0.28s
     Running `target\debug\wasmtime.exe foo.wasm`
Error: failed to run main module `foo.wasm`

Caused by:
    0: WebAssembly failed to compile
    1: Compilation error: Implementation limit exceeded

The underlying reason can be found with RUST_LOG:

$ RUST_LOG=warn cargo run foo.wasm
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
     Running `target\debug\wasmtime.exe foo.wasm`
 WARN  cranelift_codegen::isa::x86::unwind::winx64 > stack frame is too large (288 bytes) to use with Windows x64 SEH when preserving FPRs. This is a Cranelift implementation lim
it, see https://github.com/bytecodealliance/wasmtime/issues/1475
 WARN  cranelift_codegen::isa::x86::unwind::winx64 > stack frame is too large (416 bytes) to use with Windows x64 SEH when preserving FPRs. This is a Cranelift implementation lim
it, see https://github.com/bytecodealliance/wasmtime/issues/1475
Error: failed to run main module `foo.wasm`

Caused by:
    0: WebAssembly failed to compile
    1: Compilation error: Implementation limit exceeded
error: process didn't exit successfully: `target\debug\wasmtime.exe foo.wasm` (exit code: 1)

It looks like it's due to these lines introduced in https://github.com/bytecodealliance/wasmtime/pull/1216. cc @iximeow do you know how hard it'd be to fix that warning?

[alexcrichton]

also cc @peterhuene, you might be interested in this

[iximeow]

It shouldn't be terribly hard to fix, but I haven't had the time to focus on this for a bit. #1475 is indeed the issue tracking this Cranelift limitation. I'm not certain on how to change layout_stack to fix it, so it might actually be trickier.

Windows seems to prefer stack layouts with callee-save values closer to SP rather than the base of the call frame. Adjusting layout_stack so that the FPR region is positioned correctly should be all that's necessary.

I'd also assumed the original ABI incompatibility would be a simple 30 minute fix, but took significantly longer to get straight, hence my hesitance to pick it up.

[peterhuene]

When I get a spare second, I'll investigate fixing this too.

For Windows x64, we should be laying out the stack like this:

[return address]
[saved RBP] <--- frame pointer target
[saved integer registers]
[saved floating point registers] <-- address of last saved FPR = "unwind base address"
[locals / spills]
[callee argument space]
[32 bytes of shadow store space]

The problem now is that it is laid out like this:

[return address]
[saved RBP] <--- frame pointer target
[saved integer registers]
[locals / spills]
[saved floating point registers] <--- "unwind base address" might be too far from FP
[callee argument space]
[32 bytes of shadow store space]

Windows unwind information expects a small, fixed-size area for register saves relative to the frame pointer, thus it only encodes a maximum offset of 240 to reach the "unwind base address". By sticking the variable-sized stack allocation in-between where FRPs are saved, we can't always represent these frames for unwind.

[github-actions[bot]]

Subscribe to Label Action

cc @bnjbvr

This issue or pull request has been labeled: "cranelift"

Thus the following users have been cc'd because of the following labels:

• bnjbvr: cranelift

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

[peterhuene]

I nearly have a fix completed.

As I was investigating this issue I discovered #1728. As a result of using an explicit slot, there's the additional problem of accidentally saving the FPRs in the reserved area for callee shadow space, which allows for the possibility of trashing the saved values when calling into non-cranelift code (i.e. into a function that actually uses the shadow space). The problem affects any explicit slot, but will always affect saved FPRs since it's the last explicit slot laid out currently (i.e. saved FPRs get the lowest address in the frame) for Windows x64.

Instead of saving the FPRs as an explicit slot, I am treating them more like the GPRs: the total CSR space is calculated as the sum of the GPR and FPR space. Like the GPRs, an explicit stack slot isn't used. Instead, the prologue uses a relative offset from a post-adjusted SP for the saves.

Here's a function that demonstrates both the limitation in this issue (notably that spills/explicits are at a higher address the saved FPRs) as well as the shadow space problem:

set opt_level=speed_and_size
set is_pic
target x86_64 haswell

function %foo(f64, f64, f64, f64) windows_fastcall {
    fn0 = %bar(i64) windows_fastcall
    ss0 = explicit_slot 8
block0(v0: f64, v1: f64, v2: f64, v3: f64):
; explicitly use a callee-save register
[-, %xmm6]  v4 = fadd v0, v1
[-, %xmm7]  v5 = fadd v0, v1
    v6 = stack_addr.i64 ss0
    call fn0(v6)
    return
}

Old code generation:

   0:   55                      push    rbp
   1:   48 89 e5                mov     rbp, rsp
   4:   48 83 ec 50             sub     rsp, 0x50
   8:   4c 8d 9c 24 00 00 00 00 lea     r11, [rsp]
  10:   41 0f 11 33             movups  xmmword ptr [r11], xmm6
  14:   41 0f 11 7b 10          movups  xmmword ptr [r11 + 0x10], xmm7
  19:   48 8d 84 24 28 00 00 00 lea     rax, [rsp + 0x28]
  21:   48 89 c1                mov     rcx, rax
  24:   e8 00 00 00 00          call    0x29
  29:   4c 8d 9c 24 00 00 00 00 lea     r11, [rsp]
  31:   41 0f 10 33             movups  xmm6, xmmword ptr [r11]
  35:   41 0f 10 7b 10          movups  xmm7, xmmword ptr [r11 + 0x10]
  3a:   48 83 c4 50             add     rsp, 0x50
  3e:   5d                      pop     rbp
  3f:   c3                      ret

Note that [rsp] is storing a saved FPR, which is actually the first 16 bytes of the callee's shadow space :scream:.

New code generation (also with the fix to account for shadow space properly):

   0:   55                      push    rbp
   1:   48 89 e5                mov     rbp, rsp
   4:   48 83 ec 50             sub     rsp, 0x50
   8:   0f 11 74 24 40          movups  xmmword ptr [rsp + 0x40], xmm6
   d:   0f 11 7c 24 30          movups  xmmword ptr [rsp + 0x30], xmm7
  12:   48 8d 84 24 28 00 00 00 lea     rax, [rsp + 0x28]
  1a:   48 89 c1                mov     rcx, rax
  1d:   e8 00 00 00 00          call    0x22
  22:   0f 10 74 24 40          movups  xmm6, xmmword ptr [rsp + 0x40]
  27:   0f 10 7c 24 30          movups  xmm7, xmmword ptr [rsp + 0x30]
  2c:   48 83 c4 50             add     rsp, 0x50
  30:   5d                      pop     rbp
  31:   c3                      ret

With these changes, the unwind information should now always be able to describe where the saved FPRs are.