Wasmtime: wasm64 support

[joshtriplett]

We should consider supporting wasm64 modules, not just wasm32; people will want to run with large linear address spaces, both to process large amounts of data and to provide address space for shared mappings or file mappings.

Opening this issue to start discussing what that support should look like, and how we can do that with minimal complexity or duplication.

[sunfishcode]

The first step is to propose the idea for stage 0 of the CG process. Once that's accepted, we can then create a repo in the WebAssembly organization where we can coordinate and collect documentation, and avoid duplication.

Next is to design a binary encoding. In theory, all we need is a way to label a linear-memory as 64-bit, because all opcodes that operate on linear memories take an index which specifies which memory they talk about, and that can then determine the types of their operands.

(Especially with multiple memories on the horizon, I think we can say that "wasm64" shouldn't be a new language or mode. Instead, we want individual linear memories to be marked as 64-bit, so that a program could in theory have both 32-bit and 64-bit memories. Some tools, like LLVM, may continue to think of "wasm64" as a separate architecture from "wasm32", however that's just a convention.)

Then, prototyping can start, both on the producer side and consumer side. I expect we can do this prototyping upstream, rather than in separate branches, because wasm64 is something that many people want, and most of the code should be straightforward. We just need to be careful to communicate that the binary format won't be stable until it progresses further through the CG process.

On the producer side, the one tool I know of with a start on wasm64 is LLVM, though it's not complete yet, and it needs to be taught about the binary encoding.

On the consumer side, besides just teaching various components how to recognize the flag and allocate memory for it and generate code for it, there's also a question of sandboxing. To start with, we can use bounds checking, though also see here for an interesting possible optimization.

[joshtriplett]

@sunfishcode Ah, for some reason I thought there was already a preliminary specification for the binary format. https://webassembly.org/docs/future-features/#linear-memory-bigger-than-4-gib somewhat implies that ("wasm32 and wasm64 are both just modes of WebAssembly, to be selected by a flag in a module header").

Sorry for the confusion on my part.

I do think it makes sense to support the concept of both 32-bit and 64-bit memories, and for that matter, future linking models could theoretically communicate between and translate between modules written for 32-bit and for 64-bit.

(Though, even if the underlying model supports multiple memories, I wouldn't find it surprising if code targeting wasm64 uses flat 64-bit pointers and runtimes have to encode any concept of multiple address spaces into the pointer, rather than making pointers larger than 64 bits.)

[sunfishcode]

It appears I wrote that sentence back in 2015; things have evolved somewhat since then :-}. I've now submitted https://github.com/WebAssembly/design/pull/1311 to update that.

[sunfishcode]

Some discussion on wasm64 here.

[binji]

Yeah, I'd like to push this forward sooner rather than later. My biggest concerns at this point are in the consumer -- making sure that we don't have to fall back to bounds checks. If anyone has spare cycles, it would be great to see some experiments with alternate trap-handlers (including effective address calculation).

[joshtriplett]

Would architecture-specific optimizations to bounds checking be welcome? Obviously we need a correct implementation on all platforms. But if a platform can substantially speed up bounds checking and avoid per-access checks, would that be welcome?

Specifically, I would suggest memory protection keys (PK). We could allocate the memory for a given sandbox using a protection key, and enable only that protection key before jumping into the JITted code. (This would have limitations, notably that a process has a limited number of protection keys, but I think it would work well in many common cases.)

[binji]

Yeah, I think that's a great solution for some platforms. But as with SIMD, I think we'll need to make sure that we're OK with the performance on all platforms. I think we all expect to see a performance regression, but the question is how much.

[joshtriplett]

@binji Of course. Performance would need to be acceptable everywhere, but from what I understand, even with bounds checking, performance is acceptable.

[binji]

True, @aardappel has made a similar argument. We know we'll need 64-bit memories, even if they end up being slower at first. I'm mostly concerned w/ how we spec it so we can support as many optimizations as possible (including PK).

[sunfishcode]

There is now an official spec proposal repo, memory64.

[lygstate]

When this going to happen?

[aardappel]

I've implemented wasm64 support in LLVM, LLD, WABT, now working on Binaryen..

[lygstate]

Wonderfull work, support in wasmtime would be relative harder?

[tschneidereit]

Indeed, that's excellent progress, @aardappel!

Wonderfull work, support in wasmtime would be relative harder?

My understanding — which might be wrong — is that it's probably not a huge amount of work, but also non-trivial. @alexcrichton and @sunfishcode, ISTM we talked about this a while ago, do you happen to have an idea of the work involved to make this happen?

[lygstate]

Indeed, that's excellent progress, @aardappel!

Wonderfull work, support in wasmtime would be relative harder?

My understanding — which might be wrong — is that it's probably not a huge amount of work, but also non-trivial. @alexcrichton and @sunfishcode, ISTM we talked about this a while ago, do you happen to have an idea of the work involved to make this happen?

does the wasm64 and wasm32 have different abi？

[bjorn3]

They would need to have different abi's. Pointer size is part of the abi and they use different pointer sizes.

[alexcrichton]

I suspect this would be relatively simple to implement nowadays. The memory64 proposal is quite small, basically just changing memory-operating instructions to work with either 32 or 64-bit indices. The work in wasmparser is already done to implement the memory64 proposal, so all that's needed to be done is to thread it all through cranelift.

At this time this probably won't be as well optimized as memory32 since we can't naively do the exact same guard page trick we do there (reserving a 32-bit region of the address space). That being said cranelift has all the internal machinery to insert manual checks on each load/store, so we'd just need to hook that up. Overall this is likely a simple-ish refactoring of the code translator to conditionally use 64 or 32-bit indices everywhere, depending on what type each memory has.

The wasmtime API itself may not even have to change at all for something like this. We might add a flag to Memory as to whether it's 64-bit or not, but otherwise everything is pretty much the same. Although ABIs are different that's only really an artifact of compilation toolchains, once you get to the runtime it's all basically just a wasm blob.