Go: all: WebAssembly ("wasm") support

[bradfitz]

WebAssembly ("wasm") is similar to Native Client, but different notably in that other browsers plan to implement it.

http://webassembly.org/

This has been asked about a few times, so this is a tracking bug for it.

Whether we get it via cmd/compile, gccgo, or llvm-go, we can post updates here.

[minux]

@cherrymui and I discussed the possibility of a wasm GC port at length in
the past few days:

Our current conclusion is that there is no efficient way to implement
setjmp/longjmp functionality currently, therefore it's not a viable target
of a gc port. We need to wait until it gets real stack unwinding and
exception handling support.

All the other aspects looked fine, and we even designed a Go ABI under wasm
(pass g and SP on wasm stack and everything else on emulated Go stack) and
a way to modify the current MIPS backend to support wasm by emulating MIPS
registers as wasm local variables.

Perhaps GopherJS can support wasm easier.

[ernado]

From gopherjs/gopherjs#432:

However, it is not a technology that GopherJS can use. GopherJS compiles on the AST level, not on the machine code level.

WebAssembly could be used as a backend for the normal Go compiler

[4ad]

@minux The lack of threads or async i/o seems to be a bigger problem than the ABI workarounds we'd have to do because of wasm quirks.

[sbinet]

@minux, @cherrymui
could you post somewhere in more details what you did and achieved?
as you may know the go-interpreter community is considering writing an interpreter based on either LLVM or wasm bytecode.
(we may even have a GSoC student working on the wasm bytecode interpreter [1], so, just the "bytecode consumption" part, not the "bytecode production via some Go-based toolchain" one)

minimizing the amount of duplicated efforts and impedance mismatch between various components would be great.

[RaananHadar]

I highly recommend this as a strategic feature. Consider that in just a few years web programmers will flock in hordes to pick their language of choice to be compiled into web assembly. The sooner we join the race the better. Mozilla is pushing Rust hard as the web assembly language of choice...

[tiborvass]

I second what @RaananHadar said. It would be a pity if the spec didn't suit Go's needs, especially given how Go's runtime can be unique.

EDIT: Apologies for the MeToo nature of this comment :) Thanks for the pointer Brad.

[bradfitz]

Reminder: https://golang.org/wiki/NoMeToo

[vibhavp]

The WebAssembly stack machine context consists of linear memory and an operation stack. IIUC, saving the VM context can possibly be done by saving the:

1. Current PC, bytecode string
2. Linear memory, and the
3. Operation Stack

To a global list of saved contexts in the VM context struct, if any.

Thus, a setjump should simply restore these values, and continue with the interpreter loop as usual (This is similar to how the emacs bytecode VM implements signals: https://github.com/emacs-mirror/emacs/blob/master/src/bytecode.c#L785). An exception system can be implemented on the top of this, though I'm not sure how performant would this be.

Further, the webassembly spec mentions that the stack height at any moment in the bytecode is statically known, making all stack operations equivalent to operations on registers unique to each position on the stack. This paper describes a way to achieve such a stack -> register mapping for converting stack machine code to native code, allowing us to use setjmp/longjmp as usual.

Edit: What about using panic() and recover() with special WasmException{} values for signalling an exception? recover does the tough job of stack unwinding already, it shouldn't be hard to restore the interpreter loop after recovering from a caught exception.

[minux]

Yes, basically the discussion with @cherrymui resulted in similar findings.

The initial design with @cherrymui is this:
(this stage focus on getting $GOROOT/test/sieve.go to work, no async IO is
considered.)
we reuse the MIPS port, mapping the 31 MIPS registers as WASM local
variables and only use WASM stack
for intermediate results. We can't use WASM's function call mechanism for
most Go functions because
the lack of stack unwinding support.
change the mips objlink (cmd/internal/obj) to emulate each MIPS instruction
with respective WASM instructions.
This makes setjmp/longjmp (aka runtime.gogo) implementation trivial and
reuses a lot of existing effort.

Go already needs to implement its own stack, separate of the WASM stack,
because Go needs copyable
stack, stack pointer maps and stack unwinding support, which are not
available from WASM.
(This strategy also aligns with the LLVM port, we can't use LLVM stack for
almost the same reasons
(unless we add a custom backend pass to LLVM proper to encode stack pointer
maps).)

We will probably need to use a big switch to thread indirect jumps? This is
similar to NestedVM's approach.
http://nestedvm.ibex.org/

If we decide to implement a native WASM backend, then we can do this:
the Go ABI is:
all parameters (including results) on (emulated) Go stack,
at function entry, the following are on the WASM stack:
emulated SP, g.

The discussion happened more than one month ago, and I probably forgot some
details at this point.

[vibhavp]

IIUC, you can completely discard the WASM stack (according to the spec). An extendable and copyable go stack is still something that needs to be investigated.

[minux]

Yes, the initial design I mentioned only uses the WASM stack for
intermediates used in computation. All data is either on Go heap (linear
memory) or pseudo registers in WASM local variables.

I don't expect WASM spec to incorporate moving/segmented and copying stack.
It's rarely used by mainstream languages targeted by WASM.

[SerkanSipahi]

@bradfitz @minux @vibhavp What is the current state of wasm in golang? Nothing has happened since March 13th ! will it be possible transform golang to wasm in the future? is there maybe any route-map of that?

[kanocz]

if somebody will decide to implement wasm support I can offer my time for non-global tasks

[bradfitz]

@SerkanSipahi, nobody is working on this. This bug is marked LongTerm. If something starts happening, you'll see updates here.

[joeblew99]

A good roadmap ?

We have DELVE and GDLV for debugging golang with a GUI. It works well too on all Desktops and Perf is excellent.
https://github.com/derekparker/delve
https://github.com/aarzilli/gdlv

Now, GDLV is based on NUCULAR which is shiny with some nice abstractions.
https://github.com/aarzilli/nucular

SO i was thinking this would be a good basis to do what has already been done here:

• Demo: http://shi-yan.github.io/AssortedWidgets/
• Code: https://github.com/shi-yan/AssortedWidgets

There are 5 WASM go libaries now too:
https://github.com/search?l=Go&q=webassembly&ref=simplesearch&type=Repositories&utf8=%E2%9C%93

[techtonik]

NaCl is shut down - https://blog.chromium.org/2017/05/goodbye-pnacl-hello-webassembly.html
So this should probably be cleaned up - https://github.com/golang/go/tree/master/misc/nacl

[ianlancetaylor]

I believe that NaCl is used by various parts of the Go ecosystem. That is, it's not used only by Chrome. And in any case that blog announcement is about PNaCl, which, although it provides functionality similar to NaCl, is actually a completely different product based on a different technology. So removing NaCl support from Go is premature at present.

Any in any case whether or not we remove NaCl from Go has nothing to do with whether we add support for Web Assembly.

[techtonik]

I would say that PNaCl is just an extension over NaCl to provide platform independent code https://developer.chrome.com/native-client/nacl-and-pnacl I asked about NaCl deprecation here - https://groups.google.com/d/topic/native-client-discuss/wgN2ketXybQ/discussion

If is possible to get list of those various parts of the Go ecosystem where NaCl is still used?
Going through it one by one will make transitioning to WebAssembly more interesting.

[4ad]

I would say that PNaCl is just an extension over NaCl

You can say whatever you want, but it doesn't mean you are right. They are completely different ISAs, NaCL is just the target ISA (x86/arm) with some restrictions, while PNaCl is a completely different ISA for an abstract machine.

Go has never supported PNaCL, and the Go NaCL port was never suitable for Chrome anyway. It had nothing to do with Chrome, it was not usable in the browser. Whether Chrome abandons NaCL or PNaCL has no relevance to Go whatsoever. Nada, nil, no relevance. How many more times this must be said? Of course, if NaCL is abandoned completely then Go will be forced at some point to abandon it too.

WebAssembly support in Go has absolutely nothing to do with NaCL. Go might or might not get WebAssembly support. If, or when it might get such support has no relevance to the state of the NaCL port.

[bradfitz]

Here is a very important part of the ecosystem where we use nacl: https://play.golang.org/p/MfJIq8wb5-

(that runs server-side, not nacl-in-a-browser)

[techtonik]

Whether Chrome abandons NaCL or PNaCL has no relevance to Go whatsoever. Nada, nil, no relevance.

And who will support sandbox loader and toolchain then? It is imported from Chrome project, which switched efforts to WebAssembly. Wouldn't it be wise to follow and see if Go playground can already be ported from NaCl sandbox to WebAssembly sandbox? It can also be run server-side.

[ianlancetaylor]

I think everybody is in favor of supporting WebAssembly.

The time to discuss moving away from NaCl is after WebAssembly is fully working. There's no point to discussing it before then.

[vine77]

@ianlancetaylor, can you clarify "after WebAssembly is fully working"?

At a glance, it appears webassembly.org says "the initial version of WebAssembly has reached cross-browser consensus" and wasm is available in current or future browser versions from all browser vendors.

[ianlancetaylor]

@vine77 I mean after WebAssembly is fully working for Go programs, at least as well as NaCl works today.

[techtonik]

So what is the next test that should pass for WebAssembly to work for Go programs?

[ianlancetaylor]

@techtonik I feel like there is some confusion here. When we say that WebAssembly should work for Go programs, what we mean is that the Go compiler should generate WebAssembly code that can run in the browser. We mean that you should be able to write something like GOOS=wasm go build hello.go and get a program that can run inside a browser. We aren't even remotely close to that. We haven't even started. So there is no "next test." There is a lot of work to be done. And, as far as I know, nobody is actively working on it.

[joeblew99]

@ ianlancetaylor
There are 4 implementations out there for golang. One seriously impressive. See my previous comment for the links.

Boy this thread is like Chevy Chase car seen with the kids in the back seat screaming " are we there yet " :)

[randall77]

@joeblew99 : none of those links are things that can compile Go to WebAssembly.
They are things like a WebAssembly -> amd64 compiler written in Go.

[joeblew99]

I was afraid you would say that. Fair enough.
So what IS needed for the " bring up" ? So everyone knows...
Gopherjs kind of made the same mistake of not being at the right level in the compiler tool chain architecture / code generation ?

[randall77]

There's a long list. Most have to do with garbage collection, one way or another.

• Generate WebAssembly code in the SSA backend. The backend is used to register machines, adapting to WebAssembly's stack machine will take some work.
• What's a pointer? WebAssembly doesn't have a pointer type.
• How do we introspect the stack? We'd need this for garbage collection, certainly, but also panic/recover, traceback printing, etc.
• How do we suspend/resume a goroutine?
• How can we lay out the heap? WebAssembly provides only essentially sbrk. We need more general address space layout. Where do we store information about spans, ptr/nonptr bits, etc.?
• Multithreading. WebAssembly has no notion of threads at the moment, which we would need to, for example, do concurrent GC. We'd also need locks and maybe atomic operations.
• How do we provide access to the outside world for packages like os and net?
• WebAssembly doesn't have support for "goto".

That's probably not a complete list.

[techtonik]

When we say that WebAssembly should work for Go programs, what we mean is that the Go compiler should generate WebAssembly code that can run in the browser.

@ianlancetaylor would it be easier to target non-web first? NaCl was also designed to be run in browser first, but then got browser-independent loader, which Go currently uses. WebAseembly without browser - http://webassembly.org/docs/non-web/ - and it speaks about minimal shells for testing.

@randall77 is it possible to bring an actionable checklist out of this list? Like a master issue with links to issues that research each aspect in detail further?

[sbinet]

@techtonik wrt non-web: @vibhavp is working on that in the context of the GSoC internship I was alluding to a few posts above.
it's over there: https://github.com/go-interpreter/wagon

[randall77]

@techtonik: I think the right place for such a checklist is a proposal document about supporting wasm. Once such a proposal is accepted and/or people are actively working on it, they could use issues for individual tasks, sure. That's premature at this point - I know of no one actively working on either (a proposal or code).

[joeblew99]

I agree about a proposal doc.
All the I do / contradictions are in this thread and just needs it wrapped together, with a roadmap that lets stages the risk / reward balance.

I would also like to see it address graphics too as it's sticking out like a sore thumb for me.

[neelance]

I'm the author of GopherJS, a compiler from Go to JavaScript. I have written that project because I strongly believe that there should be alternatives to using JavaScript in the browser, alternatives like Go and others. I have decent knowledge on compilers, SSA, machine architecture etc., but I'm probably missing a lot of details, since I haven't written a compiler backend for machine code yet. This is why I don't feel qualified to write a full proposal doc. Still, I'd love to get some critical feedback on my thoughts about Go and WebAssembly.

While looking at WebAssembly, it strikes me as being very different to the usual machine code targets like x86 or arm. For example its architecture is a stack machine instead of a register machine. This means that it isn't immediately suitable as simply yet another target at the last stage of the Go compiler next to x86 and friends. One solution might be to not put it there, but instead have a significantly different approach on generating WebAssembly. The code for this approach would need to live besides the existing compiler stages, which would be not nice at all. One might even consider a fork of the compiler in this scenario. In short: I don't see this happening.

There might be an alternative: Emulate what we need. We may use the stack machine to emulate a register machine and hopefully do so in a reasonably performant way. WebAssembly has linear memory with load and store instructions, which is good. We would not use WebAssembly's call instruction at all and instead roll our own stack management and call mechanism. Stacks would live on that linear memory and be managed by the Go runtime. The stackpointer would be a global variable. All code would live in a single WebAssembly function. The toplevel would be a giant switch statement (or WebAssembly's br_table based equivalent) with one branch for each function. Each function would have another switch statement with one branch per SSA basic block. There are some details that I'm omitting here, but in the big picture this looks like a decent register machine to me. Of course the performance of this heavily depends on how well WebAssembly can transform these constructs into actual machine code.

So please tell me: Am I completely naive and nuts? Then I'm happy to postpone my thoughts until I've learned more. If not, then this may serve as a starting point for some actual proposal doc. Thanks.

[nathany]

In addition to all the work needed on the Go side, there are some things WebAssembly may need before the Go runtime can target it.

WebAssembly doesn't yet have threads, but it's on the roadmap and there is a spec. https://github.com/WebAssembly/threads

Regarding garbage collection, it sounds like there could be multiple options in the future, based on Lin Clark's talk https://youtu.be/HktWin_LPf4?t=28m31s

28:31
So, today, you can ship down your own garbage collector with code if you want to but is
slow for a few reasons and the community group is making it possible for WebAssembly code
to be used with just the built-in GC which is a highly optimised one that the browsers
have been working on, so it will run fast and you will have that integration.

As with gomobile, there is the language bridge to figure out.

[neelance]

@randall77 also mentioned threads as a requirement for the GC. Is this really a hard requirement for the first version? Concurrency can also be done in a single thread.

Regarding the implementation of GC my personal opinion is that I don't believe in the one GC to rule them all. I'd rather prefer for Go to have a Go-specific GC and for Python to have a Python-specific GC. By completely managing our own heap and stacks on WebAssembly's linear memory we should be able to use the GC that Go already has, as far as I can currently see.

[randall77]

@neelance Yes multiple real WebAssembly threads are a nice-to-have, not a strict requirement.

[pmaddams]

The GC proposal for WebAssembly is still a work in progress. Interested parties can participate here:

https://github.com/WebAssembly/gc

[ghost]

@neelance, I know nothing about compilers, SSA, machine architecture, etc. Can you please tell how what you've described would affect binary size? If I'm understanding correctly the size efficiency was one of the high level goals of WebAssembly. It should be one of the priorities for this Go->WebAssembly compiler as well, right?

[neelance]

@alxkchr My experience is that the repetition due to code-gen boilerplate can be compensated a lot by applying gzip. Still, my proposal is not the most efficient solution with respect to binary size. Still, some solution that can be implemented in a reasonable amount of time is better than no solution, right? ;-) Also for me personally, binary size is not one of the first priorities.

[neelance]

fyi: I have started implementing this.

[SerkanSipahi]

@neelance best news ever :) do you follow the spec? (i hope)

[neelance]

The Go spec? It is a new backend/target for the normal Go compiler, so the spec is already in there. ;)

[SerkanSipahi]

i mean wasm spec :)

[stuart-warren]

https://github.com/neelance/go/tree/wasm-wip ?

[cznic]

@neelance

fyi: I have started implementing this.

Can you please share what approach will be used to address

Our current conclusion is that there is no efficient way to implement
setjmp/longjmp functionality currently, therefore it's not a viable target
of a gc port. We need to wait until it gets real stack unwinding and
exception handling support.

(https://github.com/golang/go/issues/18892#issuecomment-276858667)

[sbinet]

@neelance: I know it's still early days, but I think it would be wiser to name the wasm GOARCH wasm32 (ditto for the packages)

[dmitshur]

@sbinet It's worth considering that the ARM architectures/packages are named arm and arm64 (similarly mips and mips64) rather than arm32 and arm64. However, I don't know whether that should be considered a good or bad example.

[neelance]

@SerkanSipahi I'm initially targeting V8 (nodejs/chrome) and using https://webassembly.github.io/spec/ as documentation on what to do.

@stuart-warren Yes, my WIP is on that branch. However, don't try to use it yet, it can't be easily built right now and it is full of copied code and stubs/hacks. I'll announce it here when it reaches some alpha state.

@cznic As written above, I am not using wasm's call instruction at all, I'm managing my own stack on the linear memory. Thus setjmp/longjmp can be implemented.

@sbinet Actually it will be a 64 bit architecture since wasm has 64 bit operations.

[vibhavp]

@neelance not entirely. The WebAssembly spec only supports 32 bit address spaces for now, a wasm64 spec is planned for later.

[neelance]

Interesting, thanks for the link. We may want to revisit this later. Currently I store everything in 64 bit variables, so int, uint and uintptr all have a size of 64 bits. However, only the first 32 bits are used for addressing memory. This is easier to implement at first.

[extemporalgenome]

It sounds like wasm64p32 then (to follow the nacl naming), or something
similar.

On Nov 4, 2017 5:28 AM, "Richard Musiol" [e-mail] wrote:

Interesting, thanks for the link. We may want to revisit this later.
Currently I store everything in 64 bit variables, so int, uint and uintptr
all have a size of 64 bits. However, only the first 32 bits are used for
addressing memory. This is easier to implemented at first.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/18892#issuecomment-341889653, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAgwpPTbfHRmoYNXLQfcPMVnARxb0UGrks5szEpjgaJpZM4L0o7D
.

[FlorianUekermann]

It sounds like wasm64p32 then (to follow the nacl naming), or something
similar.

I don't think there is a point in having two numbers, because there will only be one architecture (with a choice of address space size). From webassembly.org:

wasm32 and wasm64 are both just modes of WebAssembly, to be selected by a flag in a module header, and don’t imply any semantics differences outside of how linear memory is handled.

[jfbastien]

I don't think there is a point in having two numbers, because there will only be one architecture (with a choice of address space size). From webassembly.org:

wasm32 and wasm64 are both just modes of WebAssembly, to be selected by a flag in a module header, and don’t imply any semantics differences outside of how linear memory is handled.

That's an aspirational statement. It might turn out to be true, but the WebAssembly CG / WG haven't explored wasm64 sufficiently for me to be confident that the statement is accurate.

[bradfitz]

Let's let @neelance work. Names are easy to change later.

[TomerHeber]

I know Go is compiled directly to machine instructions (AFAIK), but am wondering to myself if someone has ever compiled Go to C. Would be interesting to try Go --> C --> wasm (though not very efficient).

[kanocz]

@TomerHeber it can be possible in form of translation to C++, not compilation, but probably this will be more work than compilation to wasm itself

[neelance]

Hi everyone. Here's an update on my work: I'm making very good progress, which is very much due to the great work of the Go team and contributors so far. Most of the code is shared between architectures/platforms, so there wasn't as much as I expected that needed to be implemented.

Here's a list of some things that are already working fine:

• running the generated wasm code in browsers and in Node.js
• basic operations, conversions, etc.
• interfaces
• goroutines & channels
• defer/panic/rescue
• reading files form disk when using Node.js
• tests of the following packages are passing: bytes, container/heap, container/list, container/ring, encoding/ascii85, encoding/asn1, encoding/base32, encoding/binary, encoding/csv, encoding/hex, errors, flag, hash/adler32, hash/crc32, hash/crc64, hash/fnv, html, image, image/color, index/suffixarray, math, math/bits, path, sort, strconv, strings, text/scanner, text/tabwriter, unicode, unicode/utf8, unicode/utf16

Some things that still need work:

• reflection
• growing the stack of goroutines
• garbage collection
• performance optimizations
• wasm file size optimizations
• a nice JS interop layer
• many other issues to make more package tests pass

I am very happy about the progress in a month and a half, and only in my spare time. I think there is a good chance that we could get this into Go 1.11. Expect my next update in January, since I'll be on vacation and then there are the holidays.

[techtonik]

Can some of those tasks be parallelized? Some bits seems interesting for holidays hack.

[neelance]

Feel free to look into it and create a PR on my fork. You can also find me on the Gophers Slack.

One area of investigation might be optimizing the usage of the stack. Right now, the register allocation phase allocates registers (local variables in wasm) for every operation. get_local and set_local are always used before and after every operation. This is unnecessary if the value could simply stay on the stack to be used by some later operation. I suggest adding a pseudo register REG_STACK and make the instruction generator skip get_local and set_local if this register is used. Then make regalloc use this register when appropriate.

[leeola]

Re:

a nice JS interop layer

Is there a sane way to implement this for wasm currently? I thought Dom/js interop was a ways out with wasm still. If this is possible, that would be huge!

[neelance]

You can import and export functions which can have float64 and int32 arguments and you have the shared linear memory. Everything else can be built around that. Only garbage collection would not be that nice.

[sbinet]

(I have been trying to post this on your (@neelance) repo but I couldn't figure out how to post issues...)

would it be possible to use the wabt toolchain instead of calling/importing js.foo ?
I've found it easier to interact with wabt instead of a full nodejs installation :)

(I am also trying to interpret a simple main.go program, translated to a.wasm here: go-interpreter/wagon#36...)

[neelance]

I have enabled issues on my fork, feel free to post there.

I don't really understand your question. Are you talking about wasm-interp? Something needs to compile&run the wasm bytecode and you need some JS environment to interact with the rest of the system, e.g. for printing to console.

[neelance]

Btw: Have you seen https://github.com/neelance/go/tree/wasm-wip/misc/wasm?

[agnivade]

This is great. For those wanting to try your hand in compiling and executing Go to wasm -

1. Create a simple hello world program which prints to console.
2. GOOS=js GOARCH=wasm ./bin/go build -o out.wasm wasm.go
3. ./misc/wasm/go_js_wasm_exec out.wasm

Enjoy !

[neelance]

If you link go_js_wasm_exec into your PATH, then you can even use go run. ;-)

[ylluminate]

Hey guys, how's the garbage collection situation with Go for WASM? We started talking about bringing Crystal over and it appears that Crystal likely shares the same problems that Go has currently with GC. Would there be any way that we could collaborate or help the WebAssembly working group with this situation?

Please feel free to chime in on the thread noted above or let us know where we could perhaps chat / possibly collaborate on this issue.

It may be that the Opal path is the more sane approach for now until WASM matures in this area, but it would be good to know that for sure before going down that path.

[matthewmueller]

Hey folks, I wanted to share what I've been working the last few months since I think a lot of the underlying code is relevant: https://github.com/matthewmueller/joy

I'm targeting JS (ES3) right now, but the compiler implements a dependency graph for all the declarations (funcs, variables, etc.). It then sorts the graph and translates only the declarations that are used in the program.

I don't think it would be too difficult to use this graph and target WebAssembly. I'm eager to help and collaborate in anyway that I can.

For more information here are some relevant links:

• https://mat.tm/joy/#faq-webassembly
• https://mat.tm/joy/#how

[ylluminate]

Regarding the GC issue for Crystal above, this might also have application for Go, note @kripken's remarks and the following discussion here: https://github.com/WebAssembly/binaryen/issues/1312#issuecomment-348409211

[johanbrandhorst]

Potentially we'll want to update https://github.com/AppCypher/webassemblylanguages?

[neelance]

I hope everyone had nice holidays. Here's the promised update:

Since my last update, I managed to get reflection, stack-growing and the garbage collection up and running. Additional improvements got it to the point that the tests of 104 out of 136 stdlib packages are passing now. Also, many of the compiler tests are green. Good progress!

Please keep in mind that I am currently focusing on feature support and correctness. I haven't done any optimizations for performance or output size yet.

[shelby3]

I applaud the experiment (and @neelance ty so much for GopherJS for the superior goroutines as compared to Promise, you probably rescued me from C++!), so the following is merely at an attempt to add potentially useful information and perhaps even maybe to raise awareness of the incongruence between Go and a WASM target. Also potentially to clarify the issue for some readers. Also because I wanted some appropriate place to dump the following @rossberg quote I found when I was independently thinking about how to compile something like gorountines to WASM.

@neelance replied:

@cznic replied:

@neelance wrote:

We would not use WebAssembly's call instruction at all and instead roll our own stack management and call mechanism. Stacks would live on that linear memory and be managed by the Go runtime. The stackpointer would be a global variable. All code would live in a single WebAssembly function. The toplevel would be a giant switch statement (or WebAssembly's br_table based equivalent) with one branch for each function. Each function would have another switch statement with one branch per SSA basic block.

Can you please share what approach will be used to address:

@minux wrote:

Our current conclusion is that there is no efficient way to implement
setjmp/longjmp functionality currently, therefore it's not a viable target
of a gc port. We need to wait until it gets real stack unwinding and
exception handling support.

@cznic As written above, I am not using wasm's call instruction at all, I'm managing my own stack on the linear memory. Thus setjmp/longjmp can be implemented.

According to the “co-designer” and “spec author” of WASM, the limitations on call, which make it inappropriate for Go, have something to do with security/safety:

@rossberg wrote:

You can't easily "replace" the call stack, as there is no way to reify return addresses (*). However, you can implement a shadow stack for your local variables in linear memory if you need to.

In fact, WebAssembly does not currently expose any notion of built-in "stack" at all. Like a number of other design decisions, that's important: because Wasm is running on the Web, it has to be "safe". That's why it's typed, has no undefined behaviour, code is separated from memory (and not addressable), casts over function types are checked, etc. Having an explicit concept of function also enables other benefits, such as effective register allocation, parallel or lazy compilation, useful debug tools, etc.

Control-flow indeed is semi-structured: branches are really only breaks and continues, so that you cannot construct irreducible control flow. That's a feature as well.

[…]

(*) You could presumably emulate explicit return addresses with indices into a giant table jump, by compiling your whole program into a single looping function. But that would probably be pretty slow and defeat much of the WebAssembly ecosystem.

So the performance and interoperability with the WASM ecosystem of what @neelance is attempting will probably be suboptimal?

Citing @neelance’s skill optimizing the performance of GopherJS.

[neelance]

I don't know what you mean by "interoperability with the WASM ecosystem", so I can't comment on that. Regarding performance I can tell you this:

Yes, the current approach is not the most optimal way to express control flow in WebAssembly, but it works today. Maybe it will be possible in the future to use WebAssembly's call instruction more, but for that, WebAssembly would need to add a few more features and it would need much more work on the Go side to be compatible with that. I'd rather have something with 80% of the performance that actually works than something that has 100% but is only hypothetical. ;-)

If you're interested in more technical details, talk to me on #webassembly at https://invite.slack.golangbridge.org/.

[shelby3]

I don't know what you mean by "interoperability with the WASM ecosystem", so I can't comment on that.

Perhaps what @rossberg is referring to with _“defeat much of the WebAssembly ecosystem”_, is interoperability with tools and other languages in the ecosystem which are not side-stepping call and the protected call stack by essentially emulating your own continuations employing switch tables?

but for that, WebAssembly would need to add a few more features

I hope that WASM would better accommodate languages with goroutines (green threads), because afaics they’re unarguably superior to the stack unwinding JavaScript Promise model.

I'd rather have something with 80% of the performance that actually works than something that has 100% but is only hypothetical. ;-)

Oh I entirely agree which is why I made sure I prefaced my prior post with my applause for your prolific efforts.

The more popular on the web/WASM, we can make Go or other languages that have green threads, then perhaps the better chance we have of getting better WASM primitives (e.g. analogous to gcc’s -fsplit-stack?) for better performance eventually.

If your implementation truly achieves 80% of the optimal performance within the current limitations of WASM, I will be pleasantly surprised. If the performance outcome is excessively bad, then it might limit the demonstration of the popularity of green threads versus JavaScript Promise callback based model (the potential of the irony of “defeat much of the WebAssembly ecosystem” if we presume it’s design has been driven by a priority on optimizing JavaScript :-) .

Note also that I’m contemplating (no firm decision yet) adding typeclass generics to Go as a transpiler (and providing my analysis of the generics proposal for Go 2), so I’m also potentially involved in doing my part to try to increase popularity.

[dolmen]

@shelby3 Please stop using formatting that makes your text so small that is not readable. If you consider that something is not worth for people to read, don't write it in the first place. Making it unreadable without giving a hint to what is hidden make readers spend twice (or more) time trying to decypher it. I think it is the opposite of your original intent.

[andybons]

@shelby3 I’ve edited your posts to stop using the small text formatting. Please stop doing that.

[gopherbot]

Change https://golang.org/cl/102835 mentions this issue: go/build, cmd/dist: add js/wasm architecture

[gopherbot]

Change https://golang.org/cl/103255 mentions this issue: wasm: add scripts for running WebAssembly binaries

[gopherbot]

Change https://golang.org/cl/103256 mentions this issue: cmd/compile: add SSA config options noAvg and noHmul

[gopherbot]

Change https://golang.org/cl/103275 mentions this issue: cmd/compile/internal/gc: factor out beginning of SSAGenState.Call

[gopherbot]

Change https://golang.org/cl/103295 mentions this issue: cmd/compile: add wasm architecture

[gopherbot]

Change https://golang.org/cl/103535 mentions this issue: cmd/compile: wasm stack optimization

[gopherbot]

Change https://golang.org/cl/103795 mentions this issue: cmd/link: add wasm architecture

[gopherbot]

Change https://golang.org/cl/103877 mentions this issue: runtime: add js/wasm architecture

[gopherbot]

Change https://golang.org/cl/103915 mentions this issue: internal/bytealg: add wasm architecture

[randall77]

Ok, looks like a lot of CLs are starting to fly by.
How are we doing with respect to the porting policy: https://github.com/golang/go/wiki/PortingPolicy ?
In particular, what will the builders look like?
I assume @neelance that you're signing up for maintaining the port?
(My apologies if one of the CLs spells all this out, I haven't read all of them.)

[neelance]

Yes, I'm going to maintain the port. @bradfitz is currently looking into setting up the builder.

[gopherbot]

Change https://golang.org/cl/106995 mentions this issue: sync/atomic: add wasm architecture

[gopherbot]

Change https://golang.org/cl/106996 mentions this issue: math: add wasm architecture

[gopherbot]

Change https://golang.org/cl/106997 mentions this issue: time: add wasm architecture

[gopherbot]

Change https://golang.org/cl/106998 mentions this issue: mime: add wasm architecture

[gopherbot]

Change https://golang.org/cl/109195 mentions this issue: syscall/js: add package

[gopherbot]

Change https://golang.org/cl/109976 mentions this issue: syscall: enable some nacl code to be shared with js/wasm

[gopherbot]

Change https://golang.org/cl/109977 mentions this issue: os: add js/wasm architecture

[gopherbot]

Change https://golang.org/cl/109995 mentions this issue: net: add js/wasm architecture

[gopherbot]

Change https://golang.org/cl/110095 mentions this issue: crypto: add js/wasm architecture

[gopherbot]

Change https://golang.org/cl/110096 mentions this issue: all: skip unsupported tests for js/wasm

[gopherbot]

Change https://golang.org/cl/112736 mentions this issue: env/js-wasm, dashboard: add start of a js-wasm builder

[gopherbot]

Change https://golang.org/cl/113515 mentions this issue: misc/wasm: make wasm_exec.js more flexible

[SCKelemen]

After reading the webassembly architecture for Go document, specifically for JavaScript syscall implementation being built on Node's 'fs' module, and being largely unimplemented for the browser side of things, I am left wondering if we could or would want to try to implement these using the _new_ browser APIs? Specifically, the FileSystem API specification seems like it could support at least some of the system calls. (FileSystem is still an Editor's Draft and is only implemented on FireFox, Chrome, and Opera. Furthermore, it is 'webkit-' prefixed on the latter two.)
I could attempt to make a proof of concept, if you think this might be a decent approach.

[gopherbot]

Change https://golang.org/cl/114197 mentions this issue: runtime, sycall/js: add support for callbacks from JavaScript

[cmaster11]

Hi everyone, I'm loving the job done by @neelance , and I was wondering if with this WASM implementation (let's say, the first release of it), it will be possible to make use of cgo too. Embed C code, compile to GO, compile to WASM. That'd be a damn superpower. You're all doing an awesome job. Awesome.

[neelance]

@cmaster11 Happy to hear that you like it. :) On your question: cgo does not compile C to Go, instead it compiles both languages to machine code with their respective compilers and then merges the results into a single binary. You can already use emscripten to compile C code to WebAssembly. In theory it should be possible to load both WebAssembly binaries at runtime and make them interact. However, I think it is unlikely that the Go project will spend time on facilitating this use case in the near future.

[PierBover]

apparently mostly Millennials

@shelby3 I'm probably older than you and I downvoted you. The problem here is simply your inability to communicate.

It's not a matter of power. @andybons actually made you a favor by editing your post so that it can be read by others.

[omac777]

js/wasm support without webidl implies we can't interoperate much with other languages from within the web browser. The wasm architecture did not seem to mention anything about webidl.

I took a good look at the emscripten webidl python tool when applied to c++ Foo class as follows.
I would love to see a golang duck type exposed via webidl in the same manner as the Foo class.
That rationale would be to interoperate with other wasm modules that are loaded at the same time, but we need the idl's available in order to be able to include and call their marshalling stubs. Otherwise we need some kind of object viewer to determine what call signatures are available from the different wasm files.

python /usr/lib/emscripten/tools/webidl_binder.py Foo.idl glue
This generates:
glue.cpp
glue.js
WebIDLGrammar.pkl

emcc -v Foo.cpp my_glue_wrapper.cpp --post-js glue.js -o output.js
This generates:
output.js
output.wasm

emrun --list_browsers
emrun --browser chrome handcrafted_Foo.html 
emrun --browser firefox handcrafted_Foo.html 
firefox handcrafted_Foo.html &
chrome handcrafted_Foo.html &

$ cat Foo.h

#ifndef FOO_H
#define FOO_H (1)

class Foo {
public:
  int getVal();
  void setVal(int v);
 private:
  int nValue;
};

#endif

$ cat Foo.cpp

#include "Foo.h"

int Foo::getVal() {
  return nValue;
}

void Foo::setVal(int v) {
  nValue = v;
}

$ cat my_glue_wrapper.cpp

#include "Foo.h"
#include "glue.cpp"

$ cat Foo.idl

interface Foo {
        void Foo();
        long getVal();
        void setVal(long v);
};

$ cat handcrafted_Foo.html

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>WebAssembly Sample</title>
  </head>

  <body>
    <h1>Web Assembly</h1>

    <div class="output">
        <pre id="log"></pre>
    </div>

    <script src="output.js"></script>
    <script>      
      "use strict";


    function log() {
        document.querySelector('#log').textContent += Array.prototype.join.call(arguments, '') + '\n';
    }

    Module.onRuntimeInitialized = _ => {
      log(`blah `);
      var f = new Module.Foo();
      f.setVal(200);
      alert(f.getVal());
      log(`blah `);
    };

    </script>

  </body>
</html>

[bradfitz]

@omac777, interop with other languages is not a goal for Go's 1.11 initial WebAssembly support.

[omac777]

Ok. I can understand webidl isn't a short-term goal, but I do have questions about how to make wasm output useful. Firstly going back to the above Foo class defined above for use with emscripten c++ along with its idl. Say I created a golang duck type that matched it. Then say I export those functions as C functions so they are callable from the C++ Foo SetVal and GetVal implementations. I know it would be slower as a result, but trying to get all the implementation built within go would be the long-term goal.

How do I create foo.go.wasm.a's and foo.go.wasm.so's? Does every foo.wasm need its own foo.go.js?
If a library full of different foos were generated, then would there be a otherfoos.go.a.js generated also?

How do I link go.wasm.a and go.wasm.so to the generated emcc output.wasm as specified in:
emcc -v Foo.cpp my_glue_wrapper.cpp --post-js glue.js -o output.js foo.go.wasm otherfoos.go.wasm.a

How to link in sdl functions that emcc supports from wasm within go for example.

When will the support for full goroutines, MAXCPUCores, channels appear for wasm?
Thanks again.

[bradfitz]

@omac777, what is in scope for Go 1.11 is this:

Go code compiles to a WebAssembly module and it runs in a browser, and we can call back and forth between JavaScript.

Ignore anything about *.a or *.so files or interop with C or GCC or emcc or SDL. None of that is in scope and won't work.

When will the support for full goroutines, MAXCPUCores, channels appear for wasm?

It fully supports goroutines and channels (that's part of Go and is required). WebAssembly only supports a single core, if that's what you mean by MAXCPUCores. That is not a limitation of Go or Go's WebAssembly support.

[omac777]

So what you're saying is I won't see anything like this any time soon?
GOARCH=wasm GOOS=js go build -o awesome.wasm.so -buildmode=c-shared foo.go

So if we have web browsers with 4 cores or 20 cores, the go generated wasm will only use one of those cores in the web browser? I don't understand how you state it is not a limitation of go or go webassembly support. Are you stating it is a limitation of WEBASSEMBLY.org spec itself?

[mohamedattahri]

@omac777 - Yes, I believe it's currently a limitation of WebAssembly itself, not the Go implementation.

[RangerMauve]

@omac777 Multithreading support in WASM is a work in progress and it'll be a while until it's released in browsers.

https://github.com/WebAssembly/threads/blob/master/proposals/threads/Overview.md

[cherrymui]

So what you're saying is I won't see anything like this any time soon?
GOARCH=wasm GOOS=js go build -o awesome.wasm.so -buildmode=c-shared foo.go

Probably in Go 1.12.

For Go calling C/C++, I think it is probably possible to wrap C functions with JavaScript, then call the JavaScript wrapper from Go, with Go 1.11.

[SerkanSipahi]

i know this is not part (maybe) of the thread but is there any documentation/tutorials how to use WebAssembly (compiling, etc) in go v1.11?

[bradfitz]

@SerkanSipahi, not yet. But there will be docs before the release.

[lazyhacker]

@SerkanSipahi I wrote down the steps I took to get it working at https://blog.lazyhacker.com/2018/05/webassembly-wasm-with-go.html.

[sbinet]

There's also this post, still relevant:
https://blog.gopheracademy.com/advent-2017/go-wasm/

[sbinet]

it occured to me we haven't (IIRC) specified exactly the mapping (nor the mechanism) for what goes into the export section of a wasm module.

I've filed #25612.

[omac777]

Using the wasm target, please clarify/document from go how to:
-use go to export/import a go function to and from wasm for other languages to use.
-link sdl2/webgl/qt to a golang binary
-catch all the sdl2/webgl/qt events
-temporarily relinquish cpu time back to the os with a sleep if we're in a loop. for example
in emscripten emscripten_sleep(10)

The output.html file from the go tool doesn't seem to succeed consistently with both firefox/chrome even if there was a successful output.html built. More should be done to ensure that output.html works. before presenting it as a success.

[bradfitz]

@omac777, we've been over most of that above. But for others just tuning in:

Using the wasm target, please clarify/document from go how to:
-use go to export/import a go function to and from wasm for other languages to use.
-link sdl2/webgl/qt to a golang binary
-catch all the sdl2/webgl/qt events

Per above, none of that is in scope for Go 1.11. It won't be supported, documented, or likely even possible.

-temporarily relinquish cpu time back to the os with a sleep if we're in a loop. for example

@neelance got callbacks & interop between JS and the Go scheduler working, so regular time.Sleep and friends should work as expected.

[omac777]

I understand Go 1.11 won't have any of these, but how far/long will any of the above be considered for go. There are people who would love to see qt cleanly integrated with go and currently the situation is we are relying on a third-party for that therecipe/qt which is all good but there are limitations (currently no wasm qt and no qt 64bit arm targets supported). autocad has a webassembly solution for windows and macos using emscripten, c++ and react. I'm still feeling golang is still constrained for certain areas of use as opposed to where c++ can be used.

There is much power to be had is c++'s approach:
emcc --clear-cache --clear-ports
em++ -v -std=c++1y hello_world_sdl.cpp -s USE_SDL=2 -s USE_SDL_IMAGE=2 EMTERPRETIFY=1 -s EMTERPRETIFY_ASYNC=1 -s WASM=1 -o hello_world_sdl.html

Why can't golang have similar behaviour?
UPDATE: I stand corrected. Golang doesn't need "-s" switches because of the cgo compile directives embedded into go code.

The clear cache and clear ports however are a good idea since there might be bugs introduced when migrating to different golang build versions i.e. go1.9 to go1.10. go1.10.3 to go1.11(when we migrate to vgo lifestyle). I recently had to do a "go build -a" between go build versions.
GOARCH=wasm GOOS=js go --clear-cache --clear-ports

[bradfitz]

Why can't golang have similar behaviour?
GOARCH=wasm GOOS=js go --clear-cache --clear-ports

FWIW, Go's cache doesn't require explicit clearing for correctness.

[agnivade]

@neelance - Now that callbacks are in, I believe basic support is complete. Shall we close this bug or did you have something else in mind ?

[neelance]

Yes, I think we can close this now. 🎉

[Jinex2012]

Great work!!! @neelance

[alecthomas]

This is awesome, great work @neelance!

[gopherbot]

Change https://golang.org/cl/120057 mentions this issue: doc/go1.11: mention GOOS/GOARCH values of WebAssembly port explicitly

[gopherbot]

Change https://golang.org/cl/120575 mentions this issue: cmd/dist: skip non-std tests on js/wasm

[matthewp]

Is it appropriate to open an issue on reducing the file size (if that is even possible)? Or is that being tracked elsewhere?

[bradfitz]

@matthewp, feel free to open a webassembly-specific size bug if you'd like. The general one is #6853.

[gopherbot]

Change https://golang.org/cl/120958 mentions this issue: net: re-implement built-in simulated network on JS and NaCl

[omac777]

Does the golang compiler have built-in control flow integrity(CFI)? I recently skimmed through the following to do with vulnerabilities in WASM prevented with CFI built-into the clang compiler:
https://www.fastly.com/blog/hijacking-control-flow-webassembly-program
https://github.com/trailofbits/clang-cfi-showcase/blob/master/cfi_vcall.cpp

[neelance]

Just skimmed it, but it seems to me as if the post says "WebAssembly is not a high level language such as JavaScript, but a low level language, so certain classes of bugs may apply if the source language (C or Go) allows them." No surprise there. This means that if Go would have such issues, they would also apply to other architectures, not only WebAssembly.

[randall77]

WebAssembly is mostly about defending the browser from nefarious WebAssembly code. It is not so much about providing guarantees within the WebAsssembly instance (although there is some of that). As @neelance said, it is up to the LanguageX->WebAssembly compilers to provide any required safety guarantees.
Go does not have CFI. I don't think it would be easy - at the point where a method starts, we've already lost the vtable. We've even lost the fact that a vtable was used.
Go is vulnerable to this exploit, in cases where there is any parallelism. For now, at least, the WebAssembly Go port (and WebAssembly, full stop) does not have parallelism so this exploit is only theoretical. The WebAssembly folks do plan to implement parallelism at some point, though.

[neelance]

@randall77 How does this affect plain linux/amd64? Is the risk somehow larger for WebAssembly?

[randall77]

The risk is the same regardless of architecture (with the exception that WebAssembly doesn't have threads yet, so it's actually zero for WebAssembly for the moment). Data races can be used to simulate unsafe, without importing unsafe.
It would be hard to exploit such a vulnerability. You either need to link in untrusted code, or somehow find and trigger a gadget in an existing binary that does a data race on an interface value and then invoke a method on the result.

[neelance]

Okay, so if your Go code has no data races, then it should be fine. Also, when running untrusted code with WebAssembly you can make sure that it can not escape the WebAssembly environment, even though it is able to have data races or simply use unsafe. Thanks for the interesting insights and sorry for this being slightly off-topic for this issue.

[omac777]

The more I examine this javascript usage from within golang, the more I perceive as an infection that will degrade the quality and the long-term maintenance of the golang code. Let me explain. Here is an example of a wonderful wasm golang app that does some impressive eye candy magic.

https://github.com/stdiopt/gowasm-experiments/blob/master/bouncy/main.go

The results are impressive. Delving furthing into the code that actually makes it happen though was disappointing because it require that every time you want to call something from the javascript side, it requires you to describe the name of the javascript function or the javascript value as a string AT EVERY TURN. As a result, there is no way for the golang compiler to check javascript correctness at compile time. It's "fly by prayer" at runtime. I would prefer to see golang functions/variables/types everywhere for long-term maintenance otherwise it defeats the purpose of using golang for wasm in my humble opinion. BTW I detest javascript and always have. It's too freestyle. Why is it that such a language has persisted in the web browsers? The answer escapes me.

Thank you for listening.

[geovanisouza92]

I agree that calling JS using strings isn't the better experience. Maybe we could generate Go interfaces from web platform specification IDL's. The downside is how to keep up with the spec, because depending on the browser it will run, the API's isn't available, while on the other side, new spec keep coming all the time as the web platform evolves.

[neelance]

@omac777 I agree with you. That's why I hope that there will be nice Go libraries around the JS low levels so most users will never have to touch syscall/js directly. Similar to the syscall package, it is ugly and almost no one uses it directly. ;-)

[dmitshur]

@omac777 For GopherJS, many people use high-level bindings for various browser APIs rather than the low-level js package directly. I suspect a similar approach will be most popular with Wasm too. I expect many of the GopherJS bindings will add support for Wasm in the same package, and new packages will be created as needed.

For reference, see https://github.com/gopherjs/gopherjs/wiki/Bindings, https://dmitri.shuralyov.com/talks/2016/Go-in-the-browser/Go-in-the-browser.slide#11 (and the following 4 slides), and https://github.com/dominikh/go-js-dom/issues/57.

[guidorice]

Also we should consider there is a bunch of stuff on the webassembly roadmap:

https://webassembly.org/docs/future-features/

that will improve the syscall story quite a lot, in the long run, like

reference DOM and other Web API objects directly from WebAssembly code;
call Web APIs (passing primitives or DOM/GC/Web API objects) directly from WebAssembly without calling through JavaScript; and
efficiently allocate and manipulate GC objects directly from WebAssembly code.

Anyways, thanks @neelance et. al. for implementing the initial version. So awesome! 🥇

[fractalbach]

What if there was a Go implementation of the DOM?

Documents and scripts could be written and executed in Go, making use of the type system.

Then, code could be generated from the Go DOM.

[yookoala]

@fractalbach: If the WebAssembly's host binding is confirmed and implemented by the work group, you can actually do DOM manipulation through WebAssembly. Then it would be reasonable to have high level libraries to abstract DOM, Documents and scripts.

But before that, it is not quite as appealing to do.

[omac777]

Go WebAssembly: Binding structures to JS references

https://medium.com/@nlepage/go-webassembly-binding-structures-to-js-references-4eddd6fd4d23

The approach is appealing. The usage is similar to json marshallling/unmarshalling within structs. That makes it an easily transferable skill.

The one thing missing is how to apply the same approach to duck-typing? I would prefer not to have the function declarations within the structure, but outside of it standalone in the same manner as duck-typing. If the service exists, it can use it. That way it is easily modifiable without affecting the core struct holding the different desired non-function values. Duck-typing rocks!

Thank again @neelance for all your work. It's greatly appreciated.
Thank you Mr. Nicolas Lepage for your points of view. They really help to crystallize what could be a better path for interoperating with all this javascript noise for the interim until the direct wasm interfacing is complete.

[agnivade]

Hello everyone, this is a high traffic thread with a lot of people subscribed. Having a general discussion like this is not fruitful to those just concerned about the original issue - which is to have WebAssembly support for Go.

Please feel free to continue the discussion in appropriate forums - golang-nuts/golang-dev. Or if you have something specific in mind, please open a new issue.

Thank you.