Crystal: Plans about WebAssembly

[sergey-kucher]

Do you have any plans about WebAssembly support?

[vendethiel]

LLVM already has a branch for that, so Crystal should be covered already ;)

[asterite]

Maybe when there's a WebAssembly specification we can start discussing this, but right now Crystal is more geared towards server and desktop applications, not the web. I'll close this until then.

[Svenskunganka]

A year later, now that a spec is in the works I figured that we could open discussion for this again? Chrome Canary and Firefox Nightly has experimental support for WebAssembly.

@asterite I understand that this is probably not a priority right now - but is this something you guys might be interested in the future?

[lbguilherme]

I believe there is not much to be done from crystal side until both browsers and the llvm target get mature enough. Clang is not ready for this yet.

We can for sure make a proof of concept that works, but not maintain it or port the whole stdlib to work on it yet. I suggest creating a milestone "post-1.0" and leaving this issue open there.

[vendethiel]

Clang is not ready for this yet.

Not sure how Clang is related to Crystal? Once it gets down to LLVM IR, does it matter?

[asterite]

@Svenskunganka I guess so, if you could write Crystal to target the browser it would be awesome. I know nothing about WebAseembly, and just a bit about asm.js, but I don't know what kind of things you can do... dom manipulation? Or just games? In any case, I think we should wait a bit.

[lbguilherme]

@vendethiel It is not in any way related or needed for crystal to work. But it is the one compiler that is working most towards targeting wasm. In fact, this is the goal of the first release of wasm: Have clang targeting wasm and running it in multiple browsers. It is safer and less work for us to let clang test the ground, solve most bugs, and get there before us.

@asterite It let's you access all standard web apis thought some kind of "LibJS", that includes DOM manipulation and everything else. All the stdlib (libc included, I think) would have to be implemented on top of that.

[jwaldrip]

@waj It may be a time to revisit/reopen this issue.

[RX14]

@jwaldrip why?

[pannous]

because the standards are now fixed and the toolchain is working nicely.
full support in most browsers and node.js

There is still a big opportunity to become _the_ language of the binary Internet, racing against c++, rust and typescript.

[Svenskunganka]

Today WebAssembly is available in all the major stable browser releases, however its features are limited. It currently has no way to access the DOM or the Web APIs except if you're using JavaScript as the middle man. The WebAssembly Roadmap looks very promising however. Planned features are to allow wasm to directly access and modify the DOM, and call Web APIs without needing any JS at all.

Wasm has an (experimental) LLVM backend, and since Crystal uses LLVM I can imagine it would ease the work required to compile to wasm. Wasm post-MVP roadmap shows plans to implement access to low-level GC, however I'm not sure how this would interact with the Boehm GC that Crystal currently uses. I imagine that a new GC would have to be built specifically for wasm, or that the Boehm GC is adjusted to support the wasm "runtime". But this is only speculation, I'm not knowledgeable enough in that area to draw a conclusion.

Looking at these in progress features, the possibilities are very promising. Being able to compile to wasm would allow the community to create "isomorphic, fullstack" Meteor-like frameworks that shares the same code on both server & client (browser). To simplify, it would allow Crystal to do everything that JS currently does, like having a React-like library with full server-side rendering support. Other examples would be a Slack alternative that compiles to both the browser and regular platforms like Linux, Windows and Mac. I would imagine that Wasm would also be available on smartphones like iOS and Android through WKWebView and the like, so that Slack client could perhaps also be able to run on those platforms using the Wasm code.

It's worth mentioning though that Wasm will not be able to do things that JS currently cannot, like opening a raw TCP socket. We would have to use the alternatives the Web APIs provide like WebSocket and WebRTC.

[drhuffman12]

:+1

[akzhan]

@ysbaddaden what about wasm LLVM target? looks like it's a time to do.

[rhysd]

As long as I know, LLVM 4.0.0 does not support wasm. LLVM HEAD already supports it. So it may be available in 4.0.1 but I'm not sure. I remember LLVM is now in semantic versioning. So I guess it comes with 5.0.0 for stable release.

[ysbaddaden]

Same as other arch's: LLVM does the underlying work, that is generate the binary code (it's awesome), but there is far more to Crystal than just the language: it's library.

Without an official libc (the musl port is unofficial and experimental, advised not to use), we're left with the same state than embedded systems: most of the core and stdlib won't be available, the GC will likely not work, ... not even thinking of the fiber context switches ASM (cold sweat).

WASM may be fun, but it's far off the Crystal roadmap. If someone is courageous (crazy?) enough to work on this, please try, you'll have fun (maybe) :-)

[pannous]

isn't libc provided by emscripten? how else would it run hello_world.c ?

Quote:

Emscripten lets you "link" with JavaScript libraries to do useful things, like render WebGL, run a browser main loop, handle a filesystem, provide bindings to JS so it's easy to call into your compiled code, etc., as well as a full set of compiled libraries like libc and so forth.'

GC can be added later.

[pannous]

Here is how rust does it https://github.com/brson/mir2wasm/

[RX14]

@pannous rust doesn't need a gc.

If someone can demonstrate libgc compiled and working on wasm then this endeavour becomes actually credible.

[pannous]

can't we start without GC? should be fine for demo code which doesn't run for extended periods of time?

[akzhan]

BDWGC on Emscripten (LLVM to WebAssembly backend) now fails on MacOS hosts (but should be OK on others - https://github.com/ivmai/bdwgc/issues/163#issuecomment-304686844).

I suppose it will be fixed with https://github.com/ivmai/bdwgc/issues/163.

[RX14]

So it looks like wasm doesn't have a normal stack at all. I have no idea how we would implement fibers in such an architecture. The note on the GC issue saying that the GC can't walk the stack to find roots is basically a show stopper too. Implementing crystal on wasm looks like it'd require a great deal of thought.

[pannous]

yep, no GC and fibres/threads in first version:
http://webassembly.org/docs/future-features/

I'm fine with those crystal features missing in the first adaptation.

[akzhan]

Looks like Crystal cannot be provided until EH, GC and coroutines landed to WebAssembly.

Just wait for these features (they will be added to browsers fast, I suppose).

[RX14]

@pannous I doubt most people would be OK with that or call it a "complete" crystal port so I don't think it's worth the effort at this point in crystal's development. If you or anyone else wishes to work on it i'm sure the core team would be happy to help or merge PRs.

[RX14]

@akzhan I doubt fibers are impossible in webassembly as-is or would get any kind of special support in the future. It's just a matter of finding out how to do it. GC and direct DOM access are things we should wait for.

[catmando]

if anybody else is interested in collaborating on a crystal -> JS transpiler let me know. I think it would have to be a subset without fibers initially. My gut feeling is that we be best to leverage the work down by the opal (ruby -> js) transpiler. I am guessing the major problem is that you have to map crystal class structures, method calls, etc -> JS equivilents, rather than trying to do some lower level emulation. This is what opal does.

[vendethiel]

if anybody else is interested in collaborating on a crystal -> JS transpiler let me know.

wouldn't it be easier to find a way to get a GC/wait for GC?

My gut feeling is that we be best to leverage the work down by the opal (ruby -> js) transpiler.

maybe parts of the parser can be reused, but i don't see anything else

[badlee]

👍 +1

[ylluminate]

Not sure if you guys have seen this thread yet: https://github.com/WebAssembly/binaryen/issues/1312

I believe we can get through the GC issue now with the help as noted there.

Note that by using emscripten we will be able to still run Crystal on non-WASM supporting browsers that only use pure JS.

[RX14]

I'll reopen this for now since wasm has a full spec and implementations in browsers.

[larubujo]

jack of all trades, master of none

[RX14]

@larubujo that's a great reason for not implementing or focussing on implementing wasm now or before 1.0, but I see no harm in keeping this issue open as a place to discuss possibilities. I've made it clear I'm not going to put any effort into wasm personally.

[catmando]

I don't get it... sorry.
WASM provides a way to get crystal running in the browser
Crystal running in the browser would greatly speed adoption by making it a full-stack language.
Crystal in browser would allow for a crystal React, DSL and a system similar to ruby-hyperloop to exist.

Understand that you can't do everything at once, but this does seem like it would greatly speed adoption.

What am I missing

[straight-shoota]

Widespread adaption is not an immediate goal (for the core team) - getting a stable 1.0 release of the language is. And even that is conceiveably not really immediate.

As I understand it, WASM support is not considered a critical feature for 1.0, and that seems logical. In the end, it is just another backend Crystal can compile to.

Is WASM really going to be that huge deal? I don't know.
Windows support seems certainly much more important and is a requirement for 1.0.

Now, this should not hold anyone off from working on WASM support. If you manage to accomplish something in that direction it will certainly be appreciated. And if it get's done before 1.0 release all the better.

But it's not a high priority for core developers currently. There are more than enough things to work on.

[pannous]

"Windows support seems certainly much more important"
An external observer (and fan of the language) dares to strongly disagree here.

• if you compile to wasm, crystal programs can automatically run on all platforms including windows. without installation!
• this is huge! wasm is going to be huge. prediction: it will replace the JVM as the most popular 'portable' platform.
• there are already hundreds of languages running on windows
• right now there is almost no language compiling to wasm

Don't miss the opportunity guys.

If you target wasm you get Windows for free. Also with Ubuntu subsytem you can already compile crystal on Windows? So why waste your valuable time on something almost obsolete, when you can change the world?

[ylluminarious]

Can't agree with @pannous enough. It's really sad to see how many open-source projects dwindle because of a lack of proper marketing. Crystal would be very astute to take advantage of wasm and not just scoff at it as a far-off milestone.

Crystal certainly is at a point where it could take programming languages by storm. It's even higher than Rust and several others on the TIOBE index right now. That could all change in an instant, though. And I bet it will have something to do with whether Crystal runs on wasm.

[ylluminarious]

@catmando is also very on-point and I share his confusion at the resistance to this issue.

[ylluminate]

So we're going to get this done. We have several commitments now from people. If you'd like to get involved, shoot me a note at my username at gmail and we'll get the collaboration going.

[RX14]

The problem I forsee is the standard library. We haven't got most of the stdlib portable to non-posix OSes, let alone the browser. The browser isn't a traditional OS, it has no filesystem and wasm's memory model is pretty wacky compared to the tradtional one.

It'll be extremely hard but if we can port crystal to wasm it'll mean that we know that crystal is extremely portable.

[ylluminate]

@RX14 this is precisely where I see empscripten coming in strong since it provides those for us. Then it will just be a matter of testing and benchmarking to see if we need to do something different, etc.

[RX14]

Oh, emscripten provides a libc I guess. Fair enough, thats probably easier than I thought.

[ghost]

kotlin (native) has wasm support and has garbage collection as well, maybe their approach helps

[catmando]

Stable 1.0 release vs. Wide Spread Adoption.

These are not contradictory. Wide Spread Adoption means you have more people interested in contributing, you have more people using in "anger", you get more realistic use cases, and find those edge case flaws you didn't think of until you get wider usage. This is all builds momentum towards the 1.0 release.

Huge advantage of crystal that everybody talks about is SPEED. Well if you can offload computation to the client's browser, that is really going to give you speed.

[theodorton]

Code sharing between client and server application is IMO a huge argument for targeting wasm. It seems like game development is one of the big applications on WebAssembly and from my little experience with Crystal it seems like it would be a great platform for developing games.

I did some proof-of-concept work on a Crystal to Javascript transpiler last year. My original roadmap was to have a transpiler as 1.0 and convert to tooling for wasm with a 2.0 release as the only goal of the project was to write Crystal code that could be executed one way or another in a browser environment. I'd be happy to help with any work needed. Although my experience with low-level compiler stuff is very limited I have a lot of experience with web applications and requirements for code sharing between server and client.

[ylluminate]

Great stuff @theodorton. I was also previously playing with the direct to JS route a la Opal, which I quite like the idea, BUT it really does seem that by piping things through Emscripten we can do this a lot more quickly and with markedly less effort. You might want to pop over here to chime in and chat with us.

[catmando]

I have a longer conversation here: https://gitter.im/crystal_wasm/implementation
The more I think about it, the more I think @theodorton 's approach (Crow?) is correct.
Why? Because at the moment WASM is just not mature enough. I.e. lacks GC, and as a result (and more critically) objects can't be shared with JS land.
This means that even if you can get a full crystal compiler going, you have to rewrite every JS lib in crystal. React for example.

I think if a bunch of us help what @theodorton has done so far, it might be quicker.

I am just playing devils advocate here, and can be convinced that I am wrong of course!

[andriytyurnikov]

It looks like few blockchain projects considering WASM as their primary or secondary VM, so browsers are not only runtime for WASM.

[asterite]

But blockchains are a waste of energy and resources... let's not hurry all together to destroy this planet faster, please?

[andriytyurnikov]

@asterite not those blockchains :)
Blockchains with wasm are environment friendly, and they need your support :)

[faustinoaq]

Q: Don't you need features like garbage collection and threading added to WebAssembly to make this work?

No, WebAssembly in its current state is sufficient. The .NET runtime handles its own garbage collection and threading concerns.

^ Interesting :sweat_smile:

^ https://github.com/aspnet/Blazor/wiki/FAQ#q-dont-you-need-features-like-garbage-collection-and-threading-added-to-webassembly-to-make-this-work

[Svenskunganka]

A design document from the Go core developers are available here: https://docs.google.com/document/d/131vjr4DH6JFnb-blm_uRdaC0_Nv3OUwjEY5qVCxCup4/edit

Looks like their implementation is under review and possibly being merged soon: https://go-review.googlesource.com/c/go/+/102835

Posting it in case anyone wants to take a bite at this in the future. It covers some of the questions already asked here.

[RX14]

And as we can see from that document, go's port to wasm is a bunch of hacks to work around missing features in wasm. If someone wants to make a PR hacking those in, go for it, but until then I'll be waiting for proper GC and coroutine support.

[osarrouy]

Hi everyone,

I'm a big fan of crystal and also happen to be a blockchain developer so i'd just like to know if there is any news about this issue and if i could help in any way ... ?

Thanks!

[rdp]

I guess you could target "javascript" (not wasm) via emscriptem and go single threaded. Though that's not what the title of this issue is about :) I assume that's how mruby in the browser works [?] http://qiezi.me/projects/mruby-web-irb/mruby.html (just uses the javascript built-in GC I presume?). Watch also https://github.com/WebAssembly/gc/ (WIP gc proposal). Other options: Add a "new gc" option ("reference counting with cycle checking" https://github.com/JetBrains/kotlin-native/issues/1170#issuecomment-353328876 or the "double copy stack to the heap" trick RX14 mentions). Or manage stack manually if you want to keep fibers. Doesn't sound trivial...

[sam0x17]

This provides some very useful and up-to-date high-level information on the web assembly feature set and roadmap:

https://hacks.mozilla.org/2018/10/webassemblys-post-mvp-future/

@straight-shoota @RX14

[bew]

Interesting read about threading implementation for wasm: https://rustwasm.github.io/2018/10/24/multithreading-rust-and-wasm.html
There might be some useful things for us ;)

[parruda]

I just stumbled upon this issue. I’ve been thinking about adding Opal support to FaaStRuby, and if we could compile Crystal to Wasm... Full stack Crystal development in a serverless platform. Think about the possibilities!

I wish I could contribute to this effort, but I don’t know enough about Wasm or LLVM. But you guys can be sure I would contribute to make it popular by pushing the support to the platform as soon as you guys release it!

Thanks for the hard work.

[sam0x17]

Yeah, I imagine this is going to be one of the first big pushes after 1.0, and by then other languages will have figured out some of the do's and dont's of web assembly support, so it will be that much easier

[catmando]

FYI: I just put a discussion topic around this general topic: https://forum.crystal-lang.org/t/crystal-js-transpiler/903

[theganyo]

WASM will soon be very important on the server as Envoy is testing support for dynamic filters via WASM and Istio will be fully supporting that soon. That should surely add some heat to this discussion.

[jan-zajic]

LLVM WebAssembly target is production ready since LLVM 8.0.0. See http://releases.llvm.org/8.0.0/docs/ReleaseNotes.html#changes-to-the-webassembly-target .

[jwoertink]

Just keeping the talk going, this article was just released https://www.w3.org/blog/news/archives/8123. I think why this is important is because just like HTML, CSS, and JavaScript, WASM will be a web standard with the W3C backing it. I know with all the stuff this language still needs, this may be a little lower priority, but I think getting a "rough" start merged, that would allow us to really hash out how things should work. We could start with a preview or something like how the MT stuff is going, or just get some initial things like the windows support is doing.

Edit: Here's a spec link https://webassembly.github.io/spec/core/index.html

[RX14]

A "Hello world" for this feature would be 90% of the work :)

Keep your expectations low, this is about as hard as asking for the compiler to compile to JS directly.

[richardanaya]

Any updates on this? A lot of these comments sound like people are hoping for too much to be done at once. Just figure out how to talk with imported and exported functions, don't worry about supporting a standard library, or figuring out magical ways to talk to the dom (libraries can figure this out).

[jwoertink]

I was able to get a "hello" world from crystal done. Though, not in the way you'd want it. I basically just hand wrote bytes 😂 I did come across this gist showing how to compile LLVM with wasm support. I tried to see if I could get crystal compiled this way, but my eyes glossed over. It's a bit beyond my knowledge. Maybe someone else here can point out some stuff? My assumption is that if you can compile crystal with the latest LLVM, and add the right flags, maybe it'll just work™

[jwoertink]

Here's what I did https://gist.github.com/jwoertink/b8fb8430ed819dd4041a56ff6ed3ef64 if anyone is interested in expanding on that, or whatever.