Rust: Tracking issue for supporting asm.js and WebAssembly without Fastcomp

[tlively]

Breaking Rust's dependency on Fastcomp will allow upgrades to Rust's LLVM to be much smoother because they won't depend on Fastcomp being updated. Smoother upgrades will allow LLVM to be kept up to date more easily (https://github.com/rust-lang/rust/issues/42389), which will be beneficial across the board but especially for WebAssembly as its LLVM backend matures. It is necessary that the asmjs and wasm targets emit object files instead of LLVM bitcode so that bitcode version mismatches between Rust and Emscripten won't be a problem. Work that needs to be done to break the dependency on Fastcomp includes:

• [ ] finish wasm2asm and integrate it into Emscripten as a backend for asm.js (https://github.com/WebAssembly/binaryen/issues/1141)
• [ ] make Emscripten take WebAssembly object files as input (blocked on LLD supporting WebAssembly https://groups.google.com/forum/#!topic/llvm-dev/BwFL_ulYX4E, https://reviews.llvm.org/D34851)
• [ ] make wasm32-unknown-emscripten emit WebAssembly object files
• [ ] update asmjs-unknown-emscripten to emit WebAssembly object files and use the future wasm2asm backend in Emscripten
• [ ] remove PNaCl/NaCl support, since it depends on Fastcomp (#42420)

[est31]

@tlively I guess #42420 would also be part of this? If so, can it be added to the list?

[tlively]

@est31 I'm not sure I understand why #42420 is part of this. Does NaCl/PNaCl support depend on Fastcomp as well?

[est31]

@tlively I'm no LLVM expert, but when I look into rust's fork of LLVM, then lib/Target/JSBackend/NaCl/ is the place I can find the NaCl backend while upstream LLVM seems to lack the JSBackend subdir entirely. I also can't find any meaningful references to NaCl in upstream LLVM so it wasn't moved or something. It looks like an addition by fastcomp.

[dylanmckay]

Can confirm - the entire JavaScript backend lives inlib/Target/JSBackend/.

Every JS-specific modification to LLVM outside of this directory is decorated like so

@LOCALMOD-BEGIN
< .. js specific code .. >
@LOCALMOD-END

It looks like an addition by fastcomp.

It is

[est31]

cc #45905

[pepyakin]

(continuing discussion from https://github.com/rust-lang/rust/pull/45905)

@RReverser

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

But what should do Web version then? Should it implement fs, net, env, etc emulation in JS? Or should it trap? If we chose to implement emulation then we will need a JS shim library, which will call "real" APIs or emulated ones depending on which "sub-environment" we running on. But then, what if user wants to run only one "sub-environment"? Or user doesn't want either fs or net, or nothing at all?
Emscripten solves this problem with preprocessing of JS files. Should we do our own preprocessing?
But should we? Given that there is emscripten which do the job already?

Applications of wasm should be truly beyond the web. Fast, safe and deterministic execution seems to be pretty desired properties!

• IoT,
• plugins, scripts, and other embeddings in larger programs,
• deterministic execution, especially for p2p (i.e. blockchain applications),
• universal drivers,
• mobile apps,
• desktop and server apps,

Some of environments can implement complete std. However, others can't.
These features can or cannot be supported:

• processes,
• filesystems,
• TCP/UDP,
• environment variables,
• stdin, stdout
• etc...

Even not all WASM features are always desired (due to performance reasons and/or need of deterministic execution):

• threads 🦄,
• gc 🦄,
• simd 🦄,
• floating points,
• growing memory,

I think, this makes wasm to be more like ISA than end user platform. (Not to mention that we even have problems of combining JS/Web and JS/Node).

To properly handle all this zoo, I think, we need something like portability lint and maybe a few more triples, like this strawman:

• wasm32-unknown-unknown - generic target that assumes nothing about it's environment. Suitable for the web.
• wasm32-unknown-node - target specially for node. I think it might support all of the std?

Well other environments might provide own implementations of these system bindings if they need to.

It will make portability lint useless, isn't it?

[badboy]

I still think having wasm32-unknown-unknown providing the bare minimum (with a libstd where it makes sense) will get us a long way.

But what should do Web version then? Should it implement fs, net, env, etc emulation in JS? Or should it trap?

It should implement shims for these things.

If we chose to implement emulation then we will need a JS shim library, which will call "real" APIs or emulated ones depending on which "sub-environment" we running on.

Yes, we will need this shim. That's the same as it works in Emscripten today.

But then, what if user wants to run only one "sub-environment"?

If it is possible to modularize these different things, it would be up to the user to load the necessary things before loading the wasm module (again: I did not look into Emscripten's shims yet and how easy it would be to extract those things).
This does not even have to be part of Rust itself btw.
But I also don't expect every module out there needing or wanting to have this environment provided (JavaScript<->Wasm interaction is still the slower thing in the whole execution).

(I'm gonna read that portability lint RFC now)

[pepyakin]

I still think having wasm32-unknown-unknown providing the bare minimum (with a libstd where it makes sense) will get us a long way.

I agree on this too. But it seems we disagree on what "bare minimum" means : ) For me, "bare minimum" doesn't include fs, net, processes, rng, etc.

That's the same as it works in Emscripten today.

This does not even have to be part of Rust itself btw.

OK, here is my direct question then: why don't use Emscripten if one needs full-blown std library with support of Web, Node? 😃

[steveklabnik]

IMO, wasm32-unknown-unknown should have no shims. wasm32-unknown-web would be something more like emscripten, with shims for a bigger chunk of functionality.

non-web applications of wasm won't want the shims, and code that doesn't use the shims won't want the shims. Keep the minimum small, build from there, IMO.

[SimonSapin]

Rather than wasm32-unknown-unknown having an std crate with most of the APIs returning an "unsupported" error, could it not have std at all? If there’s functionality in std that is supported in "pure" wasm, isn’t that a sign that this functionality belongs in another crate like libcore or liballoc?

[pepyakin]

@SimonSapin yeah, maybe.

One thing that comes to my mind is threads.

WASM threads propolsal (it's still WIP though) suggests adding atomic.wake & atomic.wait operations, which roughly corresponds to thread::park/Condvar.

[pepyakin]

There is one more point: things that only could be implemented in terms of libcore AND liballoc automatically goes into libstd. For example, io::Read and io::Write.
I can imagine usage of them in pure wasm context.

Also, if I squint enough, I can imagine implementation of HashMap outside of std.

[SimonSapin]

There is one more point: things that only could be implemented in terms of libcore AND liballoc automatically goes into libstd. For example, io::Read and io::Write.

liballoc depends on libcore, so I don’t think depending on both is a sufficient reason to have things in libstd. In the case of io::Read and io::Write it’s probably rather because they depend on io::Error, which depends on std::sys for OS errors.

As to HashMap, I believe the reason it’s not in liballoc is that its default hasher requires a pseudo-random number generator to be seeded from the OS.

[NikVolf]

HashMap being hasher agnostic and residing in liballoc would be helpful btw

[pepyakin]

liballoc depends on libcore, so I don’t think depending on both is a sufficient reason to have things in libstd. In the case of io::Read and io::Write it’s probably rather because they depend on io::Error, which depends on std::sys for OS errors.

Oh, I see. That's unfortunate.

As to HashMap, I believe the reason it’s not in liballoc is that its default hasher requires a pseudo-random number generator to be seeded from the OS.

Agree with @NikVolf on that

[SimonSapin]

I don’t know if a type item can specify a default type parameter (in the case of HashMap, S = RandomState) while still allowing it to be overridden.

[RReverser]

@simonsapin I imagine something like pub type HashMap<K, V, S = RandomState> = core::HashMap<K, V, S> in std would work (where core variant doesn't have any default).

[SimonSapin]

It would be alloc rather than core, but yeah that might work. Who feels like submitting a PR? :)

[RReverser]

@pepyakin

OK, here is my direct question then: why don't use Emscripten if one needs full-blown std library with support of Web, Node? 😃

For me, the reason not to use Emscripten is to avoid duplication and complexity in the building toolchain and not because of the libraries. The opposite, having emulation of native APIs that works on Node.js/Web has always been the most exciting part of Emscripten to me as it allows to write fully native apps and libraries and run them on different environments without any hassle.

[pepyakin]

For me, the reason not to use Emacripten is to avoid duplication and complexity in the building toolchain and not because of the libraries.

Can you elaborate on the duplication and complexity part?

[RReverser]

Can you elaborate on the duplication and complexity part?

It's about presence of LLVM in both Rust and Emscripten when you could just use Rust directly (and LLVM versions of both currently need to be kept compatible when you use Rust's -emscripten- targets).

Also, Emscripten evolves much slower and harder to contribute to in my experience, so when you need to implement something Rust-specific for the WASM/JS output, it might take much longer to try and implement it on Emscripten side than if we had everything done only on Rust side.

[cretz]

I'll drop my two cents here as an author of a non-web WASM backend: Emscripten and the idea that JS emul needs to be Rust's concern is off. The WASM community needs to get together and create a libc-esque set of abstractions that backends can support (like emscripten or mine). In the meantime, I'd really like to not see low-level impls for a specific backend (JS/Web) be part of the Rust lang. That the shim exists already is enough. If you want JS support, do it elsewhere external to the lang.

[cretz]

@chpio - Meh, the abstraction doesn't really matter. It can just be nix syscalls like emscripten does for all I care. Just WASM import what you expect and let the backend fill em in, but of course a consistent abstraction would be nice. But the real nice thing would be if all WASM frontends could define something they share, but that seems unlikely (Emscripten basically uses libc and other libs to assume this).

I just tossed my opinion in here that it should not be part of the Rust-lang repo (or even the core devs' concern) to implement JS forms of the stdlib. I haven't yet played w/ the unknown-unknown compilation target yet, but one way would be to have a separate wasm file emitted of all the abstracted functions and just have unreachable as the one and only command for all functions. Then just import that into the real wasm file...it can be supplanted by backend authors as need be.

[RReverser]

to implement JS forms of the stdlib

Yeah, obviously either way Rust won't know implementation details of JS side, so for what I care, it can be a separate npm package. What's important here is to have calls out, and they will be needed and same no matter what target you use (whether it's JS or non-JS).

[chpio]

https://www.tockos.org/blog/2017/crates-are-not-safe/#usage-of-the-standard-library-is-pervasive

Any dependency we would want to use for Tock needs to include #![no_std] so that the compiler does not try to include the standard library. Again we surveyed all of the crates on crates.io and found that 93% (11488/12360) of crates use the standard library (i.e. do not have #![no_std]). [...] However, when including required dependencies, the number increases to 97% (12023/12360).

[alexcrichton]

Hey all! Exciting to see all the current discussion on this! I figured I could help throw in my viewpoints as well.

Future of wasm32-unknown-unknown

Overall I'm quite happy with how this target is shaping up. You've got access to big chunks of the standard library (aka libcore + liballoc) and I think that with the upcoming proposals we'll be able to just start filling out the implementation of types like Condvar for example (as well as Mutex) using the various wasm instructions.

The biggest blocker I know of today to "big usage" is the fact that the compile time is dog slow and you get a bunch of LLVM asserts if you don't compile with optimizations. The "dog slow" compile times are because we force LTO on everyone (wheee!) and will get fixed with lld. The WebAssembly support of lld has been merged upstream into lld itself, but unfortuantely we don't get too far in using it today. Once we solve this problem though I think we'll be in great shape for the target itself.

More broadly I also feel somewhat uncomfortable precisely how the wasm target interacts with the outside world. For example extern { fn foo(); } is required to come from the env module, but presumably the env there should be configurable? There's various other bits and pieces too, but I'd in general recommend that anyone looking to use this right now is at least aware of these issues and how the target may get tweaked in the future wrt these precise integration/exposure details.

Future of a wasm libstd

I'm not particularly happy about all the errors returned from src/libstd/sys/wasm/*.rs. I think modules like std::{thread, fs, net} just shouldn't exist on the wasm target. Unfortunately though the primary enabler for this, the portability lint, is not current implemented. Additionally I'm not sure libstd is ready for such an organization (slicing and dicing) yet as well, but we can probably discuss more with the portability lint. In the meantime I think that "return errors everywhere" is the best solution we have today.

I'm also not particularly happy about modules like this one. That and how 1.0 % b may get lowered to fmod by LLVM. These sorts of functionality silently may require imports when instantiating a wasm module which is pretty unfortunate.

In general we, in the standard library, have no means of importing something that you as the instantiator don't have to worry about. For example when you say use std::thread you need not concern yourself that it imports a bunch of stuff. Similarly if we decided to implement f64::tan as a call to Math.tan in JS you ideally wouldn't have to worry yourself about that either!

Unfortunately though I don't think we have many options available to us. It sounds like one day wasm may be a full-fledged JS module citizen, but until then I think we need to keep our dependencies as slim as can be. That way users who instantiate a wasm module will need to provide as little as possible to instantiate it.

Wasm + node/web

I'd personally think that we can't really do anything here right now. Basically all due to what I mentioned above of how we can't import anything without actually forcing you to define it. That is, even if we knew we were executing in node, it's not like we have any new superpowers we could access!

Perhaps eventually when we can import functionality directly I do think two targets could make sense. For example a node target could make things like println! actually work by default (man that would be nice).

Wasm pipelines

One possible solution to this import problem may be some sort of standard pipeline though. I think that the standard pipeline for the near future will need to include wasm-gc if you care about binary size (as it strips compiler-rt goo that you don't need). I also think that wasm-opt from the Binaryen toolkit will also want to be a member of the standard pipeline as it can shrink code size even further after wasm-gc.

I wonder if maybe there's a pipeline or something like that for dealing with imports/exports? Like what if we could automatically hook up Math.random into HashMap::new() without you having to do anything? It'd be neat but I'm also not entirely sure that it'd be 100% possible unfortunately.

I'm curious if others have thoughts though!

[alexcrichton]

Oh right I should also say that I don't see the Emscripten targets going away any time soon. They're still incredibly useful if you're porting existing applications as you really do want all those emulation layers. Maybe like in a decade we can remove them if no one is using Emscripten, but for now I think Emscripten still has a solid enough niche that we should continue to support it as we do today.

[jonhere]

2 cents.
Locked on env isn't a big deal IMO, just polish. it just means you can't as easily bulk import something say Math. old link (semi) on the issue. Being able to interact with webasembly table would give a small speed boost with dynamically assigned functions.

Threads will come in time, (hoping not too much.) My thought (not read anywhere) is only the main instance thread will be able to call imports.

I love that wasm32-unknow-unknown is minimal. In terms of getting println! and other libstd items: maybe a solution is expose a back-end Trait that programmers could supply their own structure to it, (copy default one / include crate.)

Debugging is a pain point. "Runtime Error: unreachable executed" this message should ideally be better.
Stack trace: wasm-function[55]@ read this no idea if firefox would use it but can see no mention of debug in the wast.

[RReverser]

Like what if we could automatically hook up Math.random into HashMap::new() without you having to do anything? It'd be neat but I'm also not entirely sure that it'd be 100% possible unfortunately.

@alexcrichton Why not emit own JS in addition to just .wasm that would include needed imports? (similarly to Emscripten)

Then pure-wasm users will just provide own imports, but at least all the Web users of WebAssembly will be able to instantiate with correct JS counterparts without hassle.

[alexcrichton]

@jonhere compiling with debuginfo (-g) I think may help at least show more functions in the stack trace, but I agree that the debugging story isn't great.

@RReverser initially the new target was intended to be as bare as possible (no extra js fluff) but I think we'll go down the road at some point of emitting js glue which would give us a lot more flexibility of what to do and how to implement it. I'm just wary to go down that path too quickly!

[rpjohnst]

I would prefer wasm32-unknown-unknown to remain as bare-bones and freestanding as possible, even if that means not supporting the stdlib out of the box. Having a target like that is important, similar to the targets people use for kernel dev. (I suppose it would be sufficient to let people make their own target.json for that case, though.)

If we start emitting JS glue, it should probably be part of a separate target. wasm32-unknown-web, or something. That target could subset the stdlib (via the portability lint) to only what the web platform provides without any emscripten-like emulation. It would be good to keep the JS glue as minimal as possible here, with an eye toward being able to remove it if (when?) wasm allows direct access to the web platform without imports.

Concepts like "the system allocator" muddy the waters here somewhat, though- the web platform probably isn't ever going to provide that, so making a statically-linked one like dlmalloc-rs the default allocator for non-#[no_std] use of wasm32-unknown-unknown might be reasonable.

[RReverser]

I would prefer wasm32-unknown-unknown to remain as bare-bones and freestanding as possible, even if that means not supporting the stdlib out of the box.

What's the difference for users who don't care about stdlib? Right now, with "bare bones" approach it will emit errors using panic!s in Rust code, with implicit imports it will still emit errors, but only due to missing JS counterparts. At the same time, for Web users it will just work as soon as you import JS glue. I see it as win-win for both sides.

[rpjohnst]

The idea is to support code that doesn't have any implicit imports, without having to strip out a bunch of stuff. With the portability lint and/or no stdlib, this is totally doable and it's an important use case.

[RReverser]

sigh I really hoped that with this new target Rust would finally have a hassle-free support for WebAssembly on Web / Node.js - something Emscripten could never do due to really slow turnarounds and hard integration with JS, but it starts sounding like most people are against it and transparent integration of Rust + JS is still not happening 😢

[SimonSapin]

I don’t think anyone has argued that such integration should not exist at all, only that it should somehow not be mandatory. Maybe that means having two targets.

[rpjohnst]

@RReverser I am not arguing that at all! That support should exist and will certainly build on the work done for wasm32-unknown-unknown. It's just that the -unknown triple is a bad place to put platform-specific glue (i.e. web/node.js). wasm32-unknown-node and/or wasm32-unknown-web would be better places for that.

[shepmaster]

In the bikesheddy realm, I'd vote for wasm32-rust-node / wasm32-rust-web for such a target. 😄

[est31]

As stdweb has gained support for the unknown target, I think we now can talk about the concrete proposal of using components from it from inside libstd. I've opened a thread on irlo.

[jontro]

But is really Math.* platform specific? I thought it was defined in the ES standards. As long as it's not specific to a specific target it should be fine for the unknown target?

[Pauan]

@jontro Wasm != JavaScript.

Wasm is specifically designed to work without JavaScript, so if you're compiling to wasm you can't assume that JavaScript APIs exist (because they might not exist!).

So, you can't use JavaScript APIs (including Math) in wasm32-unknown-unknown, but it's okay to use JavaScript APIs in wasm32-unknown-node and wasm32-unknown-web

[est31]

What @Pauan said is also noted on the webassembly website:

Non-Web environments may include JavaScript VMs (e.g. node.js), however WebAssembly is also being designed to be capable of being executed without a JavaScript VM present.

So we should keep a target that doesn't rely on any js at all. wasm32-unknown-unknown seems like a good fit.

[jonhere]

(Not arguing about not keeping it minimal or anything.)

So, you can't use JavaScript APIs

You don't directly use them at the moment, so there is nothing to stop the target having more. WebAssembly defines the API for imports and exports. When you write your rust function args (bool, *const f64) the compiler converts them to WebAssemblies 4 data types. Similar happens with the javascript engine; (the bool (,etc.) rust sends becomes Number; you can send javascript Boolean to bool the other way.)

[Pauan]

@jonhere WebAssembly is not always running in a JavaScript environment. WebAssembly has been specifically and intentionally designed so that it is completely separate and unrelated from JavaScript.

There are already implementations for running WebAssembly in C, running WebAssembly on the JVM, and even running WebAssembly on your washing machine.

Obviously none of those implementations have the JavaScript APIs available. The JavaScript APIs simply do not exist at runtime at all, so no matter what Rust does, you cannot use the JavaScript APIs, period. You will get a runtime panic (at best).

[RReverser]

@Pauan But it really doesn't matter whether API bindings on the other side are provided by JS, C, Java or anything else. All that matters is they are imported on the Rust side (since there are already some required by LLVM anyway), and all these implementations are still free to provide own native bindings on the other side without taking any interop away from WASM.

[rpjohnst]

Again, since native wasm doesn't provide Math.*, -unknown targets should not use it. This is already how things work with other -unknown targets, both in Rust and in C.

If you want to add an import yourself and provide a Math.*-like function from Javascript, C, or anything else, that's fine, nobody's stopping you. But the compiler adding those imports automatically is a non-starter.

Edit: somewhat of an exception is "builtin" functions in C- compilers will generate calls to things like memcpy without you writing them in the source, even in freestanding mode on an -unknown target. However, this can be opted out of (-fno-builtin on GCC), and those functions can be provided from within C anyway.

[CryZe]

Yes, it certainly shouldn‘t be JavaScript APIs that it expects, however the std should instead instead define symbols it requires for all of the relevant APIs (Instant, stdin, stdout, ...). These symbols can then be provided by whatever is executing the WebAssembly and a JavaScript binding that automatically provides these symbols could be auto generated by the compiler.

[Pauan]

@RReverser If there is an API that can be provided by multiple targets, then the API can use conditional compilation so that it will work on multiple targets.

But even in that situation, it still won't work for the wasm32-unknown-unknown target, because the wasm32-unknown-unknown target is supposed to contain no platform-specific code.

When you compile your code with the wasm32-unknown-unknown target, you are saying "I want this code to work on all WebAssembly implementations". Therefore the only code that you can use is native wasm code, no platform-specific code.

Of course if you use a different target, such as wasm32-unknown-web, then you can use non-wasm bindings (such as JavaScript bindings), because your specified intention is different. In that case of course the compiler should automatically use the JavaScript APIs, so everything works smoothly and seamlessly.

@CryZe That's probably a good idea! That still won't apply to wasm32-unknown-unknown, but it would be useful for the other wasm32-* targets.

[CryZe]

What‘s the benefit of a largely broken std? The std should define a proper interface it communicates with. Only the subset of this interface that you actually use should be compiled in, so you have all the benefits of having std, while not getting random panics at runtime. If you truly never want to interact with an „OS“, then why even use std in the first place? That‘s what core is for.

[RReverser]

target is supposed to contain no platform-specific code

Problem is, it's not a platform-specific code, as you saw in the examples, such bindings are required even for some basic math operations. And from there it's not far to go to implementing std for those who compile against std as @CryZe says.

then you can use non-wasm bindings (such as JavaScript bindings)

As I said, they don't have to be JS bindings, they can be Java bindings, C bindings or anything else, so -web suffix doesn't make any sense as they're not Web- or generally platform-specific bindings in the first place. They're just basic "kernel functions" that Rust WASM would depend upon.

the std should instead instead define symbols it requires for all of the relevant APIs

This is a good idea IMO.

[CryZe]

Thinking about it, since this "kernel interface" is supposed to be somewhat generic, maybe it even makes sense for Rust to have this outside of the wasm target, as a flexible way to easily hook onto any kind of OS / runtime.

[rpjohnst]

A well-defined (even stable!) set of imports required by std-on-wasm is a fantastic idea, and would form an ideal basis for the -web/-node targets.

It could even be used on the -unknown target, if it were purely opt-in. Again, when you target -unknown, that should mean zero imports aside from the ones your program declares. This scenario would be much like using liballoc and libcollections from #[no_std]- you start with nothing, and intentionally add pieces from there.

Starting from -unknown but having to rip out more stuff to get to zero imports is a no-go.

What's the benefit of a largely broken std?

@CryZe There's a lot of benefit in a subsetted std, which is why https://github.com/rust-lang/rfcs/pull/1868 was merged. Targets that simply don't support something will be able to drop it at compile time, without dropping you all the way down to core.

target is supposed to contain no platform-specific code

Problem is, it's not a platform-specific code, as you saw in the examples, such bindings are required even for some basic math operations.

@RReverser That's completely normal for an -unknown-style target. Many embedded systems lack the same "basic math operations" and since WebAssembly (currently) does as well, this must be represented in its -unknown target.

And from there it's not far to go to implementing std for those who compile against std as @CryZe says.

It is very far to go from "basic math operations" to "all of std," and much of that distance requires emulation on the web! Honestly even the -web target shouldn't support everything, it should support only the subset of std that the web platform does.

[rpjohnst]

To be more constructive, I imagine the targets looking something like this:

-unknown: Zero imports by default. Doesn't support the math ops WebAssembly lacks natively, doesn't support anything in std that would require support from outside WebAssembly. When a program declares the proper imports (similar to defining an allocator), it can start to use more operations.

-web: Imports only for things directly supported by the Web Platform. Thus can't support much of std- standard output goes directly to console.log, no File::open, no TcpStream, no std::process, and only the parts of std::thread that directly correspond to Web Workers (not familiar enough to say if such a subset even exists). Anything else (WebSockets, etc) will have to be provided by other crates.

-node: Looks a lot more like a normal desktop target, since Node actually provides those APIs. (Though since Node is event-loop based you may not actually want to use those APIs much.)

Something like -emscripten which emulates all of std on the web platform is certainly useful. But it needs to remain a separate target so that people don't have to rip out its emulation layer when they just want to write to, say, the web platform, directly. If we ever wanted to provide something like this separate from Emscripten itself we could call it... wasm32-rust-stdweb?

[RReverser]

@rpjohnst Separating web and node would be very unusual, as JS libraries are expected to work universally across different environment without any change (this is why most of Node.js libraries have their own browserified counterparts, even if they don't support all the features, just to avoid porting same code across environments but rather bundle and use out of the box).

Either way, Rust doesn't have to know about these JS implementation details, just refer to generic kernel functions as said above, so having -web or -node or any other prefix in Rust side doesn't make any sense since it won't actually provide these implementations, just expect certain C-like bindings provided by whatever is running the generated WebAssembly code.

[steveklabnik]

So, just to make sure everyone's on the same page here, WebAssembly's spec defines no platform details. Platform details are expected to be layered on top. This is specifically so that wasm can be used outside of a web context, and in fact, not only is that an important use-case overall, but an important use-case for production Rust users: Etherium plans on using wasm as their scripting language, for example.

As such, I'd very very much prefer that the unknown target implements https://webassembly.github.io/spec/ and nothing else. We should define a web target for dealing with web platform integration.

[CryZe]

The thing we are arguing is that there shouldn‘t be a -web. Maybe -std or something that has a proper std implementation, which I‘m totally okay with, if there is actually is a good reason for the target with the stubbed out std. However the fact that people are arguing for a target with a stubbed out std makes it seem that there‘s actually rather something wrong with core, which people really try to avoid.

[RReverser]

To back @CryZe's words: all the bindings we are discussing are not any specific to Web, Node.js or any other host. We don't expect nor want Rust to do any web platform integration - that can be done in custom crates and npm packages. These bindings that we need are very generic and are needed by Rust just for proper functioning of std crate on any platform, but actual implementation on the other side can be provided by absolutely any code, VM or platform that hosts WebAssembly.

This is completely orthogonal to portability lints, which might or might be not used in conjunction with such generic "kernel" bindings.

[chpio]

However the fact that people are arguing for a target with a stubbed out std makes it seem that there‘s actually rather something wrong with core, which people really try to avoid.

There is a problem with core, no one supports it.

But i like the idea of portability lints. We could even make libcore and liballoc obsolete by providing a cfg for alloc and make std and alloc switchable from the main crate.

[rpjohnst]

Separating web and node would be very unusual, as JS libraries are expected to work universally across different environment without any change (this is why most of Node.js libraries have their own browserified counterparts, even if they don't support all the features, just to avoid porting same code across environments but rather bundle and use out of the box).

@RReverser The fact of the matter is, Node supports a lot of stuff that the web does not and cannot without emulation. This means that there is a place for a -web target with a subsetted std that excludes that functionality (e.g. File::open, TcpStream, std::process) and thus does not generate imports for it.

people are arguing for a target with a stubbed out std

@CryZe To be clear, I am not arguing for a stubbed out std. (I'm not sure anyone else is either??) I'm arguing for a subsetted std that includes only what the web platform supports. That way there are three tiers to using Rust in a browser-hosted wasm:

• -unknown: You control all the imports and go without the parts of std that require any kind of support from the host (unless you explicitly add the imports and glue yourself).
• -web: The compiler generates imports and glue code for non-web-specific stuff that std needs (like @RReverser describes in https://github.com/rust-lang/rust/issues/44006#issuecomment-353447483), but only for things that can be provided by the web platform (e.g. Math.*).
• -emscripten/-stdweb: The compiler generates imports for everything std needs (the full "generic kernel bindings") and provides the necessary glue code and emulation on the Javascript side. The emulation need not be maintained by the Rust team, it would be considered part of the platform.

The -node target would follow the same principle- generate imports and glue code for everything std uses that can be provided by Node. I suspect this would be everything in std, so there would be no need for an -emscripten-like target for Node?

[RReverser]

This means that there is a place for a -web target with a subsetted std that excludes that functionality (e.g. File::open, TcpStream, std::process) and thus does not generate imports for it.

Far not necessary. There are cases when TcpStreams are implemented in browser with a WebSocket proxy (this actually works and is used in e.g. x86 emulators in browser), File::open can be implemented with File API or point to local storage and so on. It's not the job of Rust to decide what to expect on the other side from a dynamic language and how these APIs are implemented, this is up to the consumer of compiled WASM.

-web

but only for things that can be provided by the web platform (e.g. Math.*)

-emscripten/-stdweb

To reiterate points from previous comments - why would this be separated into -web or similar targets if these are not any specific to Web platform and same math bindings can be universally provided by other consumers like JVM or C?

[est31]

There are cases when TcpStreams are implemented in browser with a WebSocket proxy (this actually works and is used in e.g. x86 emulators in browser), File::open can be implemented with File API or point to local storage and so on.

APIs for getting the date or something do need javascript right now, and there is only one way of implementing them. If you want to be free in the choice of the file system, put it into your program! Why should rust play the abstraction layer here? Your library code should work on anything that impls the Read/Write trait, and it can be a cursor.

To reiterate points from previous comments - why would this be separated into -web or similar targets if these are not any specific to Web platform and same math bindings can be universally provided by other consumers like JVM or C?

You also have a separation between windows, mac and linux targets. There is wine, people can just use the windows API everywhere and yet they everyone wants to compile stuff natively. The reason is that then everything is tailored for your specific target.

[RReverser]

There is wine, people can just use the windows API everywhere and yet they everyone wants to compile stuff natively. The reason is that then everything is tailored for your specific target.

The reason for this is simply that you can't expect your consumer to have Wine. Better example would be Linux GNU targets for which Rust compiles just once with -linux-gnu and this works without complaints for pretty much everyone.

We just suggest to do with WebAssembly - list expected bindings for std (for what I care, most of them can be even regular POSIX function declarations) and don't put burden of knowing their implementation details neither on Rust compiler nor Rust developers.

JavaScript libraries have been moving strictly towards being universal (or also, so-called "isomorphic") over the last years, and it would be a huge step back for WebAssembly target to separate -web, -node or any other subtarget when it was intentionally designed to be universal in the first place.

[RReverser]

Why should rust play the abstraction layer here?

@est31 Because why not? Would you prefer to have duplicates of each crate that work on different targets, or a single crate that just uses std APIs and works in any environments (just like majority of Rust code nowadays)? What would you gain from knowing how e.g. Math.fmul is implemented on the other side of whatever is consuming your WebAssembly code?

[rpjohnst]

There are cases when TcpStreams are implemented in browser with a WebSocket proxy (this actually works and is used in e.g. x86 emulators in browser), File::open can be implemented with File API or point to local storage and so on. It's not the job of Rust to decide what to expect on the other side from a dynamic language and how these APIs are implemented, this is up to the consumer of compiled WASM.

This doesn't make any sense. Nobody is opposed to providing a target that includes or supports TcpStream emulation via WebSocket proxy. We already have one, and that's great! What I want, in addition, is a -web target that doesn't provide TcpStream because the web platform does not provide TCP.

Would you prefer to have duplicates of each crate that work on different targets, or a single crate that just uses std APIs and works in any environments?

You seem confused. We already cross-platform crates via -emscripten, and we will continue to have them for -web and -node for crates that happen not to use any parts of std that don't need emulation. The point of those targets is not to split the ecosystem, but to give people a way to generate wasm modules that only need a thin layer of glue without any emulation. See, for example, the lack of an ecosystem split between x86_64-pc-windows-msvc and x86_64-unknown-linux-gnu.

[est31]

Would you prefer to have duplicates of each crate that work on different targets, or a single crate that just uses std APIs and works in any environments (just like majority of Rust code nowadays)?

I would absolutely hate duplicates. But that won't happen when your library exposes a functionality to work on a Read/Seek/Write trait impl and doesnt just eat a path and serialize it on itself.

JavaScript libraries have been moving strictly towards being universal (or also, so-called "isomorphic") over the last years, and it would be a huge step back for WebAssembly target to separate -web, -node or any other subtarget when it was intentionally designed to be universal in the first place.

Again, C/JVM based targets might not get in touch with javascript at all, and some use cases certainly will not. Don't put the API at the Js layer! About unification, sure it's great, and I think most code will work both on a -node and a -web target, but then there are differences between browsers and for those you should be able to differentiate at compile time, just like you are able to differentiate today between i586-pc-windows-gnu and i686-pc-windows-gnu targets.

[RReverser]

We already have one, and that's great! What I want, in addition, is a -web target that doesn't provide TcpStream because the web platform does not provide TCP.

but then there are differences between browsers and for those you should be able to differentiate at compile time

Again, the main question: why would you want it? What does it give you? You'll get exactly same error for unsupported stuff right away when you try to import such WebAssembly with missing bindings on your platform, except you'll have a choice to stub it on your side if it's already a dependency of consumed library. This way it will be possible to distribute same WebAssembly library for all supported platforms, and consumers will just need to import it with their platform's bindings, without recompiling Rust code.

We already cross-platform crates via -emscripten,

-emscripten is very different and provides internal JS bindings for everything. I thought we're all talking here about a more universal target for WebAssembly than just JS?

Again, C/JVM based targets might not get in touch with javascript at all, and some use cases certainly will not.

Of course, that's the goal we want to enable. None of the suggested bindings are JS-specific, and as I said dozen of times above, none of them have to be - they're all just regular "kernel functions" universal to any platform.

[CryZe]

This all seems to come down to whether WebAssembly should be portable enough to just stick the same WebAssembly module into Java / Web / C / ... or if you should always recompile it. If it should be that portable, then we need one defined kernel interface that the module talks to. If not, then we should go for multiple separate targets, but completely miss out on the wasm module being portable at all then.

[RReverser]

This all seems to come down to whether WebAssembly should be portable enough to just stick the same WebAssembly module into Java / Web / C / ... or if you should always recompile it.

Yup, I think you summarised the two camps in this thread pretty well. I'm in the portable one - after all, it was one of primary goals of WebAssembly to abstract away from specific operating systems, virtual machines and so on, and provide a single binary format executable on any of them right away. Removing this abstraction and recompiling for each target seems unnecessary and makes WASM barely more useful than some C code or LLVM IR.

[est31]

@CryZe the portability that wasm is about is for me at least about being portable across operating systems and hardware, not across different embeddings. Embeddings change rarely and if they do, you can recompile stuff. I mean you can disagree with me on the importance of being portable across embeddings, but "completely miss out" is quite overblown in any case.

[rpjohnst]

You'll get exactly same error for unsupported stuff right away when you try to import such WebAssembly with missing bindings on your platform, except you'll have a choice to stub it on your side if it's already a dependency of consumed library.

@RReverser

You can get that functionality from the -unknown target just by selectively enabling stuff! The difference is that you have to opt into it in the leaf crate.

There should be a way to compile a wasm module+js glue that you can just throw into a browser and have it work. Thus, -web and -emscripten. That way you don't need to do any extra work to build a web application.

This all seems to come down to whether WebAssembly should be portable enough to just stick the same WebAssembly module into Java / Web / C / ... or if you should always recompile it. If it should be that portable, then we need one defined kernel interface that the module talks to. If not, then we should go for multiple separate targets, but completely miss out on the wasm module being portable at all then.

@CryZe

This is an inaccurate description of my argument. We can have both! We should define a standard interface that std requires. We should also allow people to target specific embeddings and generate the glue code for them.

[rpjohnst]

Perhaps an additional target or flag would be helpful- a way to get the -emscripten/-stdweb behavior in that it generates all the imports, but doesn't produce the glue code.

[CryZe]

Oh, then we must've been talking past each other, cause I totally agree with that. I think the standard interface is necessary for supporting all kinds of random embeds that rust has no specific target for, but specific targets could definitely have more specific bindings and APIs, like a js! macro and maybe some DOM APIs.

[est31]

So.... we all agree on a js macro then? I'm writing an RFC to add one to the language. I'll link it here.

[RReverser]

I'd definitely like to these to exist, but I don't really see why custom macro, DOM APIs etc. can't live in custom crates and npm packages just like any other userland code, especially integration-specific one.

[rpjohnst]

They can and should, along with targets to organize #[cfg] and compiler support.

[Diggsey]

So these extra targets would be nice, but for the current wasm32-unknown-unknown target we should just export function pointers for:

• writing to stdout
• writing to stderr
• setting an exit code
• reading from stdin?

Regardless of how the webassembly is being hosted, the host can choose to set these function pointers or leave them unset. If they are left unset then the behaviour can be as it is today. If they are set, then libstd calls them to handle each case.

This solves the current blocker for me, which is that I can run wasm tests in the browser, but I can't tell if they pass or fail... (libstd currently throws away stdout, stderr and the exit code).

[est31]

@RReverser We have multiple different implementations of such a macro. If you use one implementation, you are required to use that implementation's tool after the fact. If it is in the compiler, it would serve as minimal standard. Also, it would be usable by std. std can't depend on other crates, so having such a macro would be a requirement for targets with tailored std APIs.

About the fully dynamic target idea, I think I'm convinced now that having one would be useful. wasm32-unknown-unknown can serve as such. I'm not running out of RFC ideas...

[RReverser]

you are required to use that implementation's tool after the fact

You are probably right, for js! macro it makes sense as it essentially requires own linking step (at least for release builds, for debug builds it could just dynamically create new Function on the JS side with string's pointer as a cache index; then you could import a single FFI bindings to make macro work).

For DOM and such though, I think it makes sense to put them into own crates / packages, they don't require much from compiler or std internals and can build on top of js!.

[est31]

For DOM and such though, I think it makes sense to put them into own crates / packages, they don't require much from compiler or std internals and can build on top of js!.

Definitely, this belongs into crates. std shouldn't have any additional features than it provides on other platforms as well.

[est31]

@Diggsey made a PR implementing their idea. cc'ing everyone subscribed: https://github.com/rust-lang/rust/pull/47102

[alexcrichton]

We've had a ton of progress since this was opened and I'm going to close this now as we're now quite decoupled from the Emscripten backend on master and wasm32-unknown-unknown has started picking up a lot of these elements.

In general we're going to follow Emscripten master, but this is mostly a bug for Emscripten rather than rust-lang/rust at this point, so I'm going to close.