Wasm-bindgen: WebIDL binding generator

[alexcrichton]

The #[wasm_bindgen] attribute is pretty feature-ful right now and notably allows importing a class from JS and using it in Rust. Unfortunately though it's a pretty manual process to write this out for web apis, so it'd be great to do this automatically!

Thankfully there's this awesome thing called WebIDL which is a programmatic description of APIs available on the web. There's even two parsers on crates.io for WebIDL!

I think it'd be pretty neat if a WebIDL generator were added to this repo to automatically generate #[wasm_bindgen] decorated bindings. In that sense I'd imagine that we could auto-generate a bunch of *-sys crates which provide bindings for all the web-related functionality JS has to offer. At that point working with the DOM or other web APIs should be as simple as extern crate foo!

An issue like this certainly has a lot of design questions to explore as I'm sure WebIDL is far richer than what #[wasm_bindgen] supports today. We'll need to add features to #[wasm_bindgen] along the way as well as probably developing idioms to map JS to Rust, but I think it'd be great to at least start out with some simple APIs to see how it goes.

For example the README has an example:

#[wasm_bindgen]
extern {
    fn alert(s: &str);
}

but it'd be awesome if we could automatically generate this binding from a WebIDL file and place it in a crate to use. Once we have the "hello world" variants working we should hopefully have enough information to inform the next phase of design questions.

[alexcrichton]

cc @spastorino, I think you mentioned you were interested in this!

[rylev]

@alexcrichton This seems like a cool idea, but I wonder if this crate is the right place for this. I would think that this crate would offer the minimal amount of wrapping between JS and Rust/Wasm. Bindings to specific browser APIs would then be built on top of this in another crate. Just my gut feeling!

[alexcrichton]

Oh sorry yeah to be clear I don't think that these bindings will live in the wasm-bindgen crate. Instead I think we'd have auxiliary crates the depend on wasm-bindgen which contain all these annotations. That way you'd do something like:

extern crate webapi;

use webabi::alert;
use webabi::console;

fn foo() {
    console::log("foo");
    alert("bar");
}

I was thinking that this repo would be a good place for the development though as it's pretty coupled to the features of #[wasm_bindgen]-the-macro

[fitzgen]

There's even two parsers on crates.io for WebIDL!

FWIW, these look to be two versions of the same crate. The webidl crate seems to have newer releases.

[fitzgen]

So, ideally, one would be able to pull in just the webidl APIs they require, and not everything all at once. Have you thought at all about how to do this @alexcrichton ?

My first thought is that there would be different crates for different classes, but there are definitely circular dependencies here.

If I depend on a crate with #[wasm_bindgen] imports on a bunch of stuff, will wasm-gc collect unused imports? If so, then maybe this isn't actually an issue.

[fitzgen]

If I depend on a crate with #[wasm_bindgen] imports on a bunch of stuff, will wasm-gc collect unused imports? If so, then maybe this isn't actually an issue.

This seems to hold true from a couple initial experiences! :tada:

[alexcrichton]

Oh yeah for sure I was thinking there's be a suite of crates but it's true we'd have to put anything with circular dependencies in the same crate.

That being said like the libc crate anything you don't use will get eliminated at link-time.

[Diggsey]

The generators I implemented here are almost entirely independent of WebGL (ie. it would be very easy to re-purpose for other APIs) and are also designed to support multiple back-ends (although only a stdweb-based back-end is implemented right now).

[corbinu]

@alexcrichton I did some similar work a while back and one of the things I found is that the WebIDL files aren't always representative of the real web. In fact I know that Microsoft went to huge pains to get the definition files in TypeScript to match reality rather than the spec. They also continue to update monthly.

It might be better to write a TypeScript definition parser and use what is here: https://github.com/Microsoft/TypeScript/tree/master/src/lib

I would be very excited to help work on this. I am already using this project and loving it. I am also a huge fan of TypeScript. There are also already many JS modules with either definitions built in or in DefinitelyTyped so it would be amazing if Rust users could tap into that. I think it makes a lot of sense to think of things in a Rust <-> TypeScript kind of way given Rust is statically typed and that TypeScript is transparently compatible with JavaScript.

[alexcrichton]

@corbinu oh fascinating! That actually would sync up well with https://github.com/alexcrichton/wasm-bindgen/issues/18 and otherwise make it easy to import any library rather than just the native WebIDL ones. I think regardless of whether we end up using a WebIDL-based generation process it'd be fantastic to have a Typescript one as well.

If you'd be excited to help out on this I'd be excited to help you :). I think the basic goal for something like this would be to have a tool that slurps up typescript (and maybe some configuration) and spits out #[wasm_bindgen] annotated Rust source code. That way we could either hook it up to a procedural macro in Rust or otherwise make it a standalone tool so the code can be committed!

[corbinu]

@alexcrichton For sure sorry wasn't trying to rain on this effort. Just have seen it tried before.

Yes for sure thats exactly what I am thinking. I will move this over to #18.

[alexcrichton]

Oh no worries at all! I just figured it'd be good to do both, but I agree with your reasoning that it may be best to start with a Typescript -> Rust generator as that'll work for any library, not just Web ones!

[corbinu]

For sure I once it that is working we could possibly use dom.generated.d.ts to enhance the stdweb also. Maybe we should have a separate issue just called something like "Two way type interop?" that would cover everything.

[alexcrichton]

Sounds good to me!

[Dessix]

Certain WebIDL capabilities such as [NewObject] are not replicated in TypeScript declarations- and some of these attributes carry information that may present an opportunity for improved binding typing in strict languages such as Rust.

Is it possible that similar attributes could be represented in TypeScript via decorative wrappers/aliases?
Example:

type NewObject<T> = T;
type Nat = number;

class NatNumberedObject {
  constructor(number: Nat);
  number: Nat;
}
function createNatNumberedObject(number: Nat): NewObject<NumberedObject> {
  return NatNumberedObject(number);
}

The above definitions could provide Nat bindings for Haskell, or bind to u32 or similar bindings in Rust, instead of just number, if there isn't already a straightforward way of doing this.
As for NewObject, I'm not sure where the bindings could take advantage of the knowledge that the return value is a new object each time, but it shows a way to convey what WebIDL can without interfering with normal TypeScript functionality.

[alexcrichton]

@Dessix fwiw I definitely think we will want some form of attributes or tweaking parsed files as I imagine we probably won't be able to use everything literally as-is, but hopefully lots can be!

[pepsighan]

I have also developed a library weedle to parse WebIDL definitions which I made just for the sake of generating these bindings (and a learning exercise at parsers).

I am also thinking to try to build binding generator & I have a question about it.
The WebIDL definitions for DOM are not annotated with the Exceptions that the methods might throw. MDN WebIDL do have [Throws] but does not tell what exactly it throws.
So, I am thinking to add custom annotations to WebIDL definitions one piece of code at a time which will be maintained by us . Otherwise I don't know how we could correctly map the errors.

How do you think this approach would be? Other ideas?

[alexcrichton]

Oh nice @csharad! I think for a first pass I'd be fine with avoiding exposing exceptions and we could add try_* variants later perhaps which return a Result and catch exceptions. Does that sound alright?

[pepsighan]

That sound fine right now. But how would you approach catching exceptions in the future? A stricter typed exception (which the idl don't define) or a catch all exception object?

[alexcrichton]

Currently there's not a great implementation of inheritance or anything like that with wasm-bindgen, so the only way to catch an exception is to have the Err payload be exactly JsValue. Something like https://github.com/rustwasm/wasm-bindgen/issues/152 may help though to have more types of richly typed error values, but we're still aways out from that I believe.

[sendilkumarn]

there is also webidl library for parsing webidl files right? How different weedle from this?

[pepsighan]

@sendilkumarn Yes, there is. Difference between the two would be pretty much none (other than weedle using nom). Also, I developed it just for the sake of learning.

[pepsighan]

While experimenting with webidl binding generation, I came across two problems & a suggestion:

1. How to refer to identifier names which are rust keywords? I am appending _ to rust facing side, but js_name is still causing problem.

#[wasm_bindgen(js_name = self)] <-- this is causing problem 
static self_: window;

1. According to the spec the interfaces should be objects provided in the JS environment. But firefox, chrome do not implement some of them, which causes ReferenceError.
   One example would be ApplicationCache (not available in firefox) with a value provided on window.applicationCache.

Checking Object.getPrototypeOf(window.applicationCache) gets OfflineResourceList on firefox but ApplicationCache on chrome.

Importing the bindings lib without using any functionality causes problems on the browser, as in debug mode compiler does not remove unused bindings making the generated bindings unusable.

One way would be to make sure bindgen selects the appropriate type on that particular browser.

if (typeof ApplicationCache != 'undefined') {
    GetOwnOrInheritedPropertyDescriptor(ApplicationCache.prototype, 'status') 
} else { 
    // fallback
    GetOwnOrInheritedPropertyDescriptor(Object.getPrototypeOf(window.applicationCache), 'status');
}

We could pass additional metadata to type statements.

#[wasm_bindgen(fallback_type = window.applicationCache)]
type ApplicationCache;

1. Also, is there a way to provide js_namespace to static members. Though it is not required, I was trying to create an api like document as well as window::document.

type window;
type Document;

#[wasm_bindgen(js_namespace = window)]
static document: Document;

Otherwise, I could always create window.document().

[alexcrichton]

@csharad oh man this looks like some awesome progress! I'll try an answer some questions

#[wasm_bindgen(js_name = self)]

It's true yeah that self won't work as an identifier in Rust so something like static self: window; isn't gonna work (underscore at the end should be fine). Is it also not working in the macro, though? I thought that arbitrary token streams could be in there and it'd be allowed. There may be a bug though!

According to the spec the interfaces should be objects provided in the JS environment. But firefox, chrome do not implement some of them

Aha I'm not too too surprised! One option here is to use the structural attribute as well. That basically says "generate a JS shim and don't be so pedantic about what it's called". That way it may be a bit more lenient about the precise interface across browsers. Does that help here?

Also, is there a way to provide js_namespace to static members

Aha a good point! Want to open a dedicated issue for this?

[pepsighan]

Is it also not working in the macro, though?

Yeah, not working in the macro. Created an issue at #193.

... to use the structural attribute ... Does that help here?

Yeah, It worked! Though I'll have to use it everywhere 'cause I don't have a way to know whats what.

... provide js_namespace to static members. Want to open a dedicated issue for this?

Opened at #194

[alexcrichton]

@csharad oh it's actually somewhat critical that we don't use structural everywhere for various bindings. One of the main selling points of wasm-bindgen is that it's taking advantage of the future host-bindings proposal to provide faster-than-JS DOM access by ensuring that wasm can directly call C++ engine code. If we use structural then then that's not the case because a JS shim is forcibly created.

I think it's fine to use that in a few odd places but we predominantly do not want to use JS shims via structural to ensure we can continue to take advantage of that future speed boost.

[fitzgen]

@csharad can you share your fork / repo / location of where you're working on this?

[pepsighan]

@fitzgen webapi. Its not in active development currently because of my work.

[fitzgen]

I've just landed the wasm-bindgen-webidl crate to crates/webidl within this repo. The goal is to allow mutliple folks to collaborate on this project incrementally together (since it is definitely more than a single PR!). We don't need to have it working all at once, again this is larger than a single PR, but we should keep its tests passing with every PR we send in.

[alexcrichton]

This is now more-or-less done! In the sense that this issue itself isn't too useful any more and further work items are tracked at https://github.com/rustwasm/wasm-bindgen/labels/frontend%3Awebidl