Avalonia: Hosting Avalonia in WebAssembly

[vectorix]

Would it be possible to host Avalonia with WebAssembly and .NET in a browser?

Then we could write the same C# code and run it native in all browser and on all desktops.

[kekekeks]

It would be possible and we are currently considering our options. The major blocker is that HTML5 doesn't have a proper 2D drawing API, so Skia needs to be compiled to wasm and somehow linked to resulting application. The problem with linking is that Mono doesn't currently provide P/Invoke support for wasm target. Another problem is that P/Invoke means indirect calls (i. e. calls via pointer), while WebAssembly doesn't support indirect calls between modules. That means that Skia needs to be linked statically, so we'll need a custom Mono build.

It would be easier with CoreRT, since it supports static linking out of the box, but they are still implementing MSIL instructions for wasm target, so it's not ready even for an experimental port.

[legistek]

Sorry if this is an obviously terrible idea, but why wouldn't you use WebGL?

[danwalmsley]

Maybe this helps? https://github.com/aspnet/Blazor/blob/dev/README.md

[kekekeks]

@pmoorelegistek We will be using WebGL at some point. The problem is that WebGL by itself doesn't support drawing 2D vector graphics. So we need Skia anyway.

[legistek]

Oh I see what you're saying. I didn't realize it lacked basic 2D drawing. I take it the plain HTML5 canvas doesn't have good enough performance either?

I've been watching this project for awhile and would love to help how I can. Getting this on the web would be the Holy Grail. (Befitting the name Avalon(ia))

Get Outlook for Androidhttps://aka.ms/ghei36

From: Nikita Tsukanov
Sent: Friday, March 9, 12:30 AM
Subject: Re: [AvaloniaUI/Avalonia] Hosting Avalonia in WebAssembly (#1387)
To: AvaloniaUI/Avalonia
Cc: Peter N. Moore, Mention

@pmoorelegistekhttps://github.com/pmoorelegistek We will be using WebGL at some point. The problem is that WebGL by itself doesn't support drawing 2D vector graphics. So we need Skia anyway.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/AvaloniaUI/Avalonia/issues/1387#issuecomment-371725952, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMw7S1GOycAJX3aLJnlzy5xM6yq_KrTiks5tciGKgaJpZM4SR4pe.

[legistek]

Hmm. The problem with compiling Skia to WASM is you'll lose any benefits of hardware graphics acceleration on the device. Browser will be doing all the math, painting pixels, etc. I don't know that you'd ever get anything close to 60fps with a complex UI.

Have you tried just using the HTML5 Canvas or SVG?

[kekekeks]

Skia can use OpenGL ES profile for hardware-accelerated rendering. Which is basically what is provided by WebGL.
HTML5 Canvas and SVG are not sufficient to render Avalonia.

[legistek]

You're saying Skia / OpenGL will be able to access hardware acceleration (i.e. the GPU) after compiled to WASM? Are you certain about that? Everything I understand about WASM would lead me to think not.

I would think you'd be better off using three.js - or better yet porting it to C# - wrapped around WebGL.

I'd be happy to look into it some more and experiment. Do you have a working prototype using Mono-to-WASM at the moment with any rendering at all?

[kekekeks]

You're saying Skia / OpenGL will be able to access hardware acceleration (i.e. the GPU) after compiled to WASM

Emscripten toolchain has complete support for WebGL. It might be a bit difficult to make it work with Mono since it has it's own js glue code, but other than that I don't see any issues.

[danwalmsley]

https://github.com/praeclarum/Ooui/wiki/Xamarin.Forms-with-Web-Assembly

No need for Skia now,

[kekekeks]

@danwalmsley it uses HTML. Avalonia can't be rendered with HTML.

[adoris]

Is there are any try to use Skia/WebGL? Any branch for this experiment?

[legistek]

if I understand right, what @kekekeks is saying is that it's not feasible at the moment because Mono can't use P/invoke to call the Skia functions. So while they can compile Skia to WASM, there would be no way for the .NET code to access it.

What I don't understand fully, though, is why it has to be Skia. There are Javascript libraries that provide 2D drawing function wrappers over WebGL (e.g., https://threejs.org/). I'd think it at least worth an experiment to make a DrawingContext that wraps three.js and go from there. If that works, porting threejs to C# and simply including it in the Avalonia build would not be too bad of a project and would probably get you performance on par with Skia.

[kekekeks]

@pmoorelegistek The main problem is the complete lack of a proper text measurement API. We can't get our TextBox to work if we don't have an accurate position of each individual symbol. Even for our TextBlock we need to be able to measure and lay out text lines properly. There is no way we can do that with the APIs available to JavaScript.

[yowl]

I added an issue for pinvoke support on mono so this can be tracked. https://github.com/mono/mono/issues/8007

[TonyHenrique]

See
http://www.noesisengine.com/webgl/Samples.Buttons.html

[legistek]

It looks nice but what are we looking at exactly?

---

From: TONY HENRIQUE [e-mail]
Sent: Thursday, April 12, 2018 6:38:15 PM
To: AvaloniaUI/Avalonia
Cc: Peter N. Moore; Mention
Subject: Re: [AvaloniaUI/Avalonia] Hosting Avalonia in WebAssembly (#1387)

See
http://www.noesisengine.com/webgl/Samples.Buttons.html

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/AvaloniaUI/Avalonia/issues/1387#issuecomment-380976051, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMw7S61hESqCLDWhVkG6jvegNBn7KHWtks5tn-VngaJpZM4SR4pe.

[kekekeks]

Noesis has it's own rendering engine, if I recall correctly. A proprietary one. We don't have the resources to create a whole new 2D graphics library, so we are using an existing one, which is Skia. Until we have a way of linking it in on wasm target (be it Mono or CoreRT), we won't be able to get Avalonia working in the browser.

[legistek]

Hey Nikita - since this thread keeps coming alive -
I looked deep into three js and the code for text rendering could easily be adapted for text measuring. I'd be willing to make the modifications if you're willing to consider something besides skia.

And beyond that, I don't think a port of three js to C# - or at least enough of it for our purposes - would be at all that difficult.

This is my company, and as you can see from our product we have a vested interest in your project: www.legistek.com

[kekekeks]

Well, you see, the only text measurement API that I've found for HTML5 Canvas is measureText call which returns a TextMetrics object. The problem is that it doesn't support anything but text width:

As you can see, "advanced" properties are only supported on Google Chrome and need to be explicitly enabled in browser settings. Getting text width is unfortunately not enough for implementing our IFormattedText interface.

[kekekeks]

Does three.js have it's own rendering engine that can draw text using TrueType/Freetype fonts with proper subpixel rendering?

[kekekeks]

If it does, it might be worth to create some kind of a wrapper, but currently calling JS code from WASM is very inconvenient.

This is currently the only way of calling JS code from C#, which is basically an eval call.

There is some existing glue code here (JS part here) which uses the internal call from an alternative mono wasm target implementation, which is basically an eval too, so it could be adapted.

[legistek]

https://github.com/mrdoob/three.js/blob/f81506e172571ab106d0164530bbc1a4802fc2d4/src/extras/core/Font.js

They are going through the font glyphs and turning them into path geometries and then feeding them to WebGL. If they can do that then they could have measured them. It looks like nothing more than an oversight that they didn't include measuring functions. (I'm not yet clear on what font format they're expecting but my assumption would be WOFF).

re JS interop, again I would assume the endgame would be porting what we need from three.js to C# so that all the calculating code would be done in WASM and the only time you'd need to interop would be the final render into WebGL. But why not try a POC at least?

[kekekeks]

@pmoorelegistek It seems that Mono WebAssembly SDK is providing libmono and an example C source file with the entry point. That means that we can link additional libraries to the resulting runtime mostly without issues. mono-dl-wasm.c file still needs to be changed for supporting __Internal but at least we could use existing build scripts

[jmacato]

@kekekeks I'd like to bring this lib to our attention: SharpFont; could be useful for text rendering, alongside the PixelFarm renderer that i linked before.

I just noticed that PixelFarm is already using this particular library, so yeah

[birbilis]

Could be of some use:
https://github.com/mattdesl/fontpath (convert Fonts to paths)
http://paperjs.org/features/ (this seems to use Canvas, so not sure how fast it is)

I believe Three.js is an interesting option (maybe combined with FontPath above)

[birbilis]

also looked into PixelFarm mentioned above, and it does seem to use a Typography module and support various renderers (including OpenGL ES I think)
https://github.com/PaintLab/PixelFarm
https://github.com/LayoutFarm/Typography
https://github.com/LayoutFarm/HtmlRenderer

[insinfo]

NanoVG, maybe this is an excellent OpenGL-accelerated 2D API option to be ported to WebGL, it's a small API with several ports of it for other BackEnds.
https://www.youtube.com/watch?v=c8IsoDQ7ea8
https://github.com/memononen/nanovg
https://github.com/cmaughan/nanovg
https://github.com/ollix/MetalNanoVG
https://github.com/bkaradzic/bgfx/tree/master/examples/20-nanovg
https://github.com/ocornut/imgui
https://github.com/wihu/NanoGUI-Mono
https://github.com/wjakob/nanogui
https://sites.google.com/site/bitiopia/
https://github.com/cambiata/NanoVGDotNet
https://github.com/jpbruyere/Crow
https://github.com/jpbruyere/vkvg

[kekekeks]

NanoVG is a C library. So we are back to the square one where we need to somehow enable native code interop. I'll try to get Skia# working on top of WebAssembly first before trying managed alternatives (which will be super-slow because Mono currently runs managed code using interpreter)

[kekekeks]

BTW, we can't use LGPL code (nanovgdotnet is LGPL-licensed) for WASM target since everything will be linked to a single file once Mono gets a proper compiler instead of interpreter.

[PerArneng]

This would be so cool. I think it could be the killer app since there are not alot of options today for doing that.
Some demos from other UI toolikts running in webassembly in the browser:

• https://danilw.github.io/GLSL-howto/nanogui/nanogui.html
• https://pbrfrat.com/post/imgui_in_browser.html

[jpbruyere]

canva and SVG are the browser's 2d api's.

[galvesribeiro]

There are some people working Skia for wasm https://github.com/Zubnix/skia-wasm-port but like @kekekeks said, you need a custom built mono liked with skia... I have canvas APIs wrapped for Blazor https://github.com/BlazorExtensions/Canvas using the CanvasRenderingContext2D APIs. So, it wouldn't be hard to use it but, the subpixel measurement required by Avalonia text renderer would probably not work with it...

[kekekeks]

@galvesribeiro I had some success with building libSkiaSharp last month (I even have some code that actually creates some skia types from C#) and encountered two major problems:

1) The current Mono interpreter has a hardcoded list of function signatures that can be P/Invoked. If I try to call something else it fails with errors like CANNOT HANDLE COOKIE VIIFFI. We need make that call pluggable, analyze the list of imported function signatures used by SkiaSharp and generate needed trampolines

2) WASM implementations in Chrome/Firefox have an arbitrary limit on the number of functions declared in the binary fine. I was actually hitting that limit even with incomplete (no libwebp, pdf, svg, xml) libSkiaSharp and getting "functions count of 1589554 exceeds internal limit of 1000000" error. To make it load I had to remove GPU support which will hurt performance by a lot. The problem might be caused by the fact that I'm currently adding all libSkiaSharp exports to the mono.wasm binary. We might need to integrate a linker step that removes unused DllImports, reanalyze the SkiaSharp.dll to get the list of actually used imports and only then generate a separate C file. However I'm afraid that without stripping GPU support the function count will stay way to huge.

The first problem is solvable, but we might be hitting a somewhat "fundamental" limit with the second one. I'm not sure if it's possible to reduce the function count without removing the needed features.

[kekekeks]

We might also try to resort to WASM dynamic linking to make libSkiaSharp live in a separate wasm file. That should increase the total function count limit.

[galvesribeiro]

We might also try to resort to WASM dynamic linking to make libSkiaSharp live in a separate wasm file. That should increase the total function count limit.

I'm not sure with the current wasm implementations we can do that... Last time I checked, you can't dynamic load anything neither can you _link_ another .wasm.

[kekekeks]

According to this doc dynamic linking should be possible in MVP. We can't make DllImport work with it out of the box and have to resort to C codegen, but it should be possible.

[galvesribeiro]

Yeah, what I meant was that I'm not sure the browsers support that yet. There are discussions in some other frameworks where they can't so some stuff just because of that same problem. I may be wrong, its being a while since I don't check it...

[birbilis]

Can't a wasm file talk to javascript? If it can, that one could load other wasm(s) and act as a bridge between them

[kekekeks]

We need to at least somehow setup a shared linear memory for multiple wasm files.

[kekekeks]

https://github.com/mono/SkiaSharp/wiki/Building-on-Linux
https://github.com/mono/SkiaSharp/wiki/Building-on-Linux-(LEGACY)

[kekekeks]

FYI: I've spent a weekend tinkering with WASM target and SVG render target about a month ago and decided that the current performance of Mono interpreter is way too slow. It also might be a fault of our layout algorithms or me missing something since the debugging story on WASM is currently not existing

You can see the incomplete demo here: http://testapp.keks-n.net/ don't even try to open calendar or datagrid pages, they take ages to load.

[ghuntley]

Nice work @kekekeks!

[ghuntley]

@kekekeks you might enjoy this blog post re: skia and perf. Hopefully it gives you some ideas you can incorporate. https://platform.uno/skiasharp-support-for-webassembly-via-uno-platform/ :)

[moderndev]

wondering if there's anything you can share with the mono folk on improving mono for this type of workload ?

[Gillibald]

I think for now we should closely watch the UNO project what they come up with. They are already working with the Mono team.

[kekekeks]

@moderndev the thing is, it's currently kinda pointless to work on WASM backend because of performance issues both on Mono and Avalonia sides. We could get some nice demo that would be nowhere near production-ready state (i. e. seconds per frame instead of frames per second) until those issues are fixed. That's why I was focusing on XAML compiler this year instead: it help all existing platform backends and brings us closer to WASM.

Also, as the time goes by, it's getting easier to run code on WASM and it's not that hard to write a new backend, it usually takes a week or two to get things going.

[Perksey]

Have you considered using WebGL via WebAssembly?

[kekekeks]

Yes, we did, Skia can utilize WebGL for rendering.

Right now we are focusing on optimizing the core framework, since it runs way too slow on WASM target as it is. At the same time WASM tooling getting improvements, so it will be easier to get a port later.

[legistek]

I've spent a LOT of time working with Blazor, both in server-side and WASM mode, and have several observations that may be helpful:

First, just to get the obvious out of the way, Mono WASM (even interpreted) is way fast enough today to handle the backend of a lookless UI. By that I mean dependency properties, binding, styling, etc. etc. I know because I implemented them in a new Blazor framework we've created for a proprietary project. (I'd pre-compile the XAML if possible though; I don't know if Avalonia can do that, sorry).

Second, the bottleneck with making a UI with WASM is in the interop with Javascript. For reasons that escape me, the WASM designers in their wisdom chose not to give the WASM VM direct access to the DOM. Therefore anything that touches on what the user sees in the browser has to ultimately go through JS. And, of course, this requires serializing things to JS-friendly strings (which WASM has no inherent support for, and are thus super inefficient) that JS then de-serializes back into its own object model. Big stinking mess.

So how is that problem solved? Minimize interop, obviously.

Blazor does this by using a retained mode rendering system. You build a render tree in Razor or C# (Razor compiles to C#, so same thing), Blazor serializes it and sends it to the JS runtime, which then constructs the DOM. Then Blazor diffs the render tree every time anything changes, and only sends the changes back through JS. The initial render can be slow if the DOM is very complex, though this will certainly be better if not unnoticeable with AOT compilation. Subsequent renders feel like native apps. Again, even with interpreted Mono, and a completely "lookless" render tree. Meaning for every UIElement equivalent in a WPF-type visual tree there'd be a <div> with unique styling.

Of course this is conceptually very similar to what WPF does. WPF uses a retained mode rendering compositor that is deep under the hood. When you handle OnRender all you're doing is helping to build a tree of commands that get sent to a compositor in a native DLL. The compositor is what operates at 60fps and what actually does all the heavy lifting (i.e, what WebGL or Skia would be doing).

Point being, if you're going to successfully build a web platform for Avalonia or anything else, IMHO you're going to need to build an entirely Javascript-based compositing engine and use a retained mode rendering system like Blazor or WPF. A Skia WASM port, I don't believe will get you nearly acceptable performance because there will still be too much interaction between WASM and JS. I would forget about Skia entirely, and focus on designing data structures representing a render tree that is built with C# and can be efficiently diff'd, serialized and sent to the JS runtime only when it changes. I'd design that JS compositing engine probably with ThreeJS. Again, I'd stay away from Skia or anything else that would require a lot of interop between WASM and JS.

There's zero doubt this can be done, because we know Blazor works, and that's with using plain old HTML + CSS as the compositing engine. There's every reason to think it would be even faster and more impressive with a WebGL based compositing engine. But unfortunately I think that compositing engine has to be 100% Javascript to get anything close to 60fps.

[drearyrainDeng]

I've spent a LOT of time working with Blazor, both in server-side and WASM mode, and have several observations that may be helpful:

First, just to get the obvious out of the way, Mono WASM (even interpreted) is way fast enough today to handle the backend of a lookless UI. By that I mean dependency properties, binding, styling, etc. etc. I know because I implemented them in a new Blazor framework we've created for a proprietary project. (I'd pre-compile the XAML if possible though; I don't know if Avalonia can do that, sorry).

Second, the bottleneck with making a UI with WASM is in the interop with Javascript. For reasons that escape me, the WASM designers in their wisdom chose not to give the WASM VM direct access to the DOM. Therefore anything that touches on what the user sees in the browser has to ultimately go through JS. And, of course, this requires serializing things to JS-friendly strings (which WASM has no inherent support for, and are thus super inefficient) that JS then de-serializes back into its own object model. Big stinking mess.

So how is that problem solved? Minimize interop, obviously.

Blazor does this by using a retained mode rendering system. You build a render tree in Razor or C# (Razor compiles to C#, so same thing), Blazor serializes it and sends it to the JS runtime, which then constructs the DOM. Then Blazor diffs the render tree every time anything changes, and only sends the changes back through JS. The initial render can be slow if the DOM is very complex, though this will certainly be better if not unnoticeable with AOT compilation. Subsequent renders feel like native apps. Again, even with interpreted Mono, and a completely "lookless" render tree. Meaning for every UIElement equivalent in a WPF-type visual tree there'd be a <div> with unique styling.

Of course this is conceptually very similar to what WPF does. WPF uses a retained mode rendering compositor that is deep under the hood. When you handle OnRender all you're doing is helping to build a tree of commands that get sent to a compositor in a native DLL. The compositor is what operates at 60fps and what actually does all the heavy lifting (i.e, what WebGL or Skia would be doing).

Point being, if you're going to successfully build a web platform for Avalonia or anything else, IMHO you're going to need to build an entirely Javascript-based compositing engine and use a retained mode rendering system like Blazor or WPF. A Skia WASM port, I don't believe will get you nearly acceptable performance because there will still be too much interaction between WASM and JS. I would forget about Skia entirely, and focus on designing data structures representing a render tree that is built with C# and can be efficiently diff'd, serialized and sent to the JS runtime only when it changes. I'd design that JS compositing engine probably with ThreeJS. Again, I'd stay away from Skia or anything else that would require a lot of interop between WASM and JS.

There's zero doubt this can be done, because we know Blazor works, and that's with using plain old HTML + CSS as the compositing engine. There's every reason to think it would be even faster and more impressive with a WebGL based compositing engine. But unfortunately I think that compositing engine has to be 100% Javascript to get anything close to 60fps.

so if alavonia want to get better performance on wasm ,it needs to convert xaml controls to DOM elements? can we reuse blazor render trees ?(if it is decomposable)

[legistek]

so if alavonia want to get better performance on wasm ,it needs to convert xaml controls to DOM elements?

Yes and no. I see two overall possibilities. (1) Render to HTML and CSS; (2) render to WebGL.

In (1), yes you'd generally have a 1:1 correspondence between every UIElement and a DOM element (generally a div). You'd use CSS to "draw" backgrounds, borders, etc., and implement transforms. One significant problem is you'd lose pixel-perfect rendering and potentially be beholden to quirks between browsers. However with very disciplined use of CSS you can get almost perfect rendering on all browsers (particularly if the framework is handling the layout and every div is position: fixed). This is how my prototype Xamarin Forms Blazor render platform works.

In (2) you'd treat the whole browser window as one HTML Canvas and use WebGL to render everything. Now as has been pointed out before, WebGL doesn't inherently support 2D graphics so you'd need to use something JavaScript-based like ThreeJS. In this scenario you'd only have 1 DOM element (the Canvas). Since WebGL can achieve high FPS theoretically this should give us pixel perfect rendering AND native performance. In this case we'd have the Avalonia render context produce a render tree that is sent to a custom made Javascript runtime at the end of the render pass and translated into drawing commands by the runtime. The JS runtime would then invoke WebGL/ThreeJS accordingly.

can we reuse blazor render trees ?(if it is decomposable)

For (1), possibly, but I think we could do a lot better. As far as I can tell the DOM building is happening in C# and actual text strings of HTML and CSS are being sent to the JS runtime. They have to do it this way, I assume, because Blazor supports arbitrary HTML content. In our case, even in (1) we'd be using only a small and strict subset of HTML and CSS, so I imagine a much more efficient DOM building engine could be devised. And in (2) the number of drawing commands and parameters would be even smaller. In either case, I think we would want to serialize to numbers rather than JSON strings.

Here's a possible example of how you could represent a serialized drawing command that would cover rectangles, ellipses, and lines (all numbers are 4-bytes, either floats or int32):

(Bytes)
[00-03] Command
[04-07] X
[08-11] Y
[12-15] Width
[16-19] Height
[20-23] Opacity
[24-27] Left Border Thickness
[28-31] Top Border Thickness
[32-35] Right Border Thickness
[36-39] Bottom Border Thickness
[40-43] TopLeft Corner Radius
[44-47] TopRight Corner Radius
[48-51] BottomLeft Corner Radius
[52-55] BottomRight Corner Radius
[56-59] Background Brush ID (defined separately)
[60-63] Border Brush ID (defined separately)
[64-67] Effect ID (defined separately)
[68-71] Transform ID (defined separately)

So 72 bytes per primitive could cover the vast majority of 2D drawing except for text, and require no JSON serialization between JS and WASM. Compare that to the amount of HTML needed to describe the same thing:

<div style="position: fixed; left: 0px; top: 0px; width: 100px; height: 100px; ....." </div>

That's over 100 characters and not even close to complete.

Bottom line, no I wouldn't try to copy Blazor's structures but rather do something much more tailored to the limited number of primitives used by a lookless UI.

[kekekeks]

Note that proposed methods will have severe problems with advanced text layout. The layout has to happen on C# side, so text measurement and layout also has to happen on C# side. That basically forces us to transfer each symbol as font+glyph+size+position structure and pray that browser will render it correctly.

[kekekeks]

My initial Avalonia-on-Blazor-via-SVG experiment has used a fixed-width font with hardcoded measurements. Even those had problems with characters outside of the font.

[legistek]

That basically forces us to transfer each symbol as font+glyph+size+position structure and pray that browser will render it correctly.

Don't you have that issue on any platform?
But either way, using WebGL + threeJS you'd avoid any dependence on the browser's rendering.

(Yes, threeJS would need to be forked to add text measuring but this should be trivial since it already supports text rendering).

[Gillibald]

The main issue is that browsers do not understand glyph indices etc. We could very much "transfer" a GlyphRun to the browser side but then we still have to rasterize the glyphs ourselves. That will not yield pixel perfect results.

We could just transfer the characters of a GlyphRun and let the Browser do all the shaping again. This will most likely produce decent results but uses more resources.

[Gillibald]

We can measure text etc but we can't apply kerning or any other glyph positioning. Everything else is possible.

[kekekeks]

let the Browser do all the shaping again. This will most likely produce decent results but uses more resources.

The problem is that it is likely to produce different results. Which will break the layout.

[Gillibald]

It should not produce different results in size if the browser follows common standards. The only difference is the quality of rendered glyphs.

[legistek]

It should not produce different results in size if the browser follows common standards
Well, what browsers SHOULD do and what they DO do is not always the same.

Anyway if I'm understanding correctly the challenge is to make sure that text measurements in the layout pass match what the renderer actually produces. But this should be the case if the renderer and the measuring system are part of the same codebase. Thus, if you used HTML+CSS to render, and canvas measureText to measure, you should get agreement between the two. Likewise if you used WebGL + ThreeJS (or even Skia) to both measure and render text, you should get agreement.

The layout pass in C# would still need interop with JS to measure strings in either case, but some clever caching of string and font property combinations along with the fact that full layout passes don't happen terribly often should get you where you need to be.

[legistek]

Also just an FYI, invoking JS from WASM does NOT require an async call in the Mono runtime, even though the current version of Blazor does. Earlier prototypes of Blazor allowed synchronous invocation of JS from C#. (I believe they changed this so that Blazor apps could be transparently switched between running server side - which of course requires an async call - and client side). So you can implement a JS-based text measuring method in your drawing context without having to rewrite everything to be async.

[kekekeks]

JS-based text measuring

The problem is that there are no text measurement APIs on JS side. At least there weren't when I've last checked.

measureText from canvas is simply not sufficient

[legistek]

measureText from canvas is simply not sufficient

Well, there's a lot more you can do in JS to get the text measurements you need. measureText is one of many tools.

The problem is that there are no text measurement APIs on JS side. At least there weren't when I've last checked.

Take a look at three.js's code for rendering text from TTF or WOFF:

https://github.com/mrdoob/three.js/blob/37285df967f0a4d0b1578703e665b35df04361c2/src/extras/core/Font.js

This could easily be expanded to measure rather than just render.

[Gillibald]

We don't gain much from that. We already have the ability to generate svg data for a given string. This will still not produce pixel perfect results.

I think we can work something out with my new text layout that produces GlyphRun objects under the hood. We then just transfer these GlyphRuns to the Javascript side and let it process there again. We can apply same font features etc. but have to rely on the browser's shaping capabilities to produce the same results. In reality this will not be an issue because every browser uses HarfBuzz as well.

https://github.com/Gillibald/Avalonia/blob/080084f14ae6aa367ac825b4199c67a09de86620/src/Avalonia.Visuals/Media/GlyphRun.cs

A lightweight version of this will just contain the characters and info about font properties.

[legistek]

@Gillibald so you'd measure in C#, pass just the text and font info to the browser, and pray it measures the way you assumed it would?

The only way you can guarantee agreement between the measurement and the rendering is for both to be done by the same code.

And since you have to minimize the amount of data moving between C#/wasm and JS, that means the only viable solution is to do both measurement and rendering browser side.

[Gillibald]

Rendering only differs on sub pixel level. Measures are defines by the data that a font defines. The only possible difference in measurement comes from different shaping. So for example our shaping engine selects a ligature for specified text but the browser doesn't. This will measure differently because we use one glyph but the browser would use two glyphs. If we would let the browser measure everything we would need to cross Javascript / Wasm interop two times. Also the amount of data would be huge. Because we would need to transfer measures of each glyph.

[legistek]

Measures are defines by the data that a font defines. The only possible difference in measurement comes from different shaping.

In a perfect world maybe. Work with browsers much? :/

If we would let the browser measure everything we would need to cross Javascript / Wasm interop two times. Also the amount of data would be huge. Because we would need to transfer measures of each glyph.

Cache measurements by font+size+style+string.

[birbilis]

Can't you use a JavaScript-based text rendering engine or something so that you can control what it does? (or a more general 2D rendering engine that also supports text like https://two.js.org/examples/)

[rstm-sf]

https://github.com/mono/SkiaSharp/issues/1219

[aboimpinto]

Uno Platform (https://platform.uno/) is rendering the WPF from Microsoft to WebAssembly. There should not that difficult to port the code to render Avalonia too ...

Maybe, there could be a way to join efforts ...

[Gillibald]

Uno does not render anything as Avalonia does. They use HTML elements.

[yowl]

They also have a Skia backend, not sure it's generally available for Wasm yet, but it is the plan.

[birbilis]

related comment:

On Windows 7, we still have the WebAssembly story, but we also announced a new Skia backend, that can be wrapped in a WPF shell so you can run "natively" and outside of a browser context.

https://developercommunity.visualstudio.com/idea/378455/net-ui-standard.html

also:

When using Uno Platform to get your UWP/WinUI 3.0 and later apps on Windows 7 you now have THREE options for running your app on Windows 7: 1. Run your app as a PWA. Your Windows 7 will need Chrome installed on it. 2. Run your app on Chromium through Electron. 3. Use Uno Platform 3.0 supported Skia backend, that can be wrapped in a WPF shell so you can run natively on Windows 7 and outside of a browser context.

https://platform.uno/windows7/

[aboimpinto]

What about Ubuntu (Linux environment)?

[birbilis]

regarding Uno and Linux see here: https://platform.uno/uno-platform-for-linux/

[aboimpinto]

regarding Uno and Linux see here: https://platform.uno/uno-platform-for-linux/

I was referring Avalonia in Linux environment compiling into WebAssembly

[birbilis]

guess Avalonia could also get ideas from Uno, or maybe even use it as a backend or something?

[Gillibald]

Just to be clear. Text rendering is no longer an issue. The "only" thing we need is a platform implementation for WASM that is setting up WebGL and exposes all input events. The threading implementation might also be tricky.