Godot: Optimizing Godot for the Web

[kripken]

Hi! I saw

http://www.godotengine.org/wp/webassembly-godot-on-the-web/

which mentions asm.js issues. I'm one of the emscripten devs, and I'd like to help if you're interested.

Specifically I saw problems with Chrome and IE/Edge mentioned. Edge's latest prerelease version has asm.js optimizations enabled by default, and also has excellent WebGL perf, so I hope that Microsoft is no longer a problem, leaving only Chrome.

We should be able to run Godot well even on Chrome. What were the specific problems you encountered?

A common issue recently have been memory limitations on Chrome. I've been doing memory optimization work specifically for that, with some significant improvements (half the memory at peak in my tests), and I could help test that on Godot, if you want.

[reduz]

Hi Alon! Thanks for contacting us. To be honest, I haven't tried again
since a year (only tested Edge and it seems to work). But my impression is
that it only worked well on High end PC and Firefox. Normally, Chrome
seems to freeze for a long time compiling the asm.js codebase and use a lot
of memory, resulting in swapping and freezes. On mobile, neither Firefox,
Chrome or Safari seem to work. Unity users seem to keep complaining about
the same things so I kind of gave up.

The approach of having to pre-allocate a very large buffer for code to work
at the same time it compiles asm.js seems to be a killer combination.
Webassembly seems to tackle the problem more elegantly, by using a specific
memory manager and also not having to allocate that much memory for
compiling, so our plan was to patiently wait on WebAssembly.

If you believe these issues can be fixed (or are already fixed), I can give
it another try and point you to it if something doesn't work.

On Thu, Sep 3, 2015 at 3:08 PM, Alon Zakai [email protected] wrote:

Hi! I saw

http://www.godotengine.org/wp/webassembly-godot-on-the-web/

which mentions asm.js issues. I'm one of the emscripten devs, and I'd like
to help if you're interested.

Specifically I saw problems with Chrome and IE/Edge mentioned. Edge's
latest prerelease version has asm.js optimizations enabled by default, and
also has excellent WebGL perf, so I hope that Microsoft is no longer a
problem, leaving only Chrome.

We should be able to run Godot well even on Chrome. What were the specific
problems you encountered?

A common issue recently have been memory limitations on Chrome. I've been
doing memory optimization work specifically for that, with some significant
improvements (half the memory at peak in my tests), and I could help test
that on Godot, if you want.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417.

[kripken]

Yes, definitely there should be a very big improvement since last year, on all browsers. I think it's worth trying. And if you still see issues, I would like to debug and investigate those.

Specifically about memory on Chrome, exactly as you said, it allocates a lot of memory during compilation, then keeps it around while starting to run the game, which can lead to running out of memory. However, we seem to have identified a very simple workaround. Testing that workaround should be just several lines of code.

This workaround was identified recently in response to issues Unity has been seeing. With the workaround, I see half the memory at peak in Chrome, compared to without the workaround. Hopefully it will work similarly here.

For Edge, the Edge devs are very active on listening to devs about issues and fixing them, so if we find anything, we can report it to them.

I certainly agree WebAssembly is going to fix this a lot better :) But it will take a significant amount of time before that technology is ready, and meanwhile asm.js should be capable of running Godot.

[punto-]

I think most of our problems (other than the stuff that will be/is fixed
transparently by emscripten/the browsers/webassembly) is simple
housekeeping stuff that we don't easily know how to do, like generating the
right javascript files, generating the right html to embed that javascript
and launch the game, make it customizable when users export their game,
figuring out the javascript sound stuff.. Also we don't have a Worker
abstraction (only Thread), and there's a couple of things that are Threads
and could be Workers (because they don't necessarily need to share memory
in real time with the main thread, and they already communicate via a
message queue), that would probably improve performance on a browser.

That said, if we can improve on the issues Juan mentions, there might be a
bigger incentive to fix the other stuff, which should be relatively simple.

On Thu, Sep 3, 2015 at 8:08 PM, Alon Zakai [email protected] wrote:

Hi! I saw

http://www.godotengine.org/wp/webassembly-godot-on-the-web/

which mentions asm.js issues. I'm one of the emscripten devs, and I'd like
to help if you're interested.

Specifically I saw problems with Chrome and IE/Edge mentioned. Edge's
latest prerelease version has asm.js optimizations enabled by default, and
also has excellent WebGL perf, so I hope that Microsoft is no longer a
problem, leaving only Chrome.

We should be able to run Godot well even on Chrome. What were the specific
problems you encountered?

A common issue recently have been memory limitations on Chrome. I've been
doing memory optimization work specifically for that, with some significant
improvements (half the memory at peak in my tests), and I could help test
that on Godot, if you want.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417.

[kripken]

I'd also be happy to help with emscripten stuff like making sure you emit the right JS and HTML.

I have a checkout of the engine and I built it from source on my linux machine, is there documentation of how to test the HTML5 exporting? I can't find any on the wiki and I couldn't figure out how to make it work in the editor.

[vnen]

@kripken There's a post on the forum explaining the steps to make the HTML5 export work.

Edit: there's also this wiki page made by another user.

[punto-]

The export process in general is documented here:
https://github.com/okamstudio/godot/wiki/export . You would do this with a
game, for example demos/2d/platformer, but in this case you want to use a
"custom template" on that export dialog. Your template will have the output
from your build.

The instructions to create the template are here:
https://github.com/vkbsb/godot/wiki/export_html5

After the export, you get 4 files, the 3 you put in the template, plus the
"data.pck" with the contents of the game. If you want to skip the package
template + export step, you need to modify the html and js files that come
out of the scons build, to point to the right data with the right file size
and allocate the amount of memory you want, this is all done here:

https://github.com/okamstudio/godot/blob/master/platform/javascript/export/export.cpp#L197

then just replace them on your webserver.

If this all works we might actually be ok with the export process (the step
to create the template is normally done by our build script, and people
download the precompiled template and use it to export their game). What
I'm not sure of is if this is something that developers can insert into
their own html page, or do we need to inject that into the package when we
export.

In any case, this should be good for the workflow of building the engine
and testing it with a game. Maybe the "separate_asm" step comes before
creating the export template?

On 4 September 2015 at 18:57, Alon Zakai [email protected] wrote:

I'd also be happy to help with emscripten stuff like making sure you emit
the right JS and HTML.

I have a checkout of the engine and I built it from source on my linux
machine, is there documentation of how to test the HTML5 exporting? I can't
find any on the wiki and I couldn't figure out how to make it work in the
editor.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417#issuecomment-137790449.

[trond]

FWIW I found HTML5 export to be broken and spent some evenings getting it to work on my Mac setup, and I have made a few fixes that I found necessary to make export in general and also 3D stuff work a bit better. My editions are here: #2361

There are some obvious remaining issues which I have not come to the bottom of yet (I tested mainly with Platformer 3D), most notably srgb not working correctly (causing all ground textures to be black, which can be worked around by just ignoring the extension list for srgb and hard-coding "srgb_supported=false"), and shadowmaps not working (causing a too dark scene once srgb support is disabled), for which I have no better workaround than simply disabling shadow-casting for the sunlight in the scene.

[reduz]

@trond Those issues don't worry me much, it's probably Godot just not detecting the relevant extensions.. the biggest missing feature currently is sound support. As mixing in a thread is impossible and Godot has it's own audio mixing architecture (does not use OpenAL), will have to rewrite AudioServer to use WebAudio directly. Started working on it.

[reduz]

@kripken I'm using the WebAudio API to reimplement AudioServer, but i'm running into a few limitations. I wanted to ask mainly, how do you recommend passing the large chunks (samples) of audio to Javascript efficiently using inline EM_ASM()? This is only for registering them into WebAudio.

[godotengine]

So far it seems like AudioServer can be implemented over WebAudio, not
having much issues besides the strangeness of the API and the documentation
wtfs like this (http://imgbin.org/images/25933.png). Reverb is convolution
based.. so I don't think whoever designed webaudio was thinking of games.
I don't think it would be too difficult to generate an IIR from the Godot
reverberator, but I'd like to be able to use the computer while playing a
game. It's a shame a Lexicon style reverb isn't provided as an
alternative.. Mobile users would also have appreciated it.

On Mon, Sep 7, 2015 at 10:12 PM, Juan Linietsky [email protected] wrote:

On a side note, webaudio is retarded:

computedPlaybackRate(t) = playbackRate(t) * pow(2, detune(t) / 1200)

Why not just taking a sampling rate, for god's sake. If you have to
explain this formula to a programmer, the programmer can do it him/herself.
This is just bad API design because I want to pass a sampling rate and have
to undo the formula..

On Mon, Sep 7, 2015 at 4:08 PM, Juan Linietsky [email protected]
wrote:

@kripken https://github.com/kripken I'm using the WebAudio API to
reimplement AudioServer, but i'm running into a few limitations. I wanted
to ask mainly, how do you recommend passing the large chunks (samples) of
audio to Javascript efficiently using inline EM_ASM()? This is only for
registering them into WebAudio.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417#issuecomment-138360524.

_OkamStudio_

_OkamStudio_

[reduz]

@kripken Well, got stuck. Emscripten borked and died:

/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272
throw new JS_Parse_Error(message, line, col, pos);
^
TypeError: object is not a function
at new JS_Parse_Error (/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:261:21)
at js_error (/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272:15)
at croak (/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:742:17)
at token_error (/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:749:17)
at unexpected (/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:755:17)
at /home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:829:33
at block_ (/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:999:32)
at /home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:978:43
at /home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:982:29
at /home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:1112:43

I guess one of the calls to EM_ASM is the culprint, but I'm seriously clueless. Here's code I used:

http://pastebin.com/XKbhcDZE

[godotengine]

ok, fixed, I didn't realize I had to add a _ at the end of the EM_ASM_()
call for passing values. Would had appreciated an example that does not use
return value, and then one with return value as two separate examples.

On Tue, Sep 8, 2015 at 1:57 AM, Juan Linietsky [email protected]
wrote:

@kripken https://github.com/kripken Well, got stuck. Emscripten borked
and died:

/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272
throw new JS_Parse_Error(message, line, col, pos);
^
TypeError: object is not a function
at new JS_Parse_Error
(/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:261:21)
at js_error
(/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272:15)
at croak
(/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:742:17)
at token_error
(/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:749:17)
at unexpected
(/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:755:17)
at
/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:829:33
at block_
(/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:999:32)
at
/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:978:43
at
/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:982:29
at
/home/reduz/bin/emsdk_portable/emscripten/tag-1.34.7/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:1112:43

I guess one of the calls to EM_ASM is the culprint, but I'm seriously
clueless. Here's code I used:

http://pastebin.com/XKbhcDZE

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417#issuecomment-138435013.

_OkamStudio_

[kripken]

For passing large samples to EM_ASM, passing a pointer is best. It becomes an integer in JS, and you can then do stuff like HEAPU8.subarray(ptr, ptr+len) to get a slice of memory, without copying, of uint8s from ptr to ptr+len.

[reduz]

ah fantastic great to know this! I suppose I also should expect it to be in
whatever endianness the host is running right?

On Tue, Sep 8, 2015 at 3:35 PM, Alon Zakai [email protected] wrote:

For passing large samples to EM_ASM, passing a pointer is best. It
becomes an integer in JS, and you can then do stuff like HEAPU8.subarray(ptr,
ptr+len) to get a slice of memory, without copying, of uint8s from ptr to
ptr+len.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417#issuecomment-138663510.

[kripken]

@reduz: Yes, JS typed arrays use the host endianness. In practice, all machines today running the web use little-endian so this is not much of an issue, and the standards bodies have sort of accepted that and not tried to define anything more portable.

@okamstudio: Thanks for the feedback on the EM_ASM docs, I added more details there.

[punto-]

How well does emscripten code run on the Wii U's browser? :p (Nintendo was
actually pushing for html5 games on it a while ago too)

On 8 September 2015 at 20:49, Alon Zakai [email protected] wrote:

@reduz https://github.com/reduz: Yes, JS typed arrays use the host
endianness. In practice, all machines today running the web use
little-endian so this is not much of an issue, and the standards bodies
have sort of accepted that and not tried to define anything more portable.

@okamstudio https://github.com/okamstudio: Thanks for the feedback on
the EM_ASM docs, I added more details there
http://kripken.github.io/emscripten-site/docs/porting/connecting_cpp_and_javascript/Interacting-with-code.html?highlight=interacting#interacting-with-code-call-javascript-from-native
.

—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/2417#issuecomment-138666714.

[kripken]

I suspect almost no code using typed arrays and WebGL, including emscripten, would work on that browser, if the CPU is big-endian. Emscripten in fact verifies during startup that the system is little-endian.

[kripken]

How are things going here? Anything I can help out with regarding optimizing usage of Emscripten?

[reduz]

This is almost done, need to fix a few issues before 3.0 is out..

[akien-mga]

I think we can close this issue now as Godot 3.0's WebAssembly/WebGL 2 export is starting to perform quite well :)

If there are issues where @eska014 (our Web export maintainer) needs to take on @kripken's very helpful offer for help, I'm sure he won't hesitate to ping or ask directly on Emscripten channels. Thanks again for your support :)