Three.js: Compress GLSL for faster transfer to GPU.

[Tostifrosti]

I have a question/feature-request about the optimalisation of shaders within ThreeJS. After a quick look, I've noticed that the shaders are build from string on runtime. After this build the shaders are not compressed or optimized before sending it to the GPU (only the minified version of ThreeJS does include some sort of minified versions of the shaders).

I'm curious if you guys have looked into some sort of optimizer like: https://github.com/aras-p/glsl-optimizer/.

As the blog stated is that the performance on older hardware could rapidly improve with this optimizer. I'm not sure if current hardware will automatically optimize the given shaders, so i'm very curious if you guys have already looked into this.

[donmccurdy]

On one hand, building the shader at runtime is itself a means of doing final optimization - we can remove dead code, for example. If the shader were generated by an offline optimizer, it would be mostly immutable once it reached the browser.

As that library is used for offline optimization I’m not sure we can use it directly here, but it would be neat to hear results of passing threejs shaders through that tool, and seeing if those changes are worth trying to implement in our own shader generation code.

[pailhead]

I don’t think this fits three as is. I don’t think you’d be running this at run time, ie it doesn’t work with how three dynamically assembles shaders on the fly.

[Tostifrosti]

Sorry for any misconception, but I was not referring to offline compression or during the assembly of shaders. The idea was to compress the shader before it is send to the GPU. For example the shader could be minified, shorten some names (everything except uniforms and api calls). Also deleting parts like:

#if 0 > 0
    uniform sampler2D directionalShadowMap[ 0 ];
    varying vec4 vDirectionalShadowCoord[ 0 ];
#endif
#if 0 > 0
    uniform sampler2D spotShadowMap[ 0 ];
    varying vec4 vSpotShadowCoord[ 0 ];
#endif
#if 0 > 0
    uniform sampler2D pointShadowMap[ 0 ];
    varying vec4 vPointShadowCoord[ 0 ];
#endif

(copied from compiled shader with firefox Shader Editor plugin) could end up in some increase of FPS.

[TimvanScherpenzeel]

I have made a wrapper called glsl-minifier in the past around the compiled ASM.js file from Joshua Koo's Emscripten port zz85/glsl-optimizer of the original aras-p/glsl-optimizer source. Might be useful for if you want to test. One big downside is that the project uses an outdated ASM.js file that I haven't been able to build from the original source.

I was planning to make a Webpack plugin to do the shader minification offline (only on release builds) but I haven't gotten around to it yet.

As for doing this in realtime, I don't see a real option using this ASM.js file as it is really quite slow.

[pailhead]

What if I don’t want to minify the shaders and have some compiler laying around? I can already do that, since I can just load any shader lazily. Glsl is glsl before or after this step. Threejs can load glsl. Why should it optimize glsl?

[mrdoob]

@Tostifrosti It would be great if you could gather some performance numbers of how long it takes to send the string to the gpu in different devices. I have the feeling this is would be a micro-optimisation.

[pailhead]

Actually this should be a good argument:

import MyGLSLOptimizer from 'some_GLSL_optimizer'

material.onBeforeCompile = shader => MyGLSLOptimizer.optimize(shader)

Or

const shaderMaterial = new ShaderMaterial()
optimizer.optimize(
  parseShader(shaderMaterial)
)

Really no need for three.js to do anything extra. You can do it, i can opt out, Jake can still make up his mind.

[Usnul]

I think the point of the optimizer is not in compression. I read about it a while ago, the tool is intended to improve run-time performance of a shader, and not to reduce the size. Size reduction may or may not manifest itself as a by-product.

[TimvanScherpenzeel]

@Usnul correct it is not so much about the actual file size of the shaders but rather function inlining, dead code removal, copy propagation, constant folding, constant propagation, arithmetic optimisations and so on. In my opinion it only really makes sense as a production build step (as it harms debuggability during development) for custom shaders when you are building an application instead of trying to do this at runtime.

#define SPREAD 8.00
#define MAX_DIR_LIGHTS 0
#define MAX_POINT_LIGHTS 0
#define MAX_SPOT_LIGHTS 0
#define MAX_HEMI_LIGHTS 0
#define MAX_SHADOWS 0
#define GAMMA_FACTOR 2

uniform mat4 viewMatrix;
uniform vec3 cameraPosition;

uniform vec2 resolution;
uniform float time;
uniform sampler2D texture;

void main() {
  vec2 uv = gl_FragCoord.xy / resolution.xy;

  float v = texture2D( texture, uv ).x;

  if (v == 1000.) discard;
  v = sqrt(v);

  gl_FragColor = vec4( vec3( 1. - v / SPREAD ), 1.0 );
}

Turns into

uniform highp vec2 resolution;
uniform sampler2D texture;

void main(){
  highp vec2 a;
  a=(gl_FragCoord.xy/resolution);

  lowp vec4 b;
  b=texture2D(texture,a);

  if((b.x==1000.0)){ 
    discard;
  }

  lowp vec4 c;
  c.w=1.0;
  c.xyz=vec3((1.0-(sqrt(b.x)/8.0)));

  gl_FragColor=c;
}

[pailhead]

@TimvanScherpenzeel

Would you consider using something like webpack or something that could transform your code, strip out the necessary things etc?

if(!development)
  material.onBeforeCompile = my_shader_optimizer

I find this very obfuscated:

  lowp vec4 c;
  c.w=1.0;
  c.xyz=vec3((1.0-(sqrt(b.x)/8.0)));

And i think given how three.js works, and is being used, it might cause a problem further down the line. Googling something like an error that three logs involving variable names and lines may actually yield results, i wonder if these variable names would persist between different shader iterations and their optimizations.

I still don't understand the suggestion, would you include the optimizer into three (like three.min.js), would you run it every time etc etc? Would just doing it yourself with a couple of lines suffice?

[mrdoob]

Maybe this could sit at the browser level?

[pailhead]

Maybe this could sit at the browser level?

If by that you mean - a page loads a three.js app and then you run the optimizer, that could be one way to go, but it may be slow?

I think this would be meant to be used more like:

npm run build

And then three builds with an optimized GLSL in ShaderChunk. But i think that would have an issue of limiting the optimization, as the optimization could apply better on the entire shader rather than just the snippets. I don't know how you could unroll every possible permutation of a shader that various ifdefs can make.

Also if something like this were to be done, there are probably a billion other things that would be more important. I think stripping the if (!target) console.warn(...) would be a part of a step like this. I could be wrong about all this but this is my interpretation of it.

[siamakmirzaie]

@pailhead if i am not mistaken, that's what unity does, unroll every possible permutation and run through the very same optimizer.

I remember a discussion, maybe it was a blog on unity, back in ~2010 that this would improve the performance as much as 30% on mobile phones with some tests available. This optimization is normally done on compile time by your driver, but mobile phones then didn't. I don't know whether the Opengl ES/WebGL mobile drivers support optimization by now or they still don't because it would be a standard limitation maybe.

ps. Found one of the blogs: https://aras-p.info/blog/2010/09/29/glsl-optimizer/

[pailhead]

Which again is what onBeforeCompile would do. This is actually seems like the only use case where that callback would be appropriately named :) since for the most part it's used as something else.

That is the only point where you know for sure what you've received the whole shader. If you just ran this on the chunks somehow i don't think it would give any performance gains, and to run it at runtime/load time whatever really depends on the user's needs and those are very diverse when working with something like three.js as opposed to unity.

This would probably be best as part of npm run build but it could also be done, in your app, at (load) runtime.

[Tostifrosti]

@pailhead You problably mean onAfterCompile (if the function even exists). The function onBeforeCompile will return a object that contains the vertex & fragment shader code that hasn't been parsed yet. This code will contain things like #include or #unroll_loop which we don't want when compressing/optimizing the GLSL shader.

Compressing the shader during npm run build would not work either because the shader needs to be parsed by threejs so the compression could remove dead code, replace defines with values, etc.. (which is only known during runtime). All uniforms will, of course, be untouched.

All materials needs to be recompiled anyway when something changes within the scene or to the material itself (aside from properties that are binded to uniforms). For example when you add a light in the scene at: L146. We can use this oppertunity to optimize the shader before it is send to the gpu.

Here would be a lovely spot to place the function onAfterCompile: L573

[Usnul]

If you go down this road - optimizing shader code, I would start thinking about having a compiler in place. Three.js already does some parsing of glsl and re-writing of code.

Having an optimization stage in the compiler would help produce better code from Node-based materials too.

Part of the reason I bring up a compiler is for the AST, having one would make working with shaders a lot more flexible, including a possibility of using some other language such as HLSL instead of GLSL as the main language in three.js, and then compiling that to glsl. Having a compiler would enable fairly easy implementation of JS shaders, making debugging a lot easier. Currently writing shaders that rely on multiple dynamic inputs is a pain, there's no easy way to look inside the program to see what's going on.

[pailhead]

You problably mean onAfterCompile (if the function even exists). The function onBeforeCompile

Three.js is known for many things, but not for giving it's variables names that make the most sense.

You're right, onAfterCompile does not exist, and i think it shouldn't since on after compile you've given your GLSL to the driver and i think it's making machine code, you cant do anything more to it.

onBeforeCompile is actually on_Waay_Before_WebGL_Compile_And_Before_Three_compile or just onBeforeParse. The reason why i was referring to it is that because onAfterCompile doesn't exist, you can make your own, infact onAfterParse by parsing the shader i onBeforeCompile yourself.

This code will contain things like #include or #unroll_loop which we don't want when

I'm confused by this. Why wouldn't you want an #include statement when you're optimizing? Every time i build three, it processes all my javascript include foo from 'foo and then optimizes them, removes dead code, minifies, mangles etc.

Why couldn't some kind of an optimizer process #include <foo> first and then optimize?

onBeforeCompile = shader => {
   processIncludes(shader)
   //no more includes
   processUnroll(shader)
   //no more unrolls
   optimize_with_some_optimizer(shader)
}

Is unroll_loop an internal three thing? I thought it was one of those pragmas that already does this stuff on the driver level.

Compressing the shader during npm run build would not work either because the shader needs to be parsed by threejs so the compression could remove dead code, replace defines with values, etc.. (which is only known during runtime). All uniforms will, of course, be untouched.

It does not. It's a string, thats being transformed to another string. The browser is not the only thing capable of doing this, running a script from npm is fine too. You can use a function like this:
https://github.com/pailhead/three-refactor-chunk-material/blob/651082071c0e4edb3a602f1b67ac7059d34e1135/src/utils.js#L6-L26

So you run something like this:

npm run unroll-three-shaders 

And it spits out a bunch of .vs .fs files, or maybe a single .js per material containing the shaders with no includes.

replace defines with values, etc.. (which is only known during runtime).

I think this happens every time you sent the shader to the gpu.

All materials needs to be recompiled anyway when something changes within the scene or to the material itself (aside from properties that are binded to uniforms). For example when you add a light in the scene at: L146. We can use this oppertunity to optimize the shader before it is send to the gpu.

I dont think that this is actually good. Like you said, games will make all the possible permutations, but you still want three to compile them on the fly.

I think that this mechanism is tricky to begin with because you have the how do i need to know if i have to call needsupdate after i've changed this particular property. Now on top of that, you want to add an optimization step every time that happens?

I don't have the time but i think it's worth testing how fast it is to parse the #include and how fast is it to optimize the entire three.js super shader. I imagine this would be orders of magnitude more.

For example when you add a light in the scene at: L146. We can use this oppertunity to optimize the shader before it is send to the gpu.

For example, if your app is a game, you would never ever add a light there. However if for some reason you absolutely had to, i don't think you'd want some kind of glsl optimizer to run in the middle of your player having a gunfight.

@usnul

why not do this offline?

Sorry if my tone is coming across weird, i just find the whole chunks and everything kinda frustrating, not because i don't like them but i think they're misunderstood :)

[Usnul]

@pailhead

why not do this offline?
Sorry if my tone is coming across weird, i just find the whole chunks and everything kinda frustrating, not because i don't like them but i think they're misunderstood :)

No problem at all, it's a good question. Here's my line of thinking:

three.js messes with the shader code in the following ways:

• onBeforeCompile
• #include replacement (it's not standard glsl, three.js uses pattern matching to find these and inline ShaderChunk there)
• #pragma unroll_loop
• ShaderMaterial - whatever you supply gets decorated with additional pieces of glsl
• NodeMaterial - you might not want to consider this one as it's still not ready

With all of these - it's pretty clear that three.js transforms shader code, but it does so in an ad-hoc way, there is no unified approach, every one of these features works entirely separately.

If there was a clear place to insert AST rewrite rules - you could avoid avoid onBeforeCompile entirely, preprocessor commands #pragma and #include would not have to rely on custom parsing either.

ShaderMaterial could inspect your AST and patch it as necessary instead of appending strings.
If you offer NodeMaterial eventually - that's basically a killer app for having an AST, otherwise you end up working on string templates, which is a lot more error-prone and... well.. dumb. I love string templates as much as the next guy, but they don't compare to working on an actual syntax tree both in terms of ease of use and performance.

why not do this offline?

I think for a well-organized project, you would do just that. But if three.js positions itself as a 3d graphics framework and not just a visualization library - having code comprehension (i.e. a parser) would enable additional use-cases as well as simplifying some of the aforementioned existing ones. From parsed representation to compilation is a rather small step after that.

[Tostifrosti]

@Usnul You make some strong points on implementation and the ideology behind ThreeJS. I am still confinced however that an optimalisation could still be made during the parsing process of the shader.

The reason for this is the use case of my project: visualise products. My project loads a ton of geometry/material to visualise a product (like a car). The product can be changed by enabling/disabling some materials/geometry (like color of a car). My project already have some optimalisations in place like sharing materials and enable/disable some settings based on platform.

ThreeJS does an amazing job by visualising my products with great detail and fps (even on mobile devices). Still... I can't help to think we can push the limit by optimizing the shader by: deleting dead code, replacing values, inlining, etc.. so the GPU has an easier task of running the shader (every shader * 60 fps).

The blog I refered to in my first comment has proof that optimizing the shader will increase performance on mobile devices. Even Unity today will optimize homemade shaders by making multiple variants of the shader.

As my first question stated, I was just curious if you guys had any research on this topic. After reading all the comments I am convinced that this area could still be explored. I will try to conduct a research on this topic in the future and post results in this thread or as a new issue.

Thank you guys for your time!
@donmccurdy @Usnul @pailhead @siamakmirzaie @TimvanScherpenzeel @mrdoob