Three.js: Enhancement Proposal: Wrap OBJLoader in web worker

[kaisalmen]

Description of the enhancement

I like to propose several enhancements to the existing OBJLoader. The aim is to wrap the existing OBJLoader into a new web worker. I have created a fully working prototype based on the latest dev code: See my fork of three.js The goal is to integrate large OBJ files (>100MB, >1 million triangles) into a scene while the user is able to interact with the scene.

Proposed changes to OBJLoader:

• Parse should also be able handle an arraybuffer (configurable; OBJ content as arraybuffer from XHRLoader or for example from external after unzipping).
• Propose to break down the parse function into logical blocks/methods: preparation, processing of single line and finalization; mesh material and object creation will be moved to own methods (allow override in web worker).

• Add inline mesh processing (configurable): A mesh will be added to the container as soon as it is available. Hook for behavior override by the web worker (see below). It reduces memory consumption during parsing as objects are not stored in intermediate format in '_createParserState'.

  Web worker OBJLoader wrapper (WWOBJLoader):

• A new web worker extends the OBJLoader and overrides the extracted method responsible for building a single mesh. Then it can pass back FloatArrays and material information to a 'front end' (see below).

• Due to the adjustments to the existing OBJLoader the web worker code is very complicated.

  Web worker Front End:

• Class to use in order to be able to use the new functionality of the WWOBJLoader

• It realizes the communication and workflow with the web worker. It is responsible for the integration of the meshes delivered by the web worker into the scene.
• It can be used with both file locations and already loaded files (arraybuffer or string) and then pass the OBJ data to the web worker (arraybuffer is especially useful when OBJ file was for example stored in zip as it may already available as arraybuffer).

• Handles MTL loading via MTLLoader (before anything else is done). Materials are passed without textures to web worker which re-creates the MaterialCreator. Materials are required for the OBJLoader to work correctly, but texture data is not. This avoids passing texture data back and forth.

  Questions:

• Are the proposed changes to OBJLoader acceptable?

• I assume it is not desired to have extremely large OBJ files inside the repository. What is your opinion about this?

• I have not yet created an example in the usual single page format (only the two mentioned below). I was thinking about building astroids in Blender and then export them as OBJ (many objects; 10-20MB file size). This should be sufficient to show the possible interactivity.

  Prototype examples:

• Two examples exist on my website using WWOBJLoader:

  • WWOBJLoader with model from webgl_loader_obj_mtl
  • WWOBJLoader with zipped 150MB, 1.7 triangles model
    

    Regressions?

• Existing OBJLoader examples still work with the proposed changes implemented. I was not able to check the VR examples, though.

  Further thoughts...

• Integration of OBJLoader changes is potentially independent of other changes (some git magic will separate this...) and could be merged alone first.

Thanks in advance for providing feedback. This could be my first contribution to three.js. ;-)

Kai

Three.js version

• [x] Dev

• [x] r81

  Browser

• [x] All of them

  OS

• [x] All of them

[makc]

OBJLoader: 37363.55ms

that's impresive ) it would be nice if all loaders could be made like that

[Usnul]

it would be quite simple to parse geometry and construct it inside a worker. Especially a buffered geometry, as it could be stuffed into a transferrable - which eliminates expensive deep copy. Materials I'm not so sure about.

[jonnenauha]

We have discussed this many times with @mrdoob in other issues. You should read through https://github.com/mrdoob/three.js/issues/8704

As you can see there, @mrdoob agrees that we could create OBJLoader2. I think for a such big change + refactoring like this it would be better than trying to keep OBJLoader backwards compatible. Easier to start fresh, we can still reuse code from the current loader where it makes sense.

This would leave users that are happy with the current loader in peace, working code. We could log a deprecation warning once the OBJLoader2 has feature parity and we can show it is faster without web worker mode. I think it should support main thread mode as well, an auto fallback for browsers that don't support them and config option if you want to force main thread. For simple models I'm sure the bootstrap time of the worker is gonna be a net loss for total load time if the models are simple.

@mrdoob has also wanted some cleanup, refactoring and simplification for OBJ loader so that it would be easier to approach for new devs. My initial hunch is that this new worker based setup would be a lot more complicated for "noobs" to jump into the development :) At least harder to wrap the whole thing into your head, but still probably as easy to find the code part that needs fixes. I'm all for web workers (as I comment in the issue).

I like your ideas and effort. The demos work great, I did not look at the code closer yet (I will). I especially like the streaming of each submesh when its done, great. I think this should also be configurable so you can turn it off, or maybe even default off. Most users probably would not like to render partial meshes to the end user.

I need to read you code, if you already have this, but there needs to be a configurable worker pool that distributes parallel loads into many workers. Also its important to keep workers alive for a while after its load is done, creating a new worker and loading three.js into it expensive from my experience, sometimes hundreds of msecs (even if the file is in browser cache). I have implemented such a system and would be able to help here.

How about you create a separate repo that does not have the full three.js in it. Just your new loader and its needed bits. You could just pull latest three.js as from npm and make demo/test pages that run it with latest release. I could jump in as a contributor and help you out.

Once its fleshed out we could import it as OBJLoader2 into three.js repo. In fact, users who really need this could get the loader from the separate repo right away to test and give feedback. As the loader does not mod any core parts, a separate repo would be better imo. This would give us ability for our own release cycle, issue tracker etc.

it would be quite simple to parse geometry and construct it inside a worker. Especially a buffered geometry, as it could be stuffed into a transferrable - which eliminates expensive deep copy. Materials I'm not so sure about.

Yes the no copy transfer/transferable is a must for this kind of loader. I have implemented this kind of worker before, imo its best to submit each attribute separately and construct a new buffer geometry in main thread (just add the received attrib/buffer as is).

I would probably tranfer material metadata as an object to the main thread and create materials there. Though you could just serialize the worker created material as JSON and deserialize it back in the receiving end. I believe toObject is a standard API for pretty much everything, but would need to see if it is smart enough to omit default properties to reduce the object size. If it does serialize everything, submitting a simpler metadata would be better.

[kaisalmen]

Thank you all for providing feedback. Give me some time to process the information. I wasn't aware about #8704. It seems that I didn't check enough issue history. A thorough answer will follow.

[Usnul]

Workers are not free, if you want them as part of the core library - you need management code for it at some point.

• do we have enough workers?
• should we start a new one?
• should we destroy a worker, if it's no longer needed?

Workers currently don't allow live-code transfer, so you have to either load a separate .js files inside a worker or pass all relevant code with dependencies as a string into the worker and then use "new Function(codeString)". You could also use a URL blob (which is what will be used to create the worker initially, i suspect).

Spawning too many workers, or keeping them alive without use is probably not a good idea - there is a penalty depending on browser/device. Just something to think about, more long-term.

[kaisalmen]

@jonnenauha, yes keeping backward compatibility is important and I agree that it is a good idea to create an OBJLoader2 and also to do it in an independent repository as you suggested:
I have created a repository for OBJLoader2 prototyping. It will be filled with content/examples in the upcoming days during my spare time.

What I already achieved with my modified version of the OBJLoader is that it can be used independent of a wrapping web worker. You are still able to use it without touching any code. The WWOBJLoader is just an extension. I needed to move code into new logical blocks which in the end is a fairly heavy modification to the existing OBJLoader.

With OBJLoader2 we are able to implement more fundamental changes to the existing loader. If I am not mistaken the processing of the OBJ file content is fairly robust now as the code has received various bug fixes. So, the core processing code should stay, but we could think about optimizations (data structures, etc.).

Objectives for OBJLoader2:

• Support different input types for parsing:
  
  
  
  • string
  
  
  • arraybuffer
  
  
• Low memory usage:
  
  
  
  • Process file content serially and per line
  
  
  • Minimize internal data-structure/object usage
  
  
• Support inline-processing:
  
  
  
  • Add mesh and material to scene when it becomes available
  
  
  • Only intermediate data of current input mesh shall be kept in memory
  
  
• Provide hooks for web worker extension
  
  
  
  • Mesh creation shall be isolated within a function
  
  
  • Material creation shall be isolated within a function
  
  

Objectives for web worker wrapper/worker controller:

• Establish lifecycle:
  
  
  
  • init (controller feeds worker)
  
  
  • send material information (bi-directional)
  
  
  • parse (pass-back buffers)
  
  
• Use transferables wherever possible:
  
  
  
  • Construct BufferGeometry on main thread (worker controller)
  
  
  • Texture loading must be done on main thread (worker controller). As I understand it the TextureLoader needs document access and web workers are unable to.
  
  
• Keep the worker as generic as possible:
  
  
  
  • Already now the WWOBJLoader is not very OBJLoader specific
  
  
  • Porting the web worker to other loaders should be straightforward if mesh and material hooks are in loader
  
  
• Think about worker management (@Usnul):
  
  
  
  • worker pool (construction cost, device limitations, etc.)
  
  

That's it for now. I have added the objectives to the readme of the bespoke repository.
Should we continue the discussion here or move to a new issue in the new repository and just add status updates here?

[kaisalmen]

The simple example using male02.obj is now ported and available in the new repository.

[kaisalmen]

It took a while, but I have first results and want to share how far I have come. Feedback is very welcome.

2016-11-06: Status update

• New OBJLoader has almost reached feature parity with the existing OBJLoader
• Features still missing in comparison with existing OBJLoader:
  
  
  
  • Support for Line parsing/geometry generation is missing
  
  
  • Multi-Materials are not used, instead a mesh is created per object/group/material designation
  
  
• New features:
  
  
  
  • Flag "createObjectPerSmoothingGroup" will enforce mesh creation per object/group/material/smoothingGroup (default false; as it may lead to thousands of meshes, but useful for experiments)
  
  
  • Load from string or arraybuffer
  
  
  • Hook for web worker extension "ExtendableMeshCreator" exists already. In non-extended loader it creates meshes and attaches it to the scenegraph group.
  
  
• Web worker work has not started, yet, but code base exists from previous proposal!

Some thoughts on the code:

Approach is as object oriented as possible: Parsing, raw object creation/data transformation and mesh creation have been encapsulated in classes.

Parsing is done by OBJCodeParser. It processes byte by byte independent of text or arraybuffer input. Chars are transformed to char codes. Differnet line parsers (vertex,normal,uv,face,strings) are responsible for delivering the data retrieved from a single line to the RawObjectBuilder.

The RawObjectBuilder stores raw vertex, normal and uv information and builds output arrays on-the-fly depending on the delivered face information. One input geometry may lead to various output geometry as a raw output arrays are stored currently stored by group and material.

Once a new object is detected from the input, new meshes are created by the ExtendableMeshCreator and the RawObjectBuilder is reset. This is then just a for-loop over the raw objects stored by group_material index.

Memory Consumption

Only 60% of the original at peek (150 MB input model) has a peak at approx. 800MB whereas the existing has a peak at approx. 1300MB in Chrome.

Performance

So far, I only ran desktop tests: Firefox is generally faster than Chrome (~125%). 150MB model is loaded in ~6.4 seconds in Firefox and ~8 seconds in Chrome. Existing OBJLoader loader takes ~5.1s in Firefox and ~5.3s in Chrome.
Tests were performed on: Core i7-6700, 32GB DDR4-2133, 960GTX 4GB, Windows 10 14393.351, Firefox 49 and Chrome Canary 56.
Biggest room for improvement: Assembling a single line and then using a regex to divide it, seems to be faster than evaluating every byte and drawing conclusions. This is not what I expected. I will write a second OBJCodeParser that works differently. From my point of view OO approach is not hindering performance.

Examples:

Existing OBJLoader

New OBJLoader

Larger model not in the prototype repository (27MB zip):
Compressed 150MB Model

[Usnul]

Sounds really good! Regarding OO approach, one thing to be careful about is garbage collection, this is where performance can take a sharp hit. 800MB memory usage is curious, it's good that it takes less, but would be interesting to know how it scales (e.g. 800MB = X*a + b, what are X,a,b?).

[kaisalmen]

@Usnul: Hope this answer makes sense. :-) During parsing the max. amount of heap is used when a the biggest object is transformed (2140212 vertices in the benchmark example) from input to raw mesh data.

b: Data delivered by FileLoader (previous XHRLoader) is base memory footprint and only exists while FileLoader is kept in memory (subject to GC afterwards). My observation: Chrome does not flush this data immediately. Changing tabs sometimes enforces this.

Size of single input object from OBJ file (Varying X) + Size of raw mesh data (1.x * input data) (a):
Varying X: Input Data is immediately disregarded as new RawObjectBuilder is used (subject to GC).
a0 to aN: Output data is bound to the BufferGeometry. This size increases from start to end of parsing (not on heap as you can see below).

This is what happens in Chrome:

[kaisalmen]

With the updated I just pushed to WWOBJLoader Prototyping speed is now similar to the existing implementation: Both Firefox and Chrome parse the 150MB model under 5 seconds on my dev machine (100-200ms difference to existing OBJLoader)!

Update 2016-11-11:
Ok, I think, it will now become very hard to improve parsing speed further. New OBJLoader is on average now slightly faster then the existing implementation and this was already really fast!
I took the fastest parse time out of three runs each:
New OBJLoader: 3835.16ms (Firefox 64-bit 49.0.2)
New OBJLoader: 4251.402ms (Chrome 64-bit 56.0.2916)

Existing OBJLoader: 4102.86ms (Firefox 64-bit 49.0.2)
Existing OBJLoader: 4429.495ms (Chrome 64-bit 56.0.2916)

Btw, these things improved performance most:

• Instead of providing each byte to the different parser objects, I store a line in an array and give it to the LineParser.
• Switch case is only used for detecting line (start characters) and not entered for every single byte.
• Line and buffer arrays are not re-initialized. An index is always used and only this is reset. Yes, they grow, but to a predictable size.

[jonnenauha]

Cool stuff, good progress, I'll take a peak at the codebase. At this point I'm mostly interested in the new parser code. The web worker threading is ofc a cool addition.

Edit: The multi-material feature is quite a big one. I'll see if I can help out with that. If I make a PR can I push new example assets? I could also make some kind of unit test system with grunt that would run all the files and verify the thing is still working :) Quite useful with bigger changes.

[jonnenauha]

What is the latest code I should read, ObjLoader3.js? The WWCommons.js still pulls in ObjLoader2?

The structure of the repo is kind of confusing, some suggestions.

1. Latest three.js release should be fetched with npm or bower.
2. New source code this library provides should be in /src. ObjLoader2/3 are just two different versions of the same thing? I think the final thing should be called ObjLoader2. When you are done experimenting there should be one file imo.
3. Examples in /examples could be splitted into /test and /assets. The old obj/mtl loaders could be in /test/three.js.old or something like that. Not with /src.

Then implement tests you can run with shell eg. grunt test that executes .js tests that report the load times etc. for both old and new loader.

grunt build should produce a /build dir that contains one file that has all the things this library needs. Both human readable and minified versions.

Finally repo/library also needs a better/catchier name. "Hey dude are you using the new WWOBJLoader?" does not exactly roll off the tongue easily :) ObjLoader2 is kind of boring too though, but conveys the message clearly to three.js end users. The web worker part does not need to be in the name, its just one features, as you have improved the main thread performance as well (at least what I'm reading here).

Edit: If you agree on the points I could send a PR to you with at least some of the changes. The repo is still small so it would be easy to do at this point.

[kaisalmen]

Ok, I took the three.js repo layout as blueprint.
OBJLoader3 is the one correct one! OBJLoader2 was the original proposal (starting point for this issue).
Just pushed an update: Multi-material code is there, but there are some index vertex index issue with the MultiMaterial.

[jonnenauha]

Yeh. I think what at the end of the day goes to three.js is one or a few new files, if its decided to be merged there. Those should be the non-minified build artifacts from this lib repo. So the structure does not necessarily need to be similar. Why loaders are in examples in three.js is because they are not bundled in the build, but separately provided examples (even if they are the defacto loaders most users use).

[jonnenauha]

With quick testing I'm not seeing the perf gains. For models/obj/cerberus/Cerberus.obj three.js master OBJLoader ~100-150msec, this lib ~150-250 msec. I think the perf can be improved but I need to take a closer look. Does your figure actually parse/process the file twice? I saw some funcs that counted things like objects/materials prior to the parsing?

General observations: You have prepared all the classes to be very configurable. Most of the operational modes can be turned on/off or you can provide you own custom functions. To the point where you are wrapping the default built in function call for fetching a single byte out of the array buffer :) This might be a bit overkill, as its done thousands and thousands of times inside the parser.parse. We need to look at the hot path and make it fast.

You are also newing up a lot of "complex" objects in the start of each parse. This is probably fine, as they encapsulate state. In the parsers you are again mapping/calling lots of functions to get back to the main parser.

var parsers = {
    void: new LineParserBase( 'void' ),
    mtllib:  new LineParserStringSpace( 'mtllib', 'pushMtllib' ),
    vertices: new LineParserVertex( 'v', 'pushVertex' ),
    normals:  new LineParserVertex( 'vn', 'pushNormal' ),
    uvs:  new LineParserUv(),
    objects:  new LineParserStringSpace( 'o', 'pushObject' ),
    groups: new LineParserStringSpace( 'g', 'pushGroup' ),
    usemtls:  new LineParserStringSpace( 'usemtl', 'pushMtl' ),
    faces:  new LineParserFace(),
    lines:  new LineParserLine(),
    smoothingGroups:  new LineParserStringSpace( 's', 'pushSmoothingGroup' ),
    current: null
};

....

function LineParserBase( name, robRefFunction  ) {
    this.name = name;
    this.robRefFunction = robRefFunction;
}

function LineParserVertex( name, robRefFunction ) {
    LineParserBase.call( this, name, robRefFunction );
    this.buffer = new Array( 3 );
    this.bufferIndex = 0;
}

... later on results

robRef[ this.robRefFunction ]( this.buffer );

I think at the end of the day, as discussed elsewhere, this boils down to

• iterating arraybuffer single byte at a time, until a line is ready, then processing that line.
• versus letting regexp/string splitting do that for us and processing the lines.

The mem footprint of arraybuffer should be better as we have more control, but the speed does not seem to be there at least for this low/moderate sized mesh. Though I would guess arraybuffer parsing is faster, its just some other stuff (maybe points above) that is slower.

Simplifying the whole chain a bit and not wrapping functions over single line execs might belp. I cant really see clearly where the time is spent with chromes profiling tools at this point. Need more investigation and reading the codebase.

Edit: From my point of view, at this moment this codebase is more complex than what it is trying to replace. Not saying its a bad thing though, but it was one of mrdoobs points to make the library more approachable, which the regexp soup certainly is not either :) I would maybe start by splitting each of the classes into separate files, easing developers to first of all read the whole thing, instead of jumping inside a one big file. Could use the same module require that three.js itself is now using and the same tools to produce the final build.

[Itee]

Open parenthesis (

Why loaders are in examples in three.js is because they are not bundled in the build, but separately provided examples (even if they are the defacto loaders most users use).

@jonnenauha What think mrdoob and other "mains" contributor of three.js about keep really useful stuff in example directory ?

Due to es6 module and rollup with threeshaking only required stuff will be bundle, so it's (in my view) a non sens to keep this stuff out of the lib... If you have some ref issue about that, i will read them with attention, thank.

) Close parenthesis

@kaisalmen Really good job ! I will forked your repo as soon as possible to watch in deeper the added stuff.
I think due to "imminent" es6 refactoring of three.js may you should consider to make an ancestor class where all loader will could inherit from, in view to inherit webworker stuff easily on every other loader.

[kaisalmen]

Good evening, sorry, I was not able to answer earlier.

@jonnenauha, thanks for your comments. You pointed at one thing I did not focus on during development so far: Initialization is taking quite some time and therefore the parsing time for small models goes up. I did the same test you performed and I can confirm the results. I focused on big models files where the time for init phase becomes almost negligible and where overall parsing and mesh creation speed is as described before.

The various extension of LineParserBase seem over-engineered and should be reduced in complexity. I wanted to build small logical blocks that are easy to understand and maintain, but I guess the result is not what I wanted to achieve in the first place. I agree, re-work is required here!

But, this is only one aspect of new the loader and unfortunately you did not provide feedback on the other parts (2 and 3):

1. Input Parsing (OBJCodeParser) creates a
2. "normalized" object representation (RawObjectBuilder) that is
3. used to build the meshes (ExtendableMeshCreator)
   The "normalized" representation of the parsed data in RawObjectBuilder and indexing of geometric data according object/group/material/smoothing depending on the configuration allows creation of meshes and material become fairly easy in ExtendableMeshCreator. Just by changing the way the index is created (e.g. treatment of smoothing groups) the ExtendableMeshCreator receives different input. Because the contract between RawObjectBuilder and ExtendableMeshCreator is so simple extending it for the web worker is also straight forward.
   
   Points 2 and 3 are a lot easier to understand in the new implementation and fairly confusing in the existing one. I personally think these parts are easier to understand for other/new developers. Part 1 did not improve the situation, so far. Sorry for that! ;-)
   
   Worklist for me:
4. Improve the parsing (point 1) with focus on speed for small and large OBJ files.
5. Consider splitting code to different files, because:
   
   
   
   • web worker is then only dependent on Cache, LoadingManager and FileLoader (as far as I can see now)
   
   
   • Give ideas for general web worker approach for other loaders
   
   
6. Further improve documentation
7. Work on repository structure and provide easier access to tests / provide evidence of performance

P.s.: Screenshot: I did a page reload when I a captured the timeline with Chrome that's why things seem to appear twice.

[kaisalmen]

@jonnenauha I have overlooked some of your edits:

Edit: If you agree on the points I could send a PR to you with at least some of the changes. The repo is still small so it would be easy to do at this point.

Yes, this sounds great and I would appreciate it.

[kaisalmen]

Ok, the missing features have been implemented, but I face an issue with MultiMaterial I don't understand. @jonnenauha I know this is not a help forum, but eventually you have an idea. I implemented it very similar to you. The BufferGeometry vertex, normal and uv arrays are filled with multiple BufferGeometry.set instead of wrapping an existing array:

Only the first materialIndex is used when I set BufferGeometry.addGroup( start, count, materialIndex ). In my test a MultiMaterial with three materials is created, but all three vertex groups that are defined with addGroup use the same material (0). If I set the materialIndex in addGroup to fixed 1 or 2 for all vertex groups then all faces have material from index 1 or 2. So, MultiMaterial is defined correctly, but somehow BufferGeometry ignores the materialIndex. This is the code:
OBJLoader3.js ll. 1088-1178
Any ideas?

Some insights on how the obj data is processed. "Female02.obj" defines the first object like this (for now the code treats all 'g' as one object when useMultiMaterials is set):

g groupA
usemtl matA
s 1
f many lines...
g groupB
usemtl matB
s 2
f many lines...
s off
f one line...
s 2
f many lines...
s off
f one line...
s 2
f many lines...

will result be internally stored as (with vertices, normals and uvs ordered accordingly):

groupA|mat_A|s1 (matA with smooth shading)
groupB|mat_B|s2 (matB with smooth shading)
groupB|mat_B|s0 (cloned matB with flat shading due to 0/off)

If useMultiMaterials is true then one Buffergeometry with MultiMaterial and corresponding groups is created. If useMultiMaterials is false then three BufferGeometry with one Material each is created.

[kaisalmen]

First preview of WWOBJLoader+OBJLoader3 which allows to load multiple big OBJ files (zipped) is available here:
WWOBJLoader Testbed
Enjoy, but beware, if you load all available files, your GPU needs approx. 800MB and the browser will require more than 1.2GB of memory.

Some issues around materials are not yet resolved and the parser code is still untouched. Good news is that when OBJLoader3 is reused for loading by WWOBJLoader the parsing is faster (no new heavy init-costs). WWOBJLoader and its front-end need some work as well.
Getting there... :-)

[kaisalmen]

I have completed the parser rework. The parser code is a lot simpler and easier to understand, I think. Performance is there for small and large models. A Firefox 50.0.1 (64bit) fresh browser instance parses the 150MB model in 3.2 seconds:

Global output object count: 2127
parseArrayBuffer: 3216.56ms

Chrome is a little slower (57.0.2936.0 canary (64-bit):

Global output object count: 2127
parseArrayBuffer: 4088.800ms

Some random thoughts:

• I turned some circles to understand browser performance differences and Javascript performance problems in general.
• Both arraybuffer and text can be parsed.
• Code documentation is better already.
• Face N-Gons are not supported (this gave me some headaches and I changed the parsing approach), but it was not supported by old parser either. Triangular and quad faces are fully supported.
• Re-usage of OBJLoader like WWOBJLoader does is not an issue. I took care in resource clean-up and re-validation of the loader status and all involved objects

Next:

• Edit: Resolved: Multi-Material issues must be resolved
• Repository structure and tests
• Life-cycle of WWOBJLoader
• Split OBJLoader3 into multiple files (aim: worker without three.js import)

[kaisalmen]

@jonnenauha: I have adjusted the repository structure
Edit 2016-12-01:

• Just solved the multi-material issue: Vertex group start and offset were not correctly calculated

Edit 2016-12-02:

• WWOBJLoader and OBJLoader2 are both using MultiMaterial. Simplified code.
• Split OBJLoader2 into multiple files: Extracted OBJParser.js and OBJMeshCreator.js
• WWOBJLoader is independent of three.js. It only requires OBJParser.js

Edit 2016-12-04:

• OBJParser.js: Bugfix: group definition (g name) in OBJ file did not lead to new object creation

Edit 2016-12-11:

• Ongoing Created Branch WebWorkerLifecycle: Extracted WWProxyBase from WWOBJLoaderProxy (former WWOBJLoaderFrontEnd). Simplified life-cycle. Created WWManager that is able to create and run arbitrary WWProxyBase just by passing parameters. Target: WWManager gets a run-instructions-pipeline and spawans a configurable amount of web workers of a specific type (currently there is only WWOBJLoaderProxy). New demo/test will demonstrate this.

[kaisalmen]

The fun begins:

Web Worker OBJ Parallels Demo

• Configure a load queue from 1 to 1024 of random objects (female02, male02, vive-controller, cerberus and waltHead)
• Use 1 to 16 workers
• Interact with the scene while objects are loaded

Edit 2016-12-16:

• Renamed WWManager to WWDirector and simplified internal web worker handling and execution
• Fixed some bugs and performed additional clean-up
• Used mesh loaded callback to make things more colorful :-)
  

[kaisalmen]

Hi all,
I didn't have much time to work on this the last week, but I fixed some bugs on the demo and the web worker director class:
Web Worker OBJ Parallels Demo

Now, you can see the completion status of every web worker used:

I think, the code is feature complete and I would like to receive some feedback from you.

Any feedback on the code and the demos is very welcome!

Edit 2016-12-27: As promised some words providing an overview

In contrast to the existing OBJLoader the new OBJLoader2 consists of three pieces:

• OBJLoader2: Is the class to interact with for setting up, for loading data from a given file or for directly forwarding data to the real parser
• OBJParser: Is invoked by OBJLoader2 to parse the data and transform it into a "raw" representation
• OBJMeshCreator: Builds meshes from the "raw" representation that can be incorporared into the scenegraph.

What is the reason for separation?

The loader should be easily usable within a web worker. But each web worker has its own scope which means any imported code needs to be re-loaded and some things cannot be accessed (e.g. DOM). The aim is to be able to enwrap the parser with two different cloaks:

1. Standard direct usage
2. Embedded within a web worker

As OBJParser is independent of any other code piece of three.js or any other library, the surrounding code either needs to directly do the required integration (OBJLoader2 and OBJMeshCreator) or WWOBJLoader and the communication and data proxy (WWOBJLoaderProxy) ensure it. WWOBJLoaderProxy basically provides the same functionality as OBJLoader2 and OBJMeshCreator, but the work is done by the web worker.

WWOBJLoaderProxy is extened from WWLoaderProxyBase. The base defines the plan for usage of the proxy. One idea is to build other proxies for other web worker based loaders and the other idea is automation and orchestration.

Directing the symphony

WWDirector is introduced to ease usage of multiple WWOBJLoaderProxy. It is able to create a configurable amount of loader proxies that extend WWLoaderProxyBase via reflection just by providing parameters. An instruction queue is fed and all workers created will work to deplete it once they have been started. The usage of WWDirector is not required.

Parser POIs

The parser and mesh creation functions have reached full feature parity with the existing OBJ loader. These are some interesting POIs:

• Per default OBJLoader2 parse method requires arraybuffer as input. A fallback method for parsing text directly still exists, but it is approx. 15-20 pecent slower
• Face N-Gons are not supported identically to the old parser
• Direct re-usage of all involved classes is fully supported. I took care in resource clean-up and re-validation of status on all involved objects
• "o name" (object), "g name" (group) and new vertex definition without any other declaration lead to new object creation
• Multi-Materials are created when needed
• Flat smoothing defined by "s 0" or "s off" is supported and Multi-Material is created when one object/group defines both smoothing groups equal and not equal to zero.

Improvements

• Objects are streamed to the scene when WWOBJLoaderProxy is used. Add-only-when-fully-loaded should be added
• Check need for documentation improvement
• Test automation with focus on batch execution of tests for retrival of more robust performance numbers

Examples:

Web Worker OBJ Parallels Demo
WWOBJLoader
OBJLoader2
Original OBJLoader

Larger models not in the prototype repository:
Compressed PTV1 Model (150MB)
Compressed Sink Model (178MB)
Compressed Oven Model (150MB)

[jonnenauha]

Demo is very cool, great progress :)

@mrdoob should we start considering at some point to copy this new ObjLoader2 into three.js. I like that it is in its own repo, but it would be easier to discover from this repo. Could we still dev this in the separate repo, take issues there and at times copy the files as a pull request to three.js?

I don't really know how we should go forward with this. Should it be left as its own repo, have its own issues etc. I mean the loader is not part of three.js core, but it would be shipped as external files that would be handy for devs. Also being ObjLoader2 it won't break existing users, people can port to it when they like with quite minimal changes, a bit more changes if they wish to utilize the web-worker.

git submodules are a mess and a hassle for devs. I would not take that route myself.

[kaisalmen]

I would not go for git submodules either.
What about I create an npm package?
@jonnenauha do you see need for renaming things?
I have updated the previous post with the promised overview.

[jonnenauha]

So that we don't break code I would probably rename the JS object from THREE.ObjLoader to THREE.ObjLoader2. Keeping it the same name as the current loader could also be considered if the entire v1 API can be found from the v2 loader. At the end of the day the user including ObjLoader2.js is a clear decision to use it, but if it needs porting the app code to new funcs etc. it might be best to rename to not throw off people?

I think the best route to intergrate this library into three.js is to produce a single file, instead of multiple that need the correct load order. This can be done easily with gulp/grunt/whatever task runner in your repo. eg. grunt build would produce ./build/ObjLoader2.js without minification and ./build/ObjLoader2.min.js. The non minified version could then just be copied to three.js as a single file. And users that prefer using your repo (eg. want to update to latest release before updated to main three.js repo) they can pick it up from there or just use npm to fetch the library to their application.

I would not commit the build files into the repo, like three.js does. I think they are committed for convenience in three.js, but more convinient would be to just make a github release on each release tag. But this is a personal choice you can make, I just don't like the idea of build artifacts in the repo that can be produced with scripts.

So the build exec would concat these https://github.com/kaisalmen/WWOBJLoader/blob/master/test/webgl_loader_objloader2_direct.html#L93-L95 so users dont have to include lot of files in the right order.

For the webworker stuff you could make a second build output ObjLoader2.WW.js that concats https://github.com/kaisalmen/WWOBJLoader/blob/master/test/webgl_loader_ww_parallels.html#L91-L93 and whatever else is needed (probably needs ObjLoader.js stuff as well).

I can make the build scripts with gulp (seems to be preferred nowadays from grunt). I believe you can also instruct npm to run the build when the package is fetched (not sure, if not maybe best to put the build files in the repo after all?).

To summarise my thoughts

• Rename the used namespace to ObjLoader2
• Make a build task that will concat the library into a single file in the correct order. I would do one for non webworker JS code and one if you select to use the new webworker stuff.

Once the build step is there, we could also enable travis-ci to run the build and run tests the build against files in the repo. This can be added later.

[jonnenauha]

Just realized you import the parser in the script that gets loaded into the webworker https://github.com/kaisalmen/WWOBJLoader/blob/master/src/loaders/WWOBJLoader.js#L11

This should work ok but maybe we could also make the build to copy the parser file instead of import. This makes one network request go away and people wont need to hassle with paths in production. I believe ./my.js will be resolved from the page/path where you are, not where the script was imported from (eg. CDN)? But if that works with your example pages it probably is then resolved from the ww script ref :P

[kaisalmen]

Basically, I agree on the plan. Just, the web worker stuff needs to be split into to files as the web worker code must be isolated if I am not mistaken. You could stringify the complete web worker code, include it in a another file and do some magic, but I don't want to got that way for multiple hundred lines of code.

I am not familiar with Gulp, yet, but I will dig into it and make it work. :-) As you may have guessed by looking at the code my main language is Java. I am diving into the JavaScript world during my spare time for now. Build-automation is very well known to me, I am just used to different recipes and ingredients. ;-)

Anyway, I will start with renaming. Time may be a little short the next couple of days, but I will keep you posted.

[Cobertos]

On the topic of having to separate the worker into a separate file...

There's a technique I saw on MDN, Embedded Web Workers. Looks like you can put the code in a script tag (with a non-executing type, probably best to use an x- type to be actually valid like shaders do), make a Blob with the correct MIME type, and then use URL.createObjectURL to create a data URL from the blob, which can then be fed to new Worker().

Perhaps you might even be able to make use of Function.toString() to put the code in a function, get the source representation of it, and then make a blob from that. According to this Stack Overflow question it is non-standard though but might be useful with enough error checking. Then there would be no need for separate files.

[kaisalmen]

@Coburn37 thank you, I will investigate and see if one of the strategies could be used. One file containing all code has its charm.
@jonnenauha is the namespace THREE.WebWorker ok for the web worker code? I have introduced it and is not yet used in three.js.

[jonnenauha]

Yeah the blob approach works but not in all browsers three.js supports. I have used it before and believe Firefox and IE (<11?) had problems with it. Not sure about current state. There could be an optional file to include as ./build/WWOBJLoader2.embed.js that would just .toString() the whole thing into a global variable that can later be used to load it as a blob :) I would probably leave this out to start with...

@kaisalmen Just concat the webworker stuff instead of importScript, then provide plain and minified versions of all. I just feel the importScript parts will give users more problems, you have to option to make it dead simple.

Sane defaults are important too. If user includes minified version he expects it. So the min version of the library needs to load the minified version of the webworker js file. Same for non minified. You can do these transformation in your js task runner (gulp).

The worker namespace I don't know. I somehow have a feeling three.js wont be doing lot of webworker stuff (people here cant agree on something like this that needs strong opinionated direction in the dev, everything needs to be so generic that it works well for everyone). So probably wont be made a generic utility in three.js any time soon. Maybe but those as to THREE.ObjLoader2.WW.?

[Cobertos]

Just looked up the compatibility:

• Blob, IE 10+, FF 6+, Chrome 8+ - Don't see any known issues but of course I haven't tried it myself so...
• WebWorker, IE 10+, FF3.5+, Chrome 4+
• URL.createObjectURL, IE 10+, FF4+, Chrome 8+ - Though MDN says that only IE 11+ can use Blob URLs in WebWorkers

@jonnenauha Looks like the only browser unsupported that is supported by WebWorkers is IE 10. So I guess that's that.

[jonnenauha]

Right, I mean the ctor for taking in a blob did not work in all browsers back when I tried it. Looks like it was only IE10. I would still leave it for later, first get things going with the builds.

[kaisalmen]

Edit 2017-01-05:
Two bundles are available:

• OBJLoader2[.min].js: The new loader classes without any web worker stuff
• WWOBJLoader2[.min].js: Consists of parser, web worker, web worker proxy and director code. At worker init a string is built from code within the file that contains all code for the worker to work. It is put to a blob that is used to create the worker. Works fine and code creation takes less than one millisecond.

Edit 2017-01-07:

• Fixed bug: Minified version has broken constructors function e leading to broken blob based worker.
• Namespace THREE.WebWorker has been changed to THREE.OBJLoader2.WW
• Merged branch OnePackge back to master

Edit 2017-01-08:

• Removed some superfluous definitions in gulp instructions and blob creation code
• Proposal: WWOBJLoader2 should not include OBJLoader2Parser, then the code within both bundles is completely separated. Users of worker just need to included both bundled file. See

Edit 2017-01-09:

• Multiplied all (WW)OBJLoader2 related examples by three to be able to verify bundle, minified bundle and untouched sources work

Primary mission objectives complete ... now let's find the bugs. ;-)

[kaisalmen]

@jonnenauha, @Coburn37, I think this is ready for usage in three.js!
What do you think?

[Cobertos]

I took a deep dive into your repo and here's what I came out with. Correct me where I'm wrong because I'll probably need to integrate this into my project at some point and because I feel as though I might not be getting a good handle on some of the portions of the API. Also, the biggest issue I found was the API (again, I might've just read it wrong) and some of the redundancy/complexity of the names.

There are a few workflows to this:

• Using THREE.OBJLoader2 directly
  
  
  
  • Uses a newer parser that is more efficient/less memory intensive than the other one (due to use of ArrayBuffer, IE10+, FF4+, Chrome7+).
  
  
  • Has some gadgets like multi-materials, smooth/flat shading, and breaking up objects into multiple parts if the OBJ files calls for it.
  
  
  • Is setup as new THREE.OBJLoader2().
  
  
  • The API is more or less the same to the outside user. Internally there's now parse() and parseText() for handling text and ArrayBuffers
  
  
• Using THREE.OBJLoader2.WW.WWLoader2Proxy for the web worker version of OBJLoader2
  
  
  
  • Looks like ...Proxy is the class to use directly which pulls from all the other WW.* classes to actually build the web worker
  
  
  • Couldn't actually find an example of this, so I tried looking at WWLoaderDirector.js to figure out how this worked...
  
  
  • Setup with Object.create( proxyPrototype );, then .init(name), then .buildWebWorkerCode()
  
  
  • Run with .prepareRun({}) passed an object for what and how to run on, then .run()
  
  
• Using THREE.OBJLoader2.WW.WWLoaderDirector to use a pool of multiple web workers (inferred from example code in WWOBJLoader2Verify.js
  
  
  
  • Is setup as new THREE.OBJLoader2.WW.WWLoaderDirector( ##, ## ); with params for defining the queue size and stuff then .register(proxyProto, wwName, callbacks);
  
  
  • Running means calling validate(##,##) yet I'm not 100% certain what this does, then enqueueueForRun({}) with a object that has a bunch of different parameters for defining what to enqueue (defaults are in workerDescription) and how, then processQueue() which seems self explanatory.
  
  
  • Callbacks are progress, completedLoading, errorWhileLoading, materialsLoaded, meshLoaded and some more on a director object, not really sure what that is
  
  

Questions:

• What's the difference between buildMesh in WWMeshCreator and OBJLoader2MeshCreator? They look kind of the same but I can't think of any reason they'd need to be different except that WWMeshCreator would need to relay the data back as primitives or something because of web worker communication?
• When you say "Face N-Gons are not supported identically to the old parser" in your documentation, do you mean that n-gons are not supported, just like the old parser, or they are supported but just not like the old parser did it.
• ZipTools and other dependencies are only used for the tests and examples, correct? I don't want to have to depend on these to use them.

Concerns:

• Mostly just naming and API concerns:
  
  
  
  • Breaking out the examples & app folders into a separate examples/js would make things clearer that these files aren't used by the library but are for tests and examples (like three.js does)
  
  
  • Inconsistencies in naming: Compare filenames OBJLoader2Control.js and WWOBJLoader2Proxy.js with WWOBJLoader2 and WWLoader2Proxy respectively being the classes they house.
  
  
  • Similarly, instead of WW*.js, it would be nicer to have a WW folder and then take the WW out of the file names and class names (so it isn't as hard to follow the names like THREE.OBJLoader2.WW.WWLoader2Proxy due to the large amount of redundancy. If not that then perhaps some vastly different naming from what's currently used.
  
  
  • If WWObjLoader2 isn't really what should be directly called to make a web worker loader, I think the proxy object should be called WWObjLoader2 and WWObjLoader2 should be called WWObjLoader2Base or something because intuitively I'd use the most simply named class in my app (while Proxy seems more like an internal or very specific sort of thing).
  
  
• It doesn't look like you're using any es6 import/export stuff which I think three.js just recent (in the past 3 versions or so) converted to.

Other than that I think this is great work. If we end up switching off of OBJ to FBX I might end up having to port this to FBX too :P

[kaisalmen]

@Coburn37 Thank you for your comments. You get a thorough answer tomorrow. I need to go to bed.
I just realized that there is no simple example for WWOBJLoader2Proxy. I used the director in both cases. More tomorrow...

[kaisalmen]

The comments where really helpful and I have already started to prototype some ideas and thought about improvements where needed: Branch polish

Using THREE.OBJLoader2 directly

This does not apply only here: I was thinking to either make non-public functions like validate and finalize private to the object and passing itself to it or to mark non-public methods with a prefix or suffix. Do you have any suggestions here what is a good and common practice?

Using THREE.OBJLoader2.WW.WWLoader2Proxy for the web worker version of OBJLoader2

• A simple example is missing and I will create one where WWLoaderDirector is not used.
• I realized there is a simple way to fuse Proxy and the loader: The web worker code in its own file is no longer used. It is read, stringified and put to the blob, therefore it is the logical next step to include it as private classes to WWOBJLoader2.js. It contains all that is required!
• As mentions above public/private methods are not clearly identifiable. Documentation and example needs to be created and code documentation needs improvement

Using THREE.OBJLoader2.WW.WWLoaderDirector to use a pool of multiple web workers (inferred from example code in WWOBJLoader2Verify.js

• The constructor should not require maxQueueSize and maxWebWorkers values
• In all classes I created validate shall always ensure the object/logical construct is prepared for re-use. It is not necessarily a re-init that's why I chose a different name.
• Here validate uses the parameters to add, destroy workers if required like the Parallels demo does.
• Documentation needs improvement and public/private methods are not clearly identifiable

Questions:
What's the difference between buildMesh in WWMeshCreator and OBJLoader2MeshCreator? They look kind of the same but I can't think of any reason they'd need to be different except that WWMeshCreator would need to relay the data back as primitives or something because of web worker communication?

• WWMeshCreator just creates buffers used as transferables to transfer/communicate the vertices, normals, uvs back to the front-end which builds scenegraph objects. The web worker does only have access to that. It is completely independent of three.js code like all web worker code in contrast to OBJLoader2MeshCreator where the scenegraph integration is directly done.

When you say "Face N-Gons are not supported identically to the old parser" in your documentation, do you mean that n-gons are not supported, just like the old parser, or they are supported but just not like the old parser did it.

• "just like the old parser": Triangles, Quads, no n-gons. Non-native speaker issues, I guess. ;-)

ZipTools and other dependencies are only used for the tests and examples, correct? I don't want to have to depend on these to use them.

• Only example code requires this, correct!

Edit 2017-01-11:

Concerns:

The following is addressed on branch polish

• Each example has its own folder with html, css and js files
• Naming inconsistencies are resolved (there is always room for error of course, I will verify again). Package THREE.OBJLoader2.WW is only used in generated web worker code. WWOBJLoader2 and WWOBJLoader2Director reside directly in package THREE.OBJLoader2
• I like to keep these two longer class names even if they contain redundant information. Otherwise the name is very unspecific like ...WW.Loader
• Proxy naming issue is gone as WWOBJLoader2 is exactly this and it now internally and privately contains what becomes the web worker.

I have started to rename private methods and also started to moved functions up and down within classes to better follow the logical workflow. This is not yet completed. Documentation for public functions is also on my worklist.

[jonnenauha]

I see mostly underscore prefixing for private stuff, may it be property or function. I'm not sure if three.js has such coding conventions, they tend to at least wrapping functions for private variables they don't want to leak out. I would go with _finalize, this should tell people not to call it.

And of course leaving it out of the docs is a strong signal too :) I recommend you use jsdoc3 http://usejsdoc.org/ with https://www.npmjs.com/package/gulp-jsdoc3 or https://www.npmjs.com/package/gulp-jsdoc-to-markdown or similar. Produce the docs in the build step to /build/doc. Make your uglify/minify phase remove all comments from the source code (should do that by default).

[Cobertos]

I agree with @jonnenauha that prefixing with _ and leaving it out of the documentation would signal this well enough.

As for the renaming, I think that what you did in the polish branch is good. I saw you took out the root examples folder and consolidated everything into test which I think serves the purpose well. Your explanations helped with the other portions of my confusion as did the name changes. There's still a few little nitpicks I have from the previous post but give it some documentation and IMO it should be good to go.

[kaisalmen]

@jonnenauha and @Coburn37 I agree with your proposal to prefix private methods with _ (see edit of previous post). Doc will come once I have wrote all required function comments.

Completion and integration is near. Feels goods. Thanks for your support so far! 👍

[kaisalmen]

Public function documentation and identification of private functions via prefix is complete. I renamed some methods to make things clearer and fixed minor issues.
gulp-jsdoc3 processes the README.md, but no function comments are extracted from the js files. I need to investigate and understand what's wrong...

Edit 2017-01-14:
Okay, I found why jsdoc does not generate code doc: singletons need extra tags to be processed properly and I used them everywhere. ;-)

Edit 2017-01-16:

• Documentation for all required classes is now properly generated when gulp is run along with the bundles. Not all function arguments are yet properly described.
• Reverted move class THREE.OBJLoader2 to THREE.OBJLoader2.OBJLoader2. Late night ideas are not always the best.
• Added simple objects PrepDataArrayBuffer and PrepDataFile to ease understanding what is required for WWOBJLoader2.prototype.prepareRun

[kaisalmen]

Documentation is complete! I clarified minor things in the tests/demos and performed minor adjustments to the code (e.g. param objects).
@jonnenauha and @Coburn37 do you have further comments? Otherwise, this is now ready for integration, I think.

Edit: Each example needs to be flattened (all code inside a single HTML page) to be in-line with other three.js examples, correct? I still need to do this work.

[kaisalmen]

Example changes:

• A simple example for WWOBJLoader2 is now available. This is very similar to the OBJLoader2 example and just focuses on how to directly use the loader
• These two examples no longer need the helper code (ThreeJsApp.js and AppRunner.js). Code increase is moderate, but may ease understanding. I will port the others examples as well
• Gulp build produces three.js single page examples (css and all code inside one HTML page). I will make this available to all examples

[Cobertos]

Looks consistent in class names, file names, etc. I'm really liking this. My final suggestion would be to move the app directory out of src and into test if you can unless app is relevant to the actual OBJLoader2 (which from when I was reading, I thought it was just a tool for your own examples).

I'll have to integrate this with my app shortly. Nice work!

[kaisalmen]

Thank you! I moved all files from src/app to test/common. Loader code does not depend on these classes. It is only support code that is used by the tests and they will very likely become superfluous once wwobjloader2complex and wwparallels have been modified like the two simple ones.

[jonnenauha]

Nice work. If you feel its ready, make a proper release. Do the gulp build and make a zip file that has the ./build contents and upload it to the github release. This is faster way of people getting the library than clone > jump to tag > build. I will also give this a go at work a bit later when I get back to the relevant tasks/projects.

I would make it 1.0 and be done with it. If @mrdoob wants to slurp the library here, he can always use the latest tag from your repo. Imo its kind of irrelevant if it ends up here, it would be nice, but it can also live in the separate repo. We just need to advertise it to three.js users.

Remove the "prototyping" from the repo description too :)

I'm not sure what the best channels to let people know that this exists. Maybe @mrdoob can tweet the repo link :)

[kaisalmen]

@mrdoob, @jonnenauha and @Coburn37 I have completed the work on the examples: gulp will now produce single page versions of all examples.

WWOBJLoader has now reached version V1.0.0

I have uploaded a zip of the build folder to the release on github!

Please RT if you like!

Cheers :-)

[kaisalmen]

@jonnenauha and @mrdoob: I added a the V1.0.0 bundled code packages together with three examples on the branch WWOBJLoader2.

Consider this is a proposal. What do you think is the best way to integrate this code with three.js? What is both elegant and easy to maintain? This is a potential way forward, but eventually not the best. That is the main reason why I have not issued a merged request, yet.
Opinions wanted, thanks. :-)

[jonnenauha]

IMO the minified files should be left out. If someone decides to pull these into their project, they can worry about the minification then. I for example have build step for minifying the current ObjLoader.

It is /examples so I think the sources should be human readable :)

For better discovery I would add your ObjLoader2 repo URL to all the built files header. You have your own website there, but it should have also the project repo. https://github.com/kaisalmen/WWOBJLoader/blob/master/gulpfile.js#L30

Your build step should also have the version in the top comment or something like THREE.OBJLoader2.Version = "1.0.0";. Read the version during the build from package.json and put it somewhere. Both comment and a runtime variable would be nice. I hate it when its hard to figure out what tag/version I'm using on some lib :I

[jonnenauha]

But yes this was the way I thought it would be pulled in to three.js. You could send out a pull request on each significant release you make.

[kaisalmen]

@jonnenauha these were good ideas. Version is now included and the headers carry the reference to the development repository. I increased version to 1.0.1 as I made minor modifications to realize the changes and I added checks for Blob and URL.createObjectURL in additions to Worker in WWOBJLoader.

I have squashed and rebased the branch and will now issue the merge request.

[fraguada]

@kaisalmen I'm looking through your examples. Very interesting. I'm looking to do something similar for the THREE.ObjectLoader, specifically to allow its .parse() method to report progress for large models. If you have any additional tips learned from the process that would be applicable to doing this to the other loaders, they would be much appreciated.

[kaisalmen]

@fraguada, this lengthy post already contains some tips, but here comes a condensed list of hopefully helpful tips and POIs (btw, check out Implementation Overview):

• You need some front-end code where you interact with three.js and the web worker (or short worker) code that does performs async parsing. Workers can use three.js via imports, but every worker has its own independent instance independent from the main.
• Data exchange is realized with serialized data (copied) transferables (shared) for binary data (ArrayBuffers). You cannot exchange complex objects without serialization and therefore heavy costs.
• You need to think about logical structure of the code. Not everything is possible in the worker (e.g. DOM is not allowed) and some things should logically not be done in the worker. This is of course problem domain specific and cannot be generally answered.
• OBJLoader2 is a complete rewrite and therefore the code was allowed to be structured to have most benefit from externalizing parsing to the worker. The real data parsing code (which really takes time to execute) can be used within the worker or just in serial fashion on the main thread without the worker.
• So, the real difficult problem is to re-arrange and extract from existing code things that can be run async in the worker. This is likely no easy task.
• Import as few scripts possible to the worker code. Best is to have no imports at all. This will keep the init time down for the worker
• If possible build the worker code from existing/already in memory strings. This defends you from resources location problems.

The list is not yet complete (time is limited today and I tend to forget things as well 😉) I will edit and extend it...