Three.js: OBJ file larger than 256 MiB does not get loaded

[grzegorzborowiak]

Description of the problem

OBJ file larger than 256 MiB does not get loaded. This is because OBJLoader.js relies on storing the content in a single string, and Chrome does not allow strings larger than 256 MiB (besides, Firefox has even lower limit).

Three.js version

• [x] Dev
• [ ] r76

• [ ] ...

  Browser

• [ ] All of them

• [x] Chrome
• [ ] Firefox

• [ ] Internet Explorer

  OS

• [ ] All of them

• [ ] Windows
• [x] Linux
• [ ] Android
• [ ] IOS
  

  Hardware Requirements (graphics card, VR Device, ...)

[jonnenauha]

Yeah, the parser is fairly simple and I think the intention is to keep it simple. But I do think this could be achieved without sacrificing it. The problem of keeping it simple is that you cant eak out the perf, if you wish that anyone can read and understand the code (it is in /examples/js/ after all).

I have been thinking if I should create a separate project for a full rewrite. Haven't gotten the motivation yet.

After my last perf PR to OBJLoader is also implemented a version that kept the parsed floats/ints in typed arrays instead of JS arrays. This was quite a good improvement in perf but @mrdoob felt he wanted to keep the loader simple.

But to your problem... I think the load request needs to be done as a binary blob and read it chunk/line at a time and process the line. I feel that this needs a new parseBinary function next to the current string one, otherwise backwards comp is totally broken.

This might be good opportunity in general to do some crazier stuff to eak out the perf :) Most people will be perfectly happy with the text based parse, but once the obj file is > 10mb, the current parser starts to get slower (I've tried 66mb files).

I would also like to see it process the file asynchronously so that it wont block the main rendering thread for multiple seconds. This parseBinary could do that as well when provided with callbacks.

[mrdoob]

I feel that this needs a new parseBinary function next to the current string one, otherwise backwards comp is totally broken.

We could also start a OBJLoader2 😊

[grzegorzborowiak]

Maybe I could help you with that? I already started working on it (for my own needs). Currently my OBJLoader.js is capable of parsing ArrayBuffer, albeit I had to modify XHRLoader, too, to allow loading with responseType='arraybuffer'; small modification, just sixth parameter in load() method, which - if set - is set as responseType.

[jonnenauha]

@grzegorzborowiak I'd love to see the code if you want to push it to some repo or github gist. If I ever start another lib I'll surely welcome other contributors once the basics are fleshed out :) Or if someone beats me to it I can help out there.

Imo we could work on a better performing loader in a separate repo as its own project. Then when it reaches feature parity or surpasses THREE.ObjLoader we could think of importing it as THREE.ObjLoader2. I think separate repo is nicer for three.js users as it can have its own release cycles, you can pull it in as a lib from bower etc.

Main features I'd like to see

• Load everything with arraybuffer if its faster and more efficient.
  
  
  
  • I would assume so but havent done any testing.
  
  
• Use array buffers in the internal parser state instead of vanilla JS arrays, again if faster and more efficient.
• Async loads by default, even for parse().
• Load option (or just default to this) to use a web worker to not block the main thread.
  
  
  
  • Even small obj files take ~50-100ms on powerful desktop PCs, mobile will be painful.
  
  
• Possibly optimize and combine geometry for submeshes that have the same material and other properties that affect rendering.
  
  
  
  • I've seen files that use same material for hundreds of simple submeshes, big drawcall overhead.
  
  

[mrdoob]

albeit I had to modify XHRLoader, too, to allow loading with responseType='arraybuffer'; small modification, just sixth parameter in load() method, which - if set - is set as responseType.

Uh?

https://github.com/mrdoob/three.js/blob/master/src/loaders/XHRLoader.js#L111-L115

[jonnenauha]

@grzegorzborowiak You might be interested in my https://github.com/jonnenauha/obj-simplify project. Maybe you can trim those big files down a bit ;) The download section has cmd line tool downloads for various platforms.

Still haven't started my "OBJLoader2" :(

[grzegorzborowiak]

Thanks a lot!

I have no control over what files I need to handle, they are
customer-supplied. But maybe I could convince my customer to use your tool
to make them more compact - afterwards, he is probably interested in making
downloads faster.

On Thu, Sep 8, 2016 at 3:16 AM, Jonne Nauha [email protected]
wrote:

@grzegorzborowiak https://github.com/grzegorzborowiak You might be
interested in my https://github.com/jonnenauha/obj-simplify project.
Maybe you can trim those big files down a bit ;) The download section has
cmd line tool downloads for various platforms.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mrdoob/three.js/issues/8704#issuecomment-245466906,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARsNHmQya5zXULDwAi3znYNHLlhr2ljOks5qn2HzgaJpZM4IMrWb
.

[jonnenauha]

That is my situation exactly. I made it for obj authors who use our system so they can run it manually prior to upload. Certain software our users use to export make absolutely horrific amount of geom group / multi-materials (10-20k draw calls with few million vertices), this tool reduces three.js draw calls to the number of unique materials (usually below 20). Has worked out great for us so far :) The file size saving is just a bonus.

If you have problems with certain files please make issues, I'm actual working on a speed improvement for multi-million line .obj files but its not done yet. It does heavy parallel scanning or the file data once its parsed, to find duplicate vertex/normal/uv. Atm that tends to get quite slow if the file is big as you need to match each vertex to all the others at least once.