Three.js: Enhancement: Support for zipped file in all loaders

[v1nce]

It looks like stackoverflow is "full of" jsthree hacks for adding zip support for this or that loader.
Of course those hacks are incomplete and become deprecated with each new version of jsthree.

How difficult would it be to officialy patch the base loader to accept some kind of "ressource resolver" instead of just an url ?

[makc]

is there a base loader? since when?

also, it's not difficult, as most loaders have 'parse' method - you can completely bypass 3js' loading code, download and unzip stuff on your own, and then just parse the result.

[looeee]

is there a base loader? since when?

There's FileLoader - its used by most of the core loaders anyway. Since when I don't know though!

[makc]

I thought this was about inheritance kind of base loader. FileLoader is just renamed XHRLoader iirc

[v1nce]

It will be cool if you could just drag and drop the zip that you get at clara.io, 3dwarehouse... to the editor without having to unzip them first.

also, it's not difficult, as most loaders have 'parse' method - you can completely bypass 3js' loading code, download and unzip stuff on your own, and then just parse the result.

I think it's not that simple. Even if you unzip a collada, search for the dae and patch it to replace the relative files references (image...) with dataurl you cannot parse it using the parse of loaders.

var daeLoader = new THREE.ColladaLoader();
daeLoader.parse(collada_json_with_dataurl_for_images_as_a_string,function(){},''");

this won't work because the loader will prefix your "data:image/gif;base64,R0lGOD..." with ./ (or just anything you'll pass as 3rd arg of parse) and so it won't be able to get the images.

[makc]

if you push the square box into round hole, expect some resistance ) but, if you push strong enough, it will go through, eventually.

3js cannot unzip things on its own without making choices for people re what zip lib to use, which has nothing to do with 3d.

[yomotsu]

as @makc said, loaders are extensible, and here is an example
https://github.com/yomotsu/xmas2016/blob/master/src/js/utils/zipFileManager.js#L98-L134

load zipped file as a blob file and extract it with another lib such as JSZip, then use parse method of Loader

[yomotsu]

btw there is another compress option called Draco

[RemusMar]

It looks like stackoverflow is "full of" jsthree hacks for adding zip support for this or that loader.

The best solution is an unified binary & compressed format (such as SEA3D).
Example: http://necromanthus.com/Test/html5/SMC.html
Everything is packed into a single (small) file.

p.s.
The text files (JSON, OBJ, DAE, etc) are not suitable for 3D online projects.

[v1nce]

I know binary formats will be more efficient but I got no control on what online libraries offer.

the largest library I know of (3dwarehouse sketchup) only propose collada (or its in proprietary format)

[v1nce]

From what I understand of yomotsu code the asnwer is not

• unzip a collada
• search for the dae

• patch it to replace the relative files references (image...) with dataurl.

  var daeLoader = new THREE.ColladaLoader();
  daeLoader.parse(collada_json_with_dataurl_for_images_as_a_string,function(){},''");

but rather

register a handler for zipped ressource

THREE.Loader.Handlers.handlers.push(
    /__zip__/,
    {
        load: ( filename ) => {
            return zipFileManager.loadTexture( filename.replace( /^__zip__/, '' ) );
        }
    }
);

• unzip a collada
• search for the dae json file,
• pass it to the parser

Make sure you use the special marker you chose for the handler ("__zip__") as a prefix for the url the parser awaits => daeLoader.parse( JSON.parse( json ),..., '__zip__' + dirName );
This way, when the parser will encouter a relative path it will combine "__zip__" and the relative path and call the appropriate handler.

[mrdoob]

It will be cool if you could just drag and drop the zip that you get at clara.io, 3dwarehouse... to the editor without having to unzip them first.

Yep. I have plans on adding this 👍

[mflux]

I was able to successfully use https://gildas-lormeau.github.io/zip.js/ to unzip a file containing Collada files and its texture images directly in JS.

Passing the DAE as a blob was easy, and ColladaLoader2 can load this. However, I'm not sure how ColladaLoader2 looks for textures (as all path linkages are gone after unzipping).

The way I see the problems:

1. Upon unzipping, you have to guesstimate the folder structure. You can just scan the zip looking for .DAE files however
2. Where does the DAE attempt to look for textures? Is it possible to override where it's looking, and simply build a lookup table for the textures that we've unzipped?
3. Since zip is essentially a folder, it's necessary for this feature to load multiple models (and possibly multiple types of model formats).

Finally, the unzip and collada-parsing is quite slow. I tried putting it in a web-worker, however this requires the webworker to have THREE.js, ColladaLoader2, and zip.js.

Unfortunately (I'm not sure if it's specific to ColladaLoader), some loaders require DOMParser which is not available in a web worker.

[mrdoob]

@mflux Take a look at KMZLoader (SketchUp):
https://github.com/mrdoob/three.js/blob/dev/examples/js/loaders/KMZLoader.js#L29-L89

[mflux]

Waaiiiiiit what the heck is KMZ just Collada in disguise!?

edit Ahh no since sketchup files are skp and require sketchup to re-export as kmz (google earth).

Anyway, off-topic, please continue.

[mgreter]

Would be cool if there was some kind of global hook to define additional decompression algorithms to enable new compression file extensions. I personally use js-lzma and I also patched quite a few loaders to support it. I guess zip would not be very optimal (until I missed the case that multiple files in the archive are wanted) and would rather suggest to use gz instead. FWIW: when I evaluated my prefered compression library, I made some performance assessements (i.e. pako-gz).

[donmccurdy]

@mflux @mrdoob there is a proposal for resolving URLs (without find/replacing in the asset body) here: https://github.com/mrdoob/three.js/issues/11072#issuecomment-294279686 .. I think this would work for ZIP archives, as well.

[donmccurdy]

LoadingManager now has a setURLModifier method, which can be used to override relative paths with Blob URLs from a Zip. This requires a little setup, but is general enough to also work with drag-and-drop APIs etc.

• Docs: https://threejs.org/docs/#api/loaders/managers/LoadingManager
• PR: https://github.com/mrdoob/three.js/pull/11259

[looeee]

LoadingManager now has a setURLModifier method

I tested this out for the first time the other day to implement a drag and drop setup, and it works really well! Thanks @donmccurdy 😄