Three.js: Custom FileLoader

[Paril]

FileLoader is extremely limited with it being hardcoded and stuck to using XHR. I want to load from other sources that aren't URLs on the visible web, and having data that I already have in memory moved/copied to an Object URL seems like a hack rather than a solution to this issue (not to mention that a urlHandler is too limited for this task if the request to load a file requires asynchronous behavior) - JSZip's latest versions, for instance, are async-only, meaning previous LoadingManager-based solutions can't work.

[makc]

THREE.FileLoader.prototype = <your implementation here>

you are welcome :)

[donmccurdy]

Would LoadingManager.addHandler work for you?

[Paril]

Good point about just replacing the prototype, I hadn't considered that one! I was trying to think of a way to do that with modules but forgot about my ol' ES5 roots.

@donmccurdy: addHandler is higher level. The issue is FileLoader, which happens way before LoadingManager is even used, but @makc's solution would indeed work for my purposes. The only issue left is that a lot of loaders "abuse" the file system by prepending the current URL to the file so that it resolves to a proper URL, but I guess I can figure out a hack to undo that.

Still, I think it'd be nice to have all of this stuff agnostic and just use the raw paths (as in, if OBJLoader/MTLLoader wants to load .mtl/.png, it passes it off to FileLoader as the path as it is written in the file, and then the loader's default implementation could be an XHR-based routine that prepends the current URL).

[Mugen87]

Um, it's not clear from your posts how your intended implementation would look like. Things might get more clear if you could share some code which illustrates the solution.

[Paril]

@Mugen87 I don't have any code because I ended up going for a self-rolled solution as this was a bit of a deal-breaker for getting something up and running in time.

Basically, imagine that your only method of handling files through JS is through blobs and/or ArrayBuffers, and you are attempting to use a loader that requires subsequent file loads (ObjLoader2 is an immediate example because it requires loading .mtl as a dependency, which then loads textures as dependencies). There's no mechanism for supporting these sorts of file systems at all. The only workaround I could think of is to push all of the blobs out as Object URLs and use a urlHandler to re-write the URLs to point to the proper Object URLs, which is a bit problematic too because a lot of the Loader subclasses change the format of the paths by, say, prefixing the current address bar URL, since it is basically running on the assumption that every file being loaded through it is a loose file in a folder somewhere.

It's just fighting against the system to try to load something from anything that isn't loose files in a directory on a folder on the server, that's the main issue here. I think the FileLoader should be designed in such a way that, on a base level, all it expects is to input path -> output Promise (depending on whether the requested file needs to be loaded as text or binary), and to build XHR on top of that in such a way that it is an optional subsystem. Consider an ecosystem that isn't necessarily running on a website and doesn't have URL (as a hypothetical; obviously this isn't the case here but it points out the reliance on FileLoader).

JSZip's most recent implementation is entirely asynchronous, which means that a urlHandler can't be used for on-demand decompression because urlHandlers must be synchronous. Replacing FileLoader's prototype as described above would probably work but since I went in a different direction I can't test this right away. I hope the concept ends up helping somebody down the road that needs to load models/textures from an abstract filesystem though!

[Mugen87]

Basically, imagine that your only method of handling files through JS is through blobs and/or ArrayBuffers, and you are attempting to use a loader that requires subsequent file loads (ObjLoader2 is an immediate example because it requires loading .mtl as a dependency, which then loads textures as dependencies). There's no mechanism for supporting these sorts of file systems at all.

GLTFLoader already resolves such dependencies with a Promise-based solution. AFAICS, this works quite well. TBH, I don't understand why you want to shift this logic into a much more low level FileLoader class which should be only responsible for loading single files. The management of what files have to be loaded in what order is a task of the concrete loader class (e.g. GLTFLoader).

I have to apologize but I'm still not sure I completely understand your issue. If you have some time in the future, it would be interesting to see an alternative FileLoader class that implements your presented approach. I'm sure things will become more obvious then.

[Paril]

GLTFLoader does have logic to handle this, but you can see with usage of createObjectURL that it still requires that it pass through as a URL and through XHR even though the data may already be a blob in memory. It's an unnecessary round trip.

The best I can suggest for showing you the issue is to attempt to implement an efficient, zero-copy (as in, no usage of createObjectURL to make a dummy to re-download ArrayBuffer/string data through XHR to a Loader) model loader that loads models contained inside of a zip using the latest version of JSZip. That was the main problem I was presented with attempting to solve. Any solution I could think of was essentially a hack which required butchering URLs or decompressing every file in the zip and storing them as object URLs before attempting to actually load a model (since the dependencies of an obj file for instance need to be at the ready and can't be handled asynchronously without modification of FileLoader).

[makc]

@Paril another option would be to enable THREE.Cache or what was its name, and put all your ArrayBuffer-s there under proper URLs before you actually start loading anything.

[Paril]

@makc That's actually clever too, but that requires copies - again I feel like this is a hack to work around something that shouldn't be an issue to begin with. I already have the data ready for JS to work with, I shouldn't need to expose it externally via an Object URL for THREE to then re-download and store as a second ArrayBuffer/string.

Plus, consider a case where I have a zip with hundreds of models in them of varying formats (so you've got a lot of textures, obj/mtl, gltf, all sorts of goodies in here) and you want to ask the user which to display; compiling the list of correct objects to upload and prepare for THREE would be pretty difficult.

[Mugen87]

Closing since it was never stated how the API should look like to handle the OP's use case. Like mentioned before, a concrete API suggestion or even an initial implementation would make this issue more clear.

BTW: If asset data are already in memory in form of a string or ArrayBuffer, it's possible to directly call the parse() methods of loaders. The editor already follow this approach.