Godot-proposals: Allow to import and load external assets at run-time (in exported projects, without extra PCK files)

[Xrayez]

Describe the project you are working on:
A game with user-generated content (and less of a modding support).
Mostly related to godotengine/godot#17848.

Describe the problem or limitation you are having in your project:
See very useful discussion which illustrates the problem at godotengine/godot#17748.

Godot requires that most files must be first imported into a project (converted from various file formats to Resources, such as *.png to Texture) before they can be used in a game, and this works well for games which don't require dynamic loading of assets.

The export process involves packing all of the previously imported assets into a special *.pck file which is then unpacked upon running a particular game.

The problem comes when you want to give a player an ability to load external assets which are not part of the original game, without creating additional tools to make *.pck files to be used by a player, because players are not developers, and there has to be an intuitive way to load external assets within a game itself. See also limitations at #1212.

Loading imported resources requires you to use load() or ResourceLoader.load(). Loading non-imported, external resources require you to replicate the import process at run-time first (which is already done in the editor), and then store a reference to the loaded resource somewhere for it to be accessed later, so it's not possible to use load() for those resources, because they were not actually imported, but created at run-time instead.

This in turn leads to various confusions and misuse like with Image.load() and ResourceLoader.load().

See also related issues and other proposals:

• godotengine/godot-proposals#732
• godotengine/godot-proposals#1658
• godotengine/godot#24768
• godotengine/godot#17337
• godotengine/godot#14879
• godotengine/godot#20972

Marginally related:

• godotengine/godot#14839
• godotengine/godot#24222

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
I suggest to add a way to seamlessly load() external resources by first importing them if they are outside of the res:// path, like user:// or any other system path.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
The crude logic is the following:

• for imported resources (inside res://), look whether a particular file was previously imported. If yes, then just fetch the resource with load().

• for external resources (outside of res://), we know that they are not imported, so import a file first using the same mechanism as in the editor, and then load() it normally.

Alternatively, perhaps ResourceImporter and corresponding import() GDScript method could be added (or exposed), to allow users to import external resources at run-time first, if this proves to be easier to implement and/or maintain.

I understand that a lot of importers are tied to the editor, so this likely needs a rewrite/refactor, or adding a build option to allow those importers to be available in release builds, at the very least.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Currently, you have to use File and various load(), load_from_file(), load_from_buffer() exposed methods scattered in various classes like Image, AudioStreamSample to replicate the import process to some degree (difficult to account for various peculiarities inherent to parsing files), so none of those existing methods provide a way to import those resources exactly like in the editor.

Is there a reason why this should be core and not an add-on in the asset library?:
I believe one has to rewrite/adapt the ResourceLoader and the import mechanism for this to work, which is pretty core.

I believe implementing this proposal could solve a lot of reported issues, and even some specific loading functions could be unexposed that way.

[amoriqbal]

Is anyone working on this already?

[Calinou]

@amoriqbal As far as I know, nobody is currently working on implementing this.

Keep in mind exposing import functionality in export templates will likely increase their size noticeably, which can be problematic for mobile/web platforms.

[drwhut]

For those interested, since it seems that this FP has been declined for good, I've made a C++ module that allows for importing resources at runtime via GDScript for my game, which can be found here. Feel free to use the code in tabletop_importer.cpp as inspiration.

It's worth noting that the C++ module alone won't work in a build with tools=no, since the importing code is in the editor. This is where I also made a custom fork of Godot which essentially just has the above module, as well as macros to make sure that the engine compiles with tools=no with the importing code, without any of the other editor code (so the final executable isn't massive).

[Xrayez]

@drwhut I've looked up the module you linked and even tested it out, also added to a list of community Godot modules in goostengine/godot-modules#4 which can be compiled for testing purposes (build artifacts for editor are also uploaded), thanks. 🙂

Looking at the source, this demonstrates the issue with having to add dedicated importers and expose corresponding methods for each recognized resource type.

That said, the way I see it, a GDScript import() method would be good to add, but I still think that it should be possible to use the ResourceLoader to do this under the hood with load(), we don't really want to pollute the global scope: #1590, and this could just be narrowed down to implementation details which could be disabled at compile-time without even exposing any other additional APIs for this.

It's worth noting that the C++ module alone won't work in a build with tools=no, since the importing code is in the editor. This is where I also made a custom fork of Godot which essentially just has the above module, as well as macros to make sure that the engine compiles with tools=no with the importing code, without any of the other editor code (so the final executable isn't massive).

Yes, so I guess the most feasible way to implement this proposal is through adding build options, like scons use_importers=yes. As Calinou said, I'm afraid Godot won't allow to bloat its binary size for mobile and web. But I think that usually, those platforms don't even need run-time import capabilities anyways.

Considering the fact that this is not what every game will need, nonetheless it's impossible to make this work with GDScript nor GDNative, since we need low-level access to resource importer API which is only accessible via modules, and also requires some core modifications.

it seems that this FP has been declined for good

That doesn't seem to be optimistic! With enough of support, I hope anything is possible. 😅

I mean, see the number of feature requests and issues I linked in the proposal.

Otherwise, yeah perhaps Godot is not the right tool for this, but it seems like a low-hanging fruit to implement, it's just a matter of finding a way to expose existing functionality in the engine to make this work in a general-purpose way.

Perhaps I'd work on this myself given approval of core developers, else maybe this can be implemented in a fork/extension. 😛

[drwhut]

Looking at the source, this demonstrates the issue with having to add dedicated importers and expose corresponding methods for each recognized resource type.

This is one of the challenges with implementing this feature - you need to instance the importer singletons if they haven't been instanced already (e.g. in a release build). In some cases, I've noticed there are also different importers for the same resource. For example, the .obj resource has one importer which creates an array mesh, and another that creates a mesh instance similar to other scene importers.

It's worth noting that the C++ module alone won't work in a build with tools=no, since the importing code is in the editor. This is where I also made a custom fork of Godot which essentially just has the above module, as well as macros to make sure that the engine compiles with tools=no with the importing code, without any of the other editor code (so the final executable isn't massive).

Yes, so I guess the most feasible way to implement this proposal is through adding build options, like scons use_importers=yes. As Calinou said, I'm afraid Godot won't allow to bloat its binary size for mobile and web. But I think that usually, those platforms don't even need run-time import capabilities anyways.

That's probably the best way to go about it, and based on what I had to do to get it working for release builds, this shouldn't be too difficult to do if you know your way around SCons.

it seems that this FP has been declined for good

That doesn't seem to be optimistic! With enough of support, I hope anything is possible. sweat_smile

I mean, see the number of feature requests and issues I linked in the proposal.

Otherwise, yeah perhaps Godot is not the right tool for this, but it seems like a low-hanging fruit to implement, it's just a matter of finding a way to _expose existing functionality_ in the engine to make this work in a general-purpose way.

Haha, while I do agree that this would be a very handy feature to have (as you said, the functionality is already there, it's just exposing it to GDScript), I'm hesitant to say that this would be useful to a majority of developers. For most developers, this functionality is already implemented statically with .pck files.
That's just my opinion though. It's really up to the developers to see whether this is worth implementing.

[Xrayez]

This is one of the challenges with implementing this feature - you need to instance the importer singletons if they haven't been instanced already (e.g. in a release build).

Possibly have to refactor this logic into a dedicated register_import_types(), currently all importers initialization is hardcoded in EditorNode constructor, looking at the source.

For instance, the existing ResourceImporterWAV would be useful for #732.

I've noticed there are also different importers for the same resource.

Yeah, when we talk about importing images for example, then there are numerous ways to import as Image, Texture, BitMap if you go into the Import tab. I think it would be just a matter of providing an additional parameter, like import(Dictionary options) or something along those lines.

Haha, while I do agree that this would be a very handy feature to have (as you said, the functionality is already there, it's just exposing it to GDScript), I'm hesitant to say that this would be useful to a majority of developers. For most developers, this functionality is already implemented statically with .pck files.

I dunno, I think this is where the engine could really shine and has a good potential, it's just that none of the existing core developers really dig into the UGC/modding capabilities with Godot currently, that's all. It's a matter of creating enough demand and awareness.

[Xrayez]

Also, regarding increased binary sizes, I think it's not that much code which needs to be exposed out of the box currently, I speculate that it would only add up like 3MB to export templates. In any case, depending on my motivation, I'll see whether that's actually the case, but it should be quite possible for Godot to provide such customization level at build-time, at least, so the actual custom non-editor resource importing feature can be implemented outside of Godot more easily without having to patch the source for that.

[akien-mga]

I support this proposal, though we need to get @reduz onboard with this as he designed the import system (and thereby its limitations).

For a concrete example of use case, have a look at https://github.com/Gianclgar/GDScriptAudioImport
This is a GDScript script that can load WAV and OGG files at runtime, and thus needs to replicate the logic that the importer would do in the editor.

For OGG it's "simple enough", like the hacks that users currently have to do to load PNGs at runtime:
https://github.com/Gianclgar/GDScriptAudioImport/blob/master/GDScriptAudioImport.gd#L114-L118

But for WAV it requires a full reimplementation of the header parsing already done in the engine:
https://github.com/Gianclgar/GDScriptAudioImport/blob/56c8d70f8aa4da3ee0d498497c61e583208a515b/GDScriptAudioImport.gd#L42-L111

Here's a quick Music Player that I made to help https://store.steampowered.com/app/1369320/Virtual_Cottage/ developers to include runtime loading of user-provided music tracks:
Music Player.zip

I only handled OGG as it's easy enough from parsing the file as raw data, but for WAV I'd have to do the same as https://github.com/Gianclgar/GDScriptAudioImport

IMO that's a use case that we need to support out of the box, so importers should be available at runtime - at least for the most common data formats like PNG, JPG, OGG, WAV, and possibly MP3 if #85 is implemented.
This may not be possible/relevant for formats like glTF or Collada, which are full scene formats.

[Xrayez]

This would also be useful for #1362, as import functionality doesn't have to be tied to editor instance:

So better idea is to add new flag e.g. --import-only which would import files and after that just close Godot.
It would be great if the editor wasn't even running.

[akien-mga]

For the reference, we discussed this a few days ago with @reduz and @Calinou, see logs here:
https://freenode.logbot.info/godotengine-devel/20201020#c5530593-c5531175

A key distinction made during this chat is between import (which should stay an editor-specific process) and loading resources (which should be made as easy as possible at runtime). So we wouldn't add ways to import resources and use the imported versions with ResourceLoader at runtime, but there should be APIs for any relevant resource type to load them at runtime (as is already possible - by jumping through a few hoops - for images and sounds).

For more complex resources, @reduz suggests adding a feature to be able to export an imported resource (i.e. while in the editor) to a single file (bundling all dependencies) that could be loaded at runtime. This needs its own proposal for further discussion though.

[Xrayez]

Yeah, as I said, @reduz likely doesn't dig into UGC with Godot 😕.

Quoting you, @akien-mga:

Akien: reduz: that only helps for the use case where you (the dev) have the resource beforehand. The main use case that motivates the above proposal is loading user-provided content, like letting users load their own music or pictures

---

For more complex resources, @reduz suggests adding a feature to be able to export an imported resource (i.e. while in the editor) to a single file (bundling all dependencies) that could be loaded at runtime. This needs its own proposal for further discussion though.

Yeah, not useful for this proposal at all, and certainly not useful for me. 😄

The idea is that you should just be able to import a new resource with import(path). This will store the imported resource at user:// directory (such resources won't be packed), which then can be loaded normally with load(path). This would solve the collision mentioned by @reduz, so really this could be implemented as a parallel system to ResourceLoader.

The module provided by @drwhut in https://github.com/drwhut/open_tabletop_godot_module does something similar, but it's not general-purpose enough, because importers have to be instantiated manually for this, and of course required some core modifications to build scripts.

That said, for now this is modules territory, it seems. But I'm not sure what minimal core functionality is required to facilitate implementing this proposal via modules yet. For now, I'm limited to Image.load().

[Xrayez]

At the very least, my recommendation/suggestion to developers is that common resources like images and audio should have loaders implemented in classes, and the import process can just reuse those loading methods just like with the current Image.load().

The reason why Image.load() exists in the first place is because there are different kinds of loaders for each image format registered in the engine, and that's handled by ImageLoader internally, so I think we've just got lucky enough such a thing exists and exposed to scripting currently. 😃

[fire]

According to my reading of the logs, Reduz has rejected runtime import and export code.

Since this is the case, and I want to use my glTF module to runtime load glb files. I'm not sure what to do.

I'm presuming that gltf2 export and import at runtime will be rejected.

Since this is my requirement. Here are some workarounds:

1. Use an external Godot Engine module.
2. Create a custom out of core module that wraps all the loaders for Godot Engine and exposes it as a runtime loader.
3. Create a custom out of core module that wraps my import and export it as a runtime saver.
4. Not wanting to argue over this. So ResourceRuntimeLoader and ResourceRuntimeSaver will be designed to Godot Engine core requirements and posted at godot-extended-libraries.

API:

    Ref<EditorSceneImporterFBX> import_fbx;
    import_fbx.instance();
    ResourceRuntimeImporterScene::get_singleton()->add_runtime_importer(import_fbx);

    image_loader_svg = memnew(ImageLoaderSVG);
    ImageRuntimeLoader::add_image_runtime_format_loader(image_loader_svg);

    resource_format_saver_crypto.instance();
    ResourceRuntimeSaver::add_resource_runtime_format_saver(resource_format_saver_crypto);

    ResourceRuntimeSaver::save(path, resource, flags)
    ResourceRuntimeLoader::load(path, original_path, type_hint, no_cache, error);

May be converted to use any system that can compile c++ code to intermediary binary code to executed at runtime on any architecture.

Edited:

A loader for TSCN, TRES, STEX is not an acceptable route.

[Shadowblitz16]

So what I want to see is the ability to mark resources to be exported as certain file types...
textures - png, bmp, gif, tiff, jpeg
models - gltf, obj
other things - json, ini, cfg

Then when the editor versions of the resources are created in code they automatedly populate themselves and are ready to be used.

An ideal use for this would have a game json which contains all your object's json files which contains paths to images, models, scripts and other nested json files