Godot-proposals: Allow modification of engine's types via scripts on runtime

[creikey]

Describe the project you are working on:
A game with a complex GUI.
Describe the problem or limitation you are having in your project:
I want a centered pivot for the Control node, that has already been made in a pull request, but will not be merged due to bloat ( a lot of projects do not need it ).
Describe the feature / enhancement and how it helps to overcome the problem or limitation:
By allowing the user to add a function to built in types via script, and sharing them in the asset lib, an ecosystem of basic engine modifications can be made, allowing godot users to access advanced functionality ( a new property in the Control node that you need for your specific game ) without recompiling the engine, and thus requiring deep knowledge about C++ best practices. Allowing the user to extend engine types with scripts would make engine modification more accessible.
Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

As an example, the set function in object.cpp could be modified as such to support this behavior:

        String parent_class = get_parent_class_static();

    while(parent_class != "") {
        Ref<Script> possible_script_override = ClassDB::script_override(parent_class);
        if(possible_script_override.is_valid()) {
            possible_script_override->set(p_name, p_value);
        }
        parent_class = ClassDB::get_parent_class(get_class());
    }

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Partially, but with more base modifications, it becomes near impossible to add a component to every node that needs this functionality.
Is there a reason why this should be core and not an add-on in the asset library?:
If this functionality was core, all modifications that would be bloat if it was in the engine could be run by the user as an addon in the project, allowing for each project to have its own de facto customized engine without recompiling anything.

[TheDuriel]

I have previously ran into scenarios in which this could be useful. For example for creating my own custom MainLoop, with a corresponding _myprocess function in all scripts. Or for networking, where a piece of code must be present in several (dozen) different classes. All extending from different built ins.

[willnationsdev]

This sounds like it'd basically be introducing something akin to extension methods into the engine core. Which actually sounds like an even better idea potentially.

For example, what happens when a user installs a plugin that attempts to add a script override to a class that they too wish to have a script override on? They don't want to override the plugin's script override, lest they break the plugin. But they can't just arbitrarily merge the code of the plugin with their own, especially if the scripts are written in different languages.

But if it were a proposal to register arbitrary scripts as things that can define extension methods, then that sounds more flexible. Something like...

ClassDB::register_extension_method(StringName p_class, StringName p_method, String p_script_path) {
     extension_methods[p_class][p_method].push_back(p_script_path);
}
// example
ClassDB::register_extension_method("Control", "some_method", "res://path/to/script.gd");

# res://path/to/script.gd
static func some_method(p_control: Control) -> void:
    print("hello")

Some things that might need to be clarified: how would these callbacks relate back to the original callback? Always execute after it? Or can we also inject them before? Do we have the option of replacing the operation as a whole, or is that not feasible (knowing full well that someone doing this in script code might cause the engine to break in the process)? Or, perhaps if you register it as a replacement method, the replacement only occurs in calls made from the scripting API and does not affect internal engine usage of the method (though, I suspect that scenario would prevent OP from getting the desired behavior). Some of these questions also apply to OP's "script override" concept too.

[creikey]

@willnationsdev The proposal is intended to function exactly how normal scripts operate on top of the inheritance stack, such a script on top of Sprite, but in the middle of the inheritance stack, so methods would be able to be overridden by the script ( called first on the script ) but then the original function can be called with the .func() syntax. If multiple scripts are added, then it would be up to the developer to "merge" the two by selecting which inherits from which in the editor ui, resulting in a structure like this:

[CedNaru]

Agree on this one. There are cases where I want to add basic functionalities to an engine class high in the hierarchy instead of having to implement them in each script used in the child classes.
You can create a similar behavior by creating a base script that all other scripts will implement but it results in unnecessary inheritance trees. It's even more limited because you can only extend one script at a time and there is no trait, interface system.
A way to extend/modify the engine types has the potential to simplify some scenes and make Godot more modulable.

[reduz]

My view on this is that it's not technically possible, at last in a practical way. If Godot was programmed in C#, Java or Python, it would be easy to replace a method in a base class by your own version.

As Godot is written in C++ there is no simple to do an override of this kind. In the few places where we allow this, it requires a lot of manual work to make it happen from script. Doing this from the setter/getter API is possible in theory (to intercept and write a custom function there), but the problem is that Godot C++ code uses the C++ API, not the setters/getters, so your overrides will never be called where it matters anyway. (Only scripts, the editor and the serialization code do). If we modified the C++ code converting it to use the setters/getters would just destroy performance and make the code a lot more difficult to read and maintain.

As this is not a vital feature and, honestly, there are often other ways to solve these problems (evidenced by years of users not needing it), putting on a scale supporting this in a way that makes sense vs not doing it, I think it's not worth it.

[reduz]

So, if users actually have a need of a specific feature, it's often better that they request the specific feature first, then if we decide to not add it, we can see which way we can create for users to add them themselves. The use case needs to be understood first before flexibility is proposed. I think following these steps is better than creating an universally flexible solution which we don't know how useful it will be in the end.

You can read more about this development philosophy here:
https://docs.godotengine.org/en/stable/community/contributing/best_practices_for_engine_contributors.html

[creikey]

@reduz In object.cpp there is already a look up to the ClassDB on every set/get call, so adding another lookup to see if a specific native type has been overridden doesn't seem at a glance like it would impact performance much. This feature isn't intended to be used to modify the existing C++ API and how nodes interact with each-other, but rather attach a script to multiple nodes in a project based on a base native type ( in the middle of the type hierarchy ). I can draft a PR to add this functionality by intercepting the set/get calls, just like scripts do, allowing custom scripts to be set per native type.

[reduz]

@creikey If it's just attaching to multiple nodes, there may be other ways to do it I guess. Keep in mind that, as an example, in the new GDScript VM or in the binders, depending on the language, the setters/getters of known types are bypassed to use pointer-based calls as optimization as we assume these methods won't change. If they were fully dynamic, users would not be able to enjoy this type of optimizations.

[creikey]

@reduz as what methods are overridden or added is known on parse time, those optimizations should still function correctly. Just as scripts are attached to native types now out the outermost layer of inheritance, this proposal is about allowing scripts to be attached to inner layers of inheritance to all nodes. Here is a poorly photoshopped example of what it might look like in editor to better explain:

In this case everything that inherits from CanvasItem would route setget/function calls through the CustomClass first.

[CedNaru]

I see, so if we were to implement it this way. Only the calls made through the script API would be overridden, not the low-level calls made internally by the engine so we would have two different behaviors for the same method depending on the context.
And we can't just replace every call by ones using the script API because it would require a lookup in the ClassDb each time, killing performance (+ too much work to replace all that).

So, at best, we can only add new methods or properties but we can't change existing ones.

[Jummit]

Is modifying the engine an option? I do that all the time to add features to existing nodes.

[TheDuriel]

Limiting the system to only allow the _addition_ of properties and functions, would preserve most of what makes it attractive to people. While from what I understand, eliminate some of the technical limitations of implementing this.