Godot: Problems with preventing "engine bloat" and the AssetLibrary

[karroffel]

Sorry for the very long post. I'm not mad if you don't want to read the whole thing, so here's a short version of it.

This is a mashup or many different issues. I didn't know where a comment like this would have been appropriate, so here it is in its own issue.

TL;DR

Using assets should feel like using core-functionality. Currently it doesn't. Proposed fixes:

• add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification
• make it easier to extend custom nodes, allow cross-language script inheritance
• allow scripts to be referenced by class name and namespace to avoid super long paths in case of assets.

---

Related issues:

19178

17092

15661

13187

10635

7402

6277

5947

---

Details here ⇓

Godot is amazingly powerful for the size of the binary. Unity and Unreal are bloated in comparison, but they do offer more QoL functionality out of the box.

The situation with Godot is that there are some PRs that add good and useful new nodes which are rejected since they are "easy to code in GDScript and don't need to bloat the engine". This is a very valid and reasonable reaction - if Godot should stay a < 40 Mo download then adding new Nodes and features needs to be justified.

While I do think this is a nice approach that ensures a pretty clean core, it has some problems.

Criticism and problem description

1. Functionality in core is official and undergoes quality control

If there's an InterpolatedCamera Node in Godot, I assume that's the interpolated camera most people use, I assume that it's the most developed and "optimized for the engine" one since it's the one that's part of the engine. It has something official about it.

The good thing about a "bloated engine" with many features is that those features are official and most likely the most widely used ones, more tutorials and documentation about them etc. They have been tested in many projects and there's a single place to discuss how to improve them.

I, as a game developer, prefer official nodes over custom ones written by Some Person. Of course this approach has downsides, like huge download sizes, longer loading times and other nasty things. BUT I think in terms of user friendliness, official features are preferred over custom addons. Which one of the 4 InterpolatedCamera assets should I use? How do they differ? Is the documentation good enough? Where's the issue tracker? etc etc...

2. Downloaded assets from the AssetLibrary are just copy pasted into the project

There is no difference between an Asset from the AssetLibrary and some code/scenes from the "main project". However, there is a difference between "official" Nodes and Nodes imported from the AssetLibrary.

If I wanted to add some custom behavior to one of my InterpolatedCameras, I could add a script on it and add my functionality - all cool and nice!

If I wanted to add some custom behavior to one of my AssetInterpolatedCameras, I could add a script on it and - oh. Attaching a script to a custom Node actually replaces the script. The very same script that is the reason I downloaded the asset. This is an editor problem and a core problem at the same time.

Problems:

1. Script inheritance isn't suggested when attaching a script - override is the default for those cases.
2. Script inheritance doesn't work cross-language. A project that uses C# can't extend the camera with C#, while this would be possible with an official node.

Fixes:

1. Make the editor fill in the path of the already existing script as the base type so script inheritance is the default. But this doens't solve the problem when you want to clear your custom script for that one camera as it would just remove all scripts entirely.
2. Make script inheritance part of the Script API. Currently the scripting languages all do script inheritance on their own - they don't know how to deal with other languages. My suggestion is to add a set_base_script that accepts every type of script. Usually the scripting language implementations just use the Script API on the base class anyway, so this could be generalized.

The problems of "copy-pasting assets into the project gets amplified by the fact that all scripts are only defined by their path.

In the camera case, I could do var camera = InterpolatedCamera.new() with an official Node, but I would basically have to do var camera = preload("res://addons/com.name.interpolated_camera/scripts/interpolated_camera.gd").new().

This is more than suboptimal. I know that reduz opinion is that paths are easier to use than class names, but I know for a fact that multiple users would prefer an optional alternative system that uses namespaces and class names instead.

Especially in the context of downloaded assets, this would be a huge improvement.

Proposed solution

Downloading assets and using them should feel like they are part of the engine. If they engine is kept small and meant to be extended with assets, that workflow has to be made as seamless as possible.

The basic goal is to make using assets feel like using core-features.

My proposed solution is a mix of a few existing discussions, but I think this is the place where all of them would shine together.

1. Make the AssetLibrary more package-manager like

• Having a dependencies.json could be good enough. Using the AssetLibrary UI in the editor would just fill in the json, the download and installation happens at another time. An asset can itself have dependencies.

• Assets are saved with version associated, all stored in .local/godot/assets or something like that. They are no longer part of the project itself but available system wide.

• At export the needed files can be copied into the .pck.

• Have a way to actually copy the files into the project in case modification at the source level is desired. Only then the files should be brought into the project folder itself. "Cloned" assets could live in a hidden folder like .custom.

2. Make using downloaded scripts easier

• add a ScriptDB kind of system that can be used to reference classes without knowing the exact path

• add support for cross-language script inheritance

Conclusion

I know this is a very long post, but I hope to spark some more discussion on the AssetLibrary and also usability side of things. I think making using assets as smooth as possible is important. While things like proper versioning and allowing sub-projects will help, I think it's not enough to shift the focus to a more AssetLibrary focussed use of the engine.

[toger5]

Really like your post.

EDIT:
Damn it overlooked this paragraph

Have a way to actually copy the files into the project in case modification at the source level is desired. Only then the files should be brought into the project folder itself. "Cloned" assets could live in a hidden folder like .custom.

So you basically have covered the concerns I had. (sorry)
I just go more in depth how I imagine the workflow + the behaviour.

POST:
One workflow is overlooked IMO though.
Using assets as just a resources or a first building stones.
The current really transparent and simple approach also lets you just download a couple of assets. delete some files. Than keep some png's and a script which has some of the functionality you want in it. Currently it is possible to just work on them and modify.

If assets would be hidden in a .local/godot/assets folder this functionality would be kind of lost.

Proposal

Assets should still be visible and listed in the in editor file browser. Independent if they are inside the .local/godot/assets folder or the project folder itself. So you can browse the files and understand how it works in case there is no proper documentation, or if you want to learn from it.
Every Folder which is 'an Asset' should also be marked as one with some kind of icon so you have an overview what is an asset and where the actual files are stored.
There should also be a editor window which lists all assets. (in a table)

• Downloaded lists all the assets which were downloaded. They can be added to your project with a click of a button. But other than that they are the same than assets in the asset lib. This list is the same in any project and has no effect on the project itself. (just makes it easy to have an overview. it's also easy to clear this list and redownloaded all the assets when opening a project.)
• Always active (basically just toggles "Used in this Project (active/inactive)" for every project)Those assets are added to EVERY PROJECT on your machine (even newly created ones). (Maybe the asset itself needs to allow for that.) Those are really Editor orientated. Like a clock in the corner, git support, todo lists or to have some more default resources:(a custom environment some more themes and a bunch of sprites to work with at the beginning of each project)...
  They can be manually deactivated per project.
  They are still listed in the project file system though and behave like any other assed/plugin. so you can browse their files and see how they work/what they do.
• Used in this Project (active/inactive) this actually shows the linked folder in the project file system. and "adds" (they are still inside the .local/godot/assets folder, but behave like project local files.) files to the project.
  Here you can also choose which version of the Asset you want to have activated. And switch the version.

As soon as you want to edit/save/update a file from an Asset, you get prompted with two options

Save in .local/godot/assets

either update the asset in the .local/godot/assets which puts it into a custom user version.
It the package-manager is git based, the custom user version could be one additional commit. which gets commit --amended whenever you save.
(3.2.5/user for example) and is than available in all your other projects as well. (when updating the version in those other projects to you user version)
why is this useful? (example)
A theme from the asset store is almost perfect, but you hate the font type...
change the font type save it as a new asset version. Revisit all your other projects and change the asset version to the /user. Now you are happy.
Maybe there also could be one button to publish the user version of the asset.
So in the asset store other ppl have the option to also choose a user version with minor tweaks.
Just as an intermediate to make a pr to the asset git repo. (or the author of the asset deicides to cherry pick the changes form a /user version to a new version on master.)

Make local to project

copies the files from the .local/godot/assets folder into the current project... removes any special asset behaviour (like version...) and now you can mess with it however you want to. (also the folder loses it's custom appearance as an asset.)

This allows the asset store to also cover the use case of templates. And is a perfect choice for almost any art related asset. Since you can work on the assets to make them as you need them to be.
You can always just activate a bunch of already downloaded assets (lets say some basic prototype sprites) make them local and you have them imported.

random thought

This might even be a nice solution for your own local asset/resource lib/collection. A folder in .local/godot/assets lets you add it's content to any Godot project with a nice editor ui and without searching for it in your OS file manager.

[mhilbrunner]

I agree that both the AssetLib as well as the asset lib integration (both in the editor and with GDScript) need love if we keep pushing it as an alternative to including things in the core engine.

However, I personally disagree with how heavily this should be a plugin is pushed.

The first question we ask ourselves on a new PR shouldn't be could this be a plugin? but rather is this going to be used enough to be in core?.

There are problems with having things in plugins:

• People need to know what they're looking for in the first place. Godot is targeted at beginners. Beginner-friendly software is usually trying to have everything someone is going to (reasonably) need, because otherwise they first need to learn what they're missing and how they're going to get it.
  There is a reason beginners use Ubuntu, not Gentoo. Regardless of how nice you make that dependency system.
• Decision paralysis exists. If there is a feature in Godot, great, takes a second to use it. If there are 5 plugins for it, which am I going to use? For example, there are 5+ console plugins. Which one should a beginner choose, and why, and how much time should they reasonably spend on this decision?
• Plugins get abandoned, have less quality, lag behind when being updated. There is even less reason to trust a random plugin won't blow up on me than to trust Godot, where there are enough people behind it.
• See Unity. Their assets are mostly even paid, and it still sucks that for many things you need plugins first which eventually are badly integrated or get broken on updates - or discontinued, even though they are paid and usually can afford at least some support.
• Documentation: We have not enough of this anyway. Every feature not in core, I can't use in official tutorials/docs. Random single people for plugins will likely not provide docs, or only short examples, not the more valuable bit of providing demos or full-blown tutorials on how to integrate that feature with others seamlessly in a project.

Examples:

• Spring arm node. (https://github.com/godotengine/godot/pull/18822) Reasonably small addition, seperated into its own class so shouldn't break things, will be used by many 3D games, if not by every single third person game. Response: should be a plugin.
• Even for the RNG stuff, it was discussed to have it as a plugin. As if not 90% of games need a RNG somewhere.

Edit: To be clear, I'm all for avoiding bloat. But this is a balance, and I want to voice support for not having everything in plugins. Very game specific or not broadly used stuff, proprietary plugins, big dependencies, yes please, put those on the Asset Lib!

[dsaltares]

@karroffel This is an excellent post and I mostly agree with everything you said. I would, however, like to comment on this point:

Assets are saved with version associated, all stored in .local/godot/assets or something like that. They are no longer part of the project itself but available system wide.

This is similar to how Python pip system works. People then realised that the latest version of a package screwed the other projects you have in your system. virtualenv solves that problem. In Javascript's npm ecosystem, modules are installed locally to a project by default and system wide is an opt-in feature (npm install --global). You then check in your package-lock.json to source control to make sure everyone in the team uses the same version of every npm module and avoid conflicts.

System wide packages are good as they reduce bloat on people's systems. Version pinning is good because it helps the team keep a consistent set of versions and can help you identify a full collection of versions that work well together.

If we go the system wide route, I would suggest keeping version pinning. For example:

.local/godot/assets/fsm/1.0
.local/godot/assets/fsm/1.1

Project A could use fsm-1.0 whilst Project B uses fsm-1.1. Project C comes along and could re-use either. Users should be able to specify specific versions of plugins in their dependencies.json files or minimum patch, minor or major versions.

Once dependencies are downloaded, a dependencies-lock.json file could be generated and checked into source control.

[karroffel]

@mhilbrunner I definitely agree, I think the "this should be a plugin" mentality is pushed a bit too hard, exactly those situations sparked those thoughts. I think more common things like the SpringArm are worthy to be in the engine, as well as the InterpolatedCamera, which Juan wants to be removed, is insanely useful and cool! Even thought those things are only a few lines of code, they take a while to get right, so "this is too small" is also a bad justification for pushing towards the AssetLibrary.

My thoughts were mostly centered around "Well, the project lead said it's that way, so what can be done to make the new situation better?"

I would like if the heavy pushing of features as plugins would be lowered a bit, but either way I think the changes presented would help a lot!

---

@saltares Yeah that was my thought. I am a big fan of cargo for Rust. It operates pretty much the same. Libraries are installed in a system wide folder in all the versions that were ever needed locally. The description is in one file and a specific configuration is in a .lock file, it works pretty great!

I'm glad this sparked some discussinon already :blush:

[reduz]

Here's my view to the above points:

Pushing stuff out of the engine

Pushing stuff outside the engine is IMO the way to go. There are too many platforms where deploy size matterns (mobile, html5) and bloat is a big concern for a project the size of Godot, so we can't just go happy include everything.

Philosophy to me should be that something should be part of the engine only if ALL the criteria below are met:

1. It's used relatively often
2. It's difficult to do yourself
3. There is only one way to do this feature

This rules out spring arm, terrain and I hope it can help to, in the future, remove features such as InterpolatedCamera, Vehicle, etc. that I believe should be outside the engine.

The main concern about this is that, if something is not official it's not going to be maintained. This is where you guys are wrong.

No one said these things should not be official, we can have an official repository for them and they can be first party extensions. Demos are maintained this way and they are up to date, so i don't see why it wouldn't work for an official extension repo.

Extensions should feel like part of the engine

Regarding these points:

• make it easier to extend custom nodes, allow cross-language script inheritance

I'm very strongly against this. There is no practical benefit at all to doing this, and the implementation cost is very expensive. Sorry guys, I know it feels like a nice feature to have but weighting cost/benefit/risk to it is on the very negative side.

I also think it's positive that custom nodes or resources should appear as-if they were NOT core is a good thing for end users. Users are not stupid and this information should not be hidden for them.

If a new node is added from an extension, and is done with a script, this should be even shown in the create dialog.

• Script inheritance isn't suggested when attaching a script - override is the default for those cases.

This needs to definitely be done and would be a very welcome addition.

• add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason. Looking at how other engines work with this, it does not seem like having dependency management to me is so important.

• allow scripts to be referenced by class name and namespace to avoid super long paths in case of assets.

This is something we discussed before, but the conclusion we reached was that it should mostly be script language dependent (and not provided by the engine itself as a core ScriptDB), with a small usability enhancement in the new script dialog in case you want to use a global class.

[PLyczkowski]

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason.

I guess the add-ons should never update automatically in our case, but instead there should maybe be a list of installed ones with version listed and manual updating, so the user can request update and check right away if anything got broken.

[LikeLakers2]

This comment is separated into collapsed sections due to the large length of it.

Preamble

I came from Game Maker Studio to Godot because, despite having paid for GMS, I still felt I couldn't implement all I want without having to resort to some really hacky code or coding in C++. When I came to Godot, not only did I come to find that it was much easier to create new functionality (I've only found one thing so far that I could not easily code in GDscript, which is getting spectrum data for creating audio-reactive experiences), I was incredibly impressed at how little I actually had to code to get 95% of my ideas working.

Want a camera? Godot's got a camera that can follow a player, with little to no code from you. Want to make an animation in Godot? Not only can you make character animations, that same system can (almost) replace that cut-scene system you worked so hard on, only needing maybe five or so lines of code from you. The editor has even been used to make other editors like RPG in a Box. It's fantastic just how much you can create in Godot.

---

Now with that out of the way, hopefully anybody reading this can understand what I'm thinking coming into this as I try to respond to some of reduz's points regarding pushing things out of the engine. (This won't be replying to the "Extensions should feel like part of the engine" section as I have no thoughts on that currently.)

Removing stuff from the engine, and Guidelines

Pushing stuff outside the engine is IMO the way to go. There are too many platforms where deploy size matterns (mobile, html5) and bloat is a big concern for a project the size of Godot, so we can't just go happy include everything.

I can agree that bloat is a concern, but when I see what you're wanting to remove so far, I ask myself if it's really even worth it to have that stuff removed. Even if you remove nodes like Vehicle, InterpolatedCamera, etc... How much of a file-size difference will that really make? Maybe half a megabyte? Maybe a couple of megabytes? Even if it saves several megabytes worth, what's the difference between a 35MB download and a 45MB download anyways? The game assets will quickly fill that 10MB void regardless! Unless you plan on removing half the engine, I don't see the point.

If you really are concerned about saving as much space as possible, might I suggest making it easier for the end user to remove the nodes they don't need from the engine instead? Let the people who are going to use the engine decide whether they're okay with the bloat -- don't decide for them.

Now of course we can't just include every single suggestion given to us...

1. It's used relatively often
2. It's difficult to do yourself
3. There is only one way to do this feature

...and these guidelines are a great starting point for scrutinizing what should be allowed in the engine. There is no doubt that all three points are already being considered for every single feature request that comes through the issue tracker.

something should be part of the engine only if ALL the criteria below are met

But c'mon. The fate of a feature or node should not rely that heavily on being a core feature. Your suggested guidelines are great, don't get me wrong, but we shouldn't be using them as a checklist. That's what YoYo Games does when you suggest a feature on their forums, and is another part of why I was pushed away from their engine -- can it be made using an extension? Yes? Then our response is make an extension. Guess how many feature requests have been denied because they could be made into extensions instead, regardless of how reasonable it would be for it to be in the engine, of how easy in comparison it would be to implement, or of how the community feels about that feature.

What I'm saying is that you can definitely keep those guidelines in mind. You can consider them every time a feature is suggested. But whether all of them apply should NOT be the deciding factor in whether a feature gets in. Weigh the idea against those guidelines, consider how important each of those guidelines are and apply weight to those factors... Just because an idea doesn't seem like it's used too often doesn't automatically make it a bad idea (keep in mind you might not know the true scale of how it's used, even with several years of experience). Just because an idea is easy to implement ourselves doesn't automatically mean it's a bad idea. And just because we can implement an idea in more than one way doesn't mean it's a bad idea! Not to mention that maybe someone's suggesting idea X because they think other people would benefit from it.

Remember that this is an engine that bills itself as being beginner-friendly, and every time a beginner has to code something non-specific to their game that's not part of the engine, or go look in some asset library to get the code they need, or go some long route to get what they want out of the engine, it could push them a little further away towards another engine, or sometimes even away from game development as a whole.

tl;dr: Guidelines are great when they're used as guidelines and not checklists.

---

An official repository for first party extensions

The main concern about this is that, if something is not official it's not going to be maintained. This is where you guys are wrong.

No one said these things should not be official, we can have an official repository for them and they can be first party extensions. Demos are maintained this way and they are up to date, so i don't see why it wouldn't work for an official extension repo.

I personally disagree with this idea. Alongside with what I've said above (that removing it from the vanilla engine won't save you much on space), I have to ask if we're going to convert them all to GDscript, or if we're going to require them to re-compile the engine just to get those nodes.

• If we keep them as C++ and require a re-compile... why though? There's no good reason we should ever be required to recompile the engine unless we're changing a core part of it, or adding our own functionality. That's just additional time taken away from developing, because we're waiting for the compilation process to finish, alongside requiring the average user to install other tools they might not even use ever again just to get some first-party nodes. Now that sounds like bloat!

• If we convert them to GDscript, then that's going to come with all the limitations of GDscript. Limitations like the potentially overhead of additional child nodes (especially if you're asking the user to import this first-party node as a scene), having to use paths rather than class names to refer to their classes (though you addressed this, it seems, but I feel I need to bring it up), lower performance (regardless of how little the difference might be, it's still lower), etc.. It's a bit more ideal than C++ because you've gone from re-compiling to drag-n-drop, but...

Either option also means the files for these nodes will have to be imported into the engine with every new user (for C++) or into the project folder with every new project (for GDscript) that wants to use them. I don't know about everyone else, but that sounds like a hassle to me, and sounds like it'd make your first guideline ("It's used relatively often") very debatable. I'll try to explain why I think this.

By requiring additional effort to use those nodes, I'm fairly sure the number of usages of this first-party solution will drop, and this first-party solution really will no longer be used "relatively often" where it may have been before. That, in turn, will slowly increase the number of alternatives that recreate the same features when people stop learning there's a first-party solution -- when I'm fairly sure we'd prefer existing versions (in this case, our own) to be improved, rather than have a new one created with feature X added, a new one with Feature Y, and a new one with both of those features but not as optimized, and so on... It's creating this market for alternatives when people don't know there's a first-party solution. It's creating a mess.

Additionally, you say they'll stay maintained. That's great! But who's going to maintain them? What are our guidelines going to be for adding new things to this repository? And are we going to go to the effort of ensuring this repository of first-party extensions will always work, whether with a stable release or with master? Basically, is it worth the effort of what is effectively a separate project?

---

Summary

What I'm trying to get at here is that I don't feel, with what your message has told me, that the time's been taken to consider whether those are really the right ways to handle things. Honestly, and this is more than likely the annoyance within me speaking (so please forgive me as this will probably sound rude), it sounds more like "what reduz wants" rather than "what reduz thinks is best for the engine", with little to no consideration for "what the community wants". Sounds a lot like YoYo Games and Game Maker Studio all over again, at least to me.

[reduz]

by the way, I much prefer to implement multiple script per node support
rather than cross language inheritance. The former is just a couple lines
of code..

On Sun, 10 Jun 2018 at 13:51 PLyczkowski [e-mail] wrote:

My feeling about this is that it can either work fantastic, or result in a
complete mess and add-ons that break without an obvious reason.

I guess the add-ons should never update automatically in our case, but
instead there should maybe be a list of installed ones with version listed
and manual updating, so the user can request update and check right away if
anything got broken.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/19486#issuecomment-396063688,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z21jtcnUhRyhdyEEnaM_hfTACsxOiks5t7U6HgaJpZM4UhqDC
.

[Zireael07]

Multiple script per node has been attempted a couple of times and always failing.

[reduz]

what was attempted is the multiscript node, but it was a hassle

On Sun, 10 Jun 2018 at 15:01 Zireael07 [e-mail] wrote:

Multiple script per node has been attempted a couple of times and always
failing.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/19486#issuecomment-396068742,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z26u8Rqc1yL-BGX16CAx_iWsYEo1oks5t7V8CgaJpZM4UhqDC
.

[vnen]

I understand that assets should not look the same as the built-in stuff. But I also think they should be at a higher-class than user-made scripts. They should have a clear separation so it's easy to replace/upgrade assets without having to mine them out of your project.

Also, adding a layer for assets and let them integrate better with the engine to extend it is great way to keep the core small and clean, while easily provide new functionality via the asset library. If assets can register new types not only for nodes/resources, but also classes to be extended, in a way that is globally accessible (i.e. not path dependent), it would be a great benefit for frameworks to build on top of the engine.

Frameworks like Escoria could benefit a lot from it, and there could be other frameworks for things like RPGs, platformers, visual novels, etc.

All I'm saying here is quite abstract, but having a clear separation of what's built-in, what's third-party asset, and what is user code (however this separation is presented) will make the "keep the core small and provide extensions" go forward much easily.

[neikeq]

Cross-language inheritance sounds like a horrible idea to me.

[willnationsdev]

In the work I've been doing with the TypeDB in the EditorData, the I've been keeping a namespaced-typename to filepath HashMap that gets converted to a Dictionary in the ProjectSettings upon modification. In my branch, the GDScript parser and compiler are using this data to create global names for types. At this time, I've got custom type scripts showing up. The scripts can access and inherit them by name. What's more, this data is accessible to anything that can see ProjectSettings. So a CSharpScript seen by the Editor can generate a mapping that is entered into the ProjectSettings Dictionary, allowing types declared in C# to be accessible to GDScript. There's probably also a way to wrap the ProjectSettings Dictionary data in a C# Object that returns Script objects when accessed, that way C# would be able to access GDScript types by name too.

This technique keeps the shared namespaces out of core and purely within the Editor context, whereby languages that opt-in can prepare their own means of accessing the data.

---

I do like @karroffel's suggestion for being able to have cross-language script inheritance though (don't even know if that would be possible). Somehow automatically duplicate methods that delegate calls to the base script and also define all the same properties/signals/constants, etc.

I too would like a more streamlined access to content in the AssetLibrary, with the ability to keep some plugins for the editor overall and other plugins for just a single project.

The TypeDB system I'm making will also give editor support for...

1. Having all of one's custom scripts show up in the CreateDialog
2. Have extending a custom type script automatically make you extend that script when adding a script to the node
3. Have removing a script that extends a custom type script reinstate the custom type script after removal (so you can't remove a custom script - you must use the "Change Type" command in the Editor instead).

Also, all of the interface changes I'm making would keep the custom type scripts clearly understood as being scripts as well as keeping the script in question accessible. Users will always know what is a script and allow them to get to the script, but it still makes them feel like in-engine objects to an extent. Even within GDScript, the names are not globally accessed, as if they were a core type, but rather extracted from a global Dictionary called Scripts. So far I have it working with custom type scripts, but I would like to eventually expand to having the Dictionary register custom type scenes and any typical script or scene available in the project too, likely via the Import system if the script language doesn't have a simple means of acquiring the namespaced typename.

[QbieShay]

As I have been part of the thinking process of this proposal , i would like to point out few things:

My vision of Godot

TLDR: I imagine Godot as a tool that can be highly extended with assets and plugins in a very easy, modular and beginner friendly way.

The motive behind the rejection of my SpringArm pr are more than reasonable, and that's why @karroffel and I discussed about the direction that the engine could take to keep being lightweight and, at the same time, being user friendly.

What I'd love to see is a core engine clean, small and fast and a well integrated ecosystem of addons and plugins.
As ArchLinux has more than one package repository, I think it would be great if Godot had an "official"(mantained by Godot) repository for core functionality (and Nodes that are now core) and another repository for community. In the Godot core "extension repository" I would put all the nodes that are a pain to code but not enough used to be shipped with the vanilla version of the Engine.
Side note, what it's often not taken into account is that some features might be easy to code but might also require a lot of research before laying down these few lines.

I imagine users being able to download plugins and assets that feel like they are "extending" the engine itself: depending on what you are downloading, you have Godot-fps or Godot-platformer or Godot-racing.

Godot could be the first engine to feel really Unix-like and modular, surpassing even Unity with its editor extensions. Many of you know how good it feels when your Linux system is not "that distro", it's your very own and customized system with your very personal functionalities.

Advantages

TLDR: this approach would allow Godot management to manage separately hardcore technical core features and more user-friendly plugins without worrying too much about bloating the engine

What is good about this proposal is also the separation of concern and the handling of the contributor. I feel that, for now, it's not clear what is okay to be in the core and what is not. There is no clear rule nor statistical data to speak about. Some might say a certain feature is widely used and some other think it's not.

Dividing Godot in different "rings" gives the opportunity for a more fine grained management of PRs and features, and can give more time to @reduz and other very technical developers to focus on hardcore technical stuff, while another team would manage the extension ring, taking more into account user feedback and usability rather than having to face engine bloating, since the "extension ring" would be bloat on demand.

About multilanguage inheritance

As Godot is growing, the asset store can easily become a flaming mess. The strict division between scripting language can literally destroy any attempt that people make to sell assets and grow an ecosystem on the asset store, killing many opportunity for freelance toolmakers to make and sell tools for Godot.
Language bridges need to be more complete if we want any chanche to develop a lively ecosystem.
For now cross-language inheritance sounds great to me, but i am not technical enough to say anything on this matter. I just want to point out a problem that will likely present itself soon, expecially since there is such a huge community around C#.

Note about usability

I've been using Godot for almost a year now, and lately i feel like it happens that people that decide about features are not using the engine for anything bigger than small demos. @LikeLakers2 pointed out a problem that i feel is shared by some of the users that are not deep into the developing core team.

[swarnimarun]

I think that what is considered bloat is a trivial topic in the greater scheme of things, it's not like the decision we make now is what we will need to abide by for the whole life of Godot. Being open-source allows us to make changes whenever we like the only problem is that the more we wait the more heavy the decision of change is going to be.

Now that's also a discussion for another day. Right now in the present the facts are,

1. Godot is a compact, fast and portable game engine. And I like it alot but,....
2. Godot's executable is around 45-60MB depending on various factors. But the export template can be as large as 300MB. Now that's a big blow to the first point.
3. Godot has lot's of amazing features that make it a treat for the beginners like Camera follow, Parallax BG and Vehicles, this is a big plus over other similar game engines. As it makes creating games much simpler in Godot. And makes Godot a home for lots of newbie devs, who don't have the skills to write their own implementations for such tools.

Now what will happen if we remove features that some people consider bloat. Well the engine will surely be more compact but is that required? I don't see a problem with myself downloading a 100MB Engine if it fulfils my needs better. But what about exporting, won't a 100MB engine export bigger games?

Probably, yes, but in a world where desktops don't seem to use anything less than 256BG SSD + 1TB HDD and even Phones have space starting from a minimum of 16GB all of which which is more than likely to increase, would it be that BAD?
I mean to say we don't need to remove anything yet, we just need to start managing a few things better and that should be it.
Features should have some sort of importance list to be decided if they are supposed to be in the core or not but that doesn't mean that it becomes a law but rather a rule of thumb.

And for the nit-pickers we could remove some extremely unused tools but that's it. Features don't bloat an engine unless we are careless. And even all the bloat in Unity has not stopped people from using it, has it.
And I absolutely don't mean we should not care about bloat but being careful and being paranoid are different matters all-together.

As for multi-language inheritance, I don't really see a use for it here. Sure we should be able to interact with scripts of multiple languages but inheritance! That's likely more work than I could imagine with benefits that just don't seem to be worth it.
Whereas, multi-script might be just the solution for lot's of problems from, addons creation to better modularity. Although that is my perspective.

[nobuyukinyuu]

There's something to be said about limiting the number of node and resource types that come in the box. By squirreling away any lesser-used assets to an external download, you make it 1. a lot easier for new people to get a grip on the system just by "Messing around" without being overwhelmed by the breadth of features (decision paralysis), and 2. don't end up creating as much legacy baggage if it turns out some of the more narrow use-case-scenario assets end up having a workflow that isn't as good as something that comes afterwards with a different API.

When the latter happens with built-in's, you're stuck with them. When it happens with downloadable assets, the most popular ones tend to become de-facto standard over time, but don't have their fate permanently entwined with that of Godot. That means they can fade into obscurity without having to worry as much as breaking expected core compatibility.

[neikeq]

The cross-language inheritance point needs some explanation. Not only I consider it to be a bad idea, but I don't even see how it is remotely possible.

1. Godot's executable is around 45-60MB depending on various factors. But the export template can be as large as 300MB. Now that's a big blow to the first point.

300MB the export templates? Do you mean with assets?

[karroffel]

@neikeq the thing is that if your whole project is in C# but you are using a custom node downloaded from the AssetLibrary then that Node is just a regular node, most likely with just a GDScript attached.

If you want to "extend" the node, you can't! I guess you can add a child node to add new functionality, but that way it's not possible to override things.

---

Implementation

Currently the scripting languages use a really similar pattern for inherited calls which could be generalized.

https://github.com/godotengine/godot/blob/76875ba145c5745033d0a1c9fda2f4349e2509b3/modules/gdnative/nativescript/nativescript.cpp#L696-L712

https://github.com/godotengine/godot/blob/76875ba145c5745033d0a1c9fda2f4349e2509b3/modules/gdscript/gdscript.cpp#L638-L651

https://github.com/godotengine/godot/blob/76875ba145c5745033d0a1c9fda2f4349e2509b3/modules/mono/csharp_script.cpp#L1175-L1191

The pattern here is

Class *c = bottom;
while (c) {
    if (c->has_method(m)) {
        c->call(m);
        break;
    }
    c = c->base;
}

So it starts at the bottom of the script inheritance tree and then tries to call the upper levels if there's no such method found.

If each script had a Ref<Script> parent_script instead of the language-specific inheritance structure, this could be generalized to

if (this_class->has_method(m)) {
    this_class->call(m);
} else {
    // method not declared here, go to base class
    parent_script->call(m);
}

---

This might not be a good idea after all, but not having the option to override stuff cross-language might be a problem over time if we push the AssetLib more.

[swarnimarun]

@neikeq See this,

[akien-mga]

@swarnimarun That's export templates for all platforms, all supported arches, debug and release, their size is completely irrelevant to the "bloat" of the engine. We can decide to support only Linux X11 64-bit release and this file will be 20 MB...

[swarnimarun]

@akien-mga I was not intending to call it, bloat.
But a new user will download it full-size they won't be experienced enough to know that they can build separately or download for a specific platform.

How is it irrelevant when this is what we provide to beginner users?

[Zireael07]

@akien-mga: Export templates size matters since your argument for removing nodes was

deploy size matterns (mobile, html5)

[akien-mga]

@swarnimarun @Zireael07 Yes export templates size matter individually, but my point is that measuring the size of a bundle of templates doesn't give us any indication. The size of one template (e.g. the wasm release template, 3.4 MB zipped, or the Android release APK, 23 MB zipped with armv7 and x86 libs) is much more relevant than "downloading everything needed to develop takes 350 MB" (+ maybe 10 GB for Visual Studio if you need to build something...). What matters if what size a minimal game will take on platforms where size matters (web, mobile).

[QbieShay]

Although i think that deploy size is really important, i also think that what needs to be taken into account is also developer time. For example, Unreal has literally any feature you could be looking for, but the fact that there are so many makes the development process highly inefficient because every time you need to do something with it, you first need to figure out which builti-in functionality you need to use and then learn how to properly use it.

This is something i really love about Godot. You can code your feature straight away and then someone will come and tell you "yo, did you know that we had this built-in thing you could use?"

The problem when an engine has too many features all together is that developing with that engine becomes a huge tiring experience. Yes, it's nice to have all the features you need but it's a gigantic pain to go through all of them to figure out what you actually need. That's bloat. And that's why we are proposing to have on-demand well integrated features.

[Ranoller]

add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification.

This is the same that losing portability. My plugins are part of my project, i need that in whatever computer i´m working and i need the concrete modification of every plugin for every project... I don´t see any benefict in have my plugins stored in .local/whatever. This is kind of obfuscation my projects.

Actual system of res://addons is good... free for the user and portable around computers. If there is no complains about this system, is necesary to change that? There are a lot of other things that users really complain...

[QbieShay]

@Ranoller you would be able at any moment to copy the plugin into your game folder.
Anyway, in that case your project.godot would list your plugin as a dependency and it would be downloaded to your new computer with the correct version

[Ranoller]

Downloaded? My plugins are inside the project that is inside the pendrive, and are part of the git repository ... I don't understand proposal... Plugins change the same that other code... I change more code from plugins that gamecode, i don't wan't to number plugins by version and work outside from the project... Every one line code change will need to manually change every plugin in every computer?... Really plugins feels like part of the project... Give plugins out from git repository is nonsense to me, but i really think that i do not understand something about that...

[karroffel]

@Ranoller the idea is to move those plugins outside of the project that you don't need to modify. For example, the InterpolatedCamera will just be a node you want to use as is, not change the code of it.

The same could be the case with editor-only plugins like TODO lists or git integration etc. Those are things you want to use but not change.

This is a lot like a package manager and libraries work in every language that's not C and C++. You have a file that specifies all the dependencies and then the package manager will fetch those dependencies. This happens even when you change PC - the package manager will download any not-yet-existing packages.

When you want to modify something, then you bring the code into the project. Then it's in your git and the project with all its source code etc. But that's not always the case.

My proposal is to have things outside the project by default until you actually need to modify something.

Currently many assets are mostly templates that you do need to modify, having such a system in place might change that so things are more general purpose via APIs, not via source modification.

[neikeq]

@karroffel But how is cross-language inheritance going to work in each language? It may be easy for GDScript, which is tightly coupled with the engine, but it's a completely different story for general purpose languages like C# or other languages added on top of GDNative.
I seriously can't see how this could be achieved, even less how it could not do more bad than good.

[neikeq]

IMO it's not a good idea to modify the content of an addon your project depends on. If any modification is needed, just copy the content outside the addon folder and do your changes there.

EDIT: or "fork" the addon 😜

[QbieShay]

@neikeq What you modify is inside your project. When you want to modify an addon it is copied inside of your project and only locally modified.

[PLyczkowski]

@karroffel An idea of a project plugin and an editor plugin may be needed to avoid confusion, with clear separation of the two.

[neikeq]

@QbieShay What if there is an update that modifies the files you modified?

[karroffel]

@neikeq Script inheritance isn't THE SOLUTION to this problem, it would just be one. There's currently not a good way to override functionality of a node with a script attached when you want to use a different programming language.

If we had an option to override methods in the Script API then that would be good too! (basically externally defining new methods and overriding existing methods).

---

@PLyczkowski that's a suggestion that would be useful too, but what I propose is not limited to that - and for a reason.

Let's take the InterpolatedCamera example again. I will never touch the source code for it, I will only use it. Having the source in my project would be wasted. Also most projects I make are all 3D, so I would probably have that code copied in all my projects.

On the other hand, a git integration might be a super useful to add in custom behavior that's hard to express with git hooks, so customization there is desired.

There's no black/white line when something should be inside the project and when outside, editor or not.

[karroffel]

@neikeq

What if there is an update that modifies the files you modified?

In that case you would lose versioning information, so you can't update easily anymore. You should use a tool like git for those cases then.

The idea is

• outside the project: easier to update, easier to have cross-project
• inside project: customizable, loses asset status, no automatic updates

Basically they become just regular files once you copy them into your project.

[Ace-Dragon]

Reading this, I can say that avoiding bloat and minimizing the file size of games is a valid one. Allow me to throw a few ideas in here though.

• Godot could use a system that strips the game of code used by nodes it does not use (ie. most 2D nodes with many 3D games and most 3D nodes with many 2D games). This will reduce the pressure to not accept new node types to keep game file size down. Unity users for example are already using something similar to great results.
• Godot could perhaps have a built-in asset browser that connects to a plugin page on the Godot site. You pick which custom nodes you want and it will install automatically to the correct location. Plugins that are not used are stripped when the game is packaged.
• Perhaps even, popular custom nodes could be bundled with Godot builds (like Blender does with addons). Custom nodes should also be marked as they may usually be higher level and may, to an extent, be more of a black box than the more generic nodes.

My 2 cents anyway.

[willnationsdev]

@Ace-Dragon

1. I think your first point would be really cool to have as an engine-wide feature. If you're exported game doesn't use a Node or if a Node is not used by any of the Nodes you are using in your project, it'd be cool if it were possible to automatically exclude them from the final binary. Outside of that, you currently just have to rely on switches and recompile the engine. :-/

2. This is basically what the Asset Library already is. The issue is that it doesn't have any concept of dependencies, directly drops things into your project directory, and doesn't have any means of maintaining a given set of plugins system-wide for every project you make.

3. Custom nodes are always explicitly marked, so no need to worry about that (they have a script icon next to them because you make custom nodes using a script to extend it). Because they are scripts, they don't ever get shipped with the engine. They would be part of a plugin of some sort, so if anything, you might have something more like my godot-next repo that is a "node extensions" repository for collecting stuff. It would just be a more "official" capacity more than anything (that repo could also use a lot of work, lol).

[Ranoller]

Ok, slowly but catched... If i maintain the addons inside the project i could work in the plugin code like in another scripts without losing git integration with current project and without losing portability... But if i want to convert any of the plugins in a tool for other developers or in a general tool for all my projects i share it in a form of external package that can be actualiced with the external git repository of the tool and changed to other people with the asset library... If it is, for me is ok... If other developer uses the tool, and need their personal version, he/she need to store inside the project and in this case the plugin doesn't actualice (don't lose their own work)... It is?

[willnationsdev]

@Ranoller Yeah, I think you've gotten the gist of his proposal.

[reduz]

Some feedback on what you guys are discussing

1) If this was an actual problem that needed solving, I much prefer supporting multiple scripts per node (which is just a couple lines of code), rather than cross language inheritance (which is madness). I still don't think this is a problem that needs solving.

2) I don't like the idea of a package manager and system wide install of add-ons. I think it's a terrible idea and nothing meaningful at all is gained from this. I've seen how traumatic is even for Unity developers to update a version of an asset in large projects and see everything break and spend days fixing it.

3) I don't like the idea of package management itself. I've seen Unity users hacking away what they get from the asset store to no end the vast majority of the time. Too often they just use it as a base, then work on it and customize it to their own uses, because the asset you get is only 60/70% of the way where you want it to be.. or they just wanted to see how it's done and then do their own thing later.

I think nothing really is gained in-real-life from the proposals being discussed, so not planning to add much to this discussion myself.

[Zylann]

How much engine size can we reduce if we allow people to export without any 3D stuff in a simple way? Many games (half? more than half?) don't require 3D features but they still sit there in the same binary. I know it can theoretically be recompiled yourself with a macro on every platform though, with a bit of investment...
Also, following this idea, if modules were just "internal" dynamic libraries that are part of the official engine's distribs, adding more features to the engine would be less of a problem in the sense that they can be opted out real easy (just that), however using dynamic libraries like this could be a bit constraining for some platforms (mobiles and HTML5 mainly). So for modules I would understand that GDNative is preferred (oh wait... it has the same problems^^).

@reduz how would multiple scripts work? And how would it contribute to engine extension? Did you explain your idea in an existing issue? (asking because it's probably too much to explain it in this thread)

[starry-abyss]

add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification

No. I'd like them in the project directory for robustness's sake. But maybe with a way to update picked ones manually with package manager, and maybe with system-wide caching of them.

As for bloat concerns, I'd appreciate more control over _modules_ instead, and this was already voted on January 2018 Patreon poll:

Roadmap Priority: More options to opt-out compiling parts of the engine to create smaller binary sizes [ Expected for 3.1 ]

[rminderhoud]

It's hard to comment on this proposal since it touches a variety of things and people are commenting on different aspects and raising different issues.

Overall I think the proposal for the "dependency manager" sounds good on paper but I think the implementation will not net any benefits that simply vendoring addons in the project folder won't bring. It will require incorporating a complex mechanism for dependency management to solve 1 case of "easier to install". This project already has 3000+ issues open with a small team of contributors. Introducing something as complex as dependency management is a footgun in my opinion.

I agree with @reduz that the core should be minimalist. In my opinion anything and everything that cannot be classified as "core" should be incorporated independently. As a user I expect to Godot to be a multi-platform game _engine_. If I want the godot _game framework_ (vehicles, interpolated cameras, terrain, player controllers. etc.) it should be available as a first-class add-on (or module?) independent of core. I think improving the addon story helps with this but a full-on dependency manager and automated addon manager seems to be pushing it for low net benefit.

My final comment would be to be wary about catering to "beginners" and forsaking "power" users. Godot has been getting the attention of more advanced users due to its powerful features (modules, gdnative, gdscript, editor, etc.) and flexibility. It's these kind of users who put godot to the test, push its boundaries, explore its api and often contribute back to the project. These are the same users who can create symlinks to a directory in their file system to re-use their addons across multiple projects.

[vnen]

If the idea is to remove higher-level nodes from the core, then those nodes should be readily available and easy to find for download. Ideally we could maintain them as "official", so they have better visibility (just like the in-engine nodes have). Then official assets could be used in demos and tutorials without much problem.

The advantage of outsourcing those is that they can be improved and released without having to wait for a new engine version. So bugs could be fixed right away, since for an extension is quite simpler to pack up a new release.

I guess in the end package management would be quite a hassle most of the time. Even in the case of a new version of a custom node you added to your project, updating it might cause many other issues (even with semver, since you might have worked around a bug that is now fixed).

OTOH for some editor plugins it might make sense to have then out of the project and have them updated often. My Tiled importer plugin is an example. You most likely don't want it shipped with the game and want to have the latest version which supports more features and have bugfixes.

This applies even more to other editor extensions, like git managers and time trackers. You might not even care if your team mates have the time tracker plugin or not, it's not related to the project. Yet, you have to put in your addons folder and enable in project settings, then be careful to not commit it with the rest of the game. Those are probably the minority of the plugins, but still having system-wide plugins that are not tied to the project have a real use case.

---

I believe the middle-ground would be to create a package manager as an external tool, not have it integrated in Godot. It doesn't even need to be official (though it could be "endorsed"). If some changes are needed in the base engine to support this (such as having global plugins), then it wouldn't be much of a problem to add, especially if they are editor-only.

Then those who think it's beneficial can use it. If it becomes widespread and everybody is using it, we will have both the usage statistics and feedback on usability to decide whether and how to integrate it fully in the base engine.

---

Now, cross-script inheritance for me is a no-no (if that's even possible). Multiscript is kind of weird, but it would effectively replace cross-script inheritance in a cleaner way. Though it raises a few concerns, like: how to call the function of another script in the same object? I guess that's another discussion.

Referencing scripts by name OTOH sounds really nice. Ideally, all resources could be referenced by name so paths are not a problem to solve if you decide to move things around.

[QbieShay]

@vnen the only problem i still see is the incopatibility between addons and projects. How would you address that problem?
Because if there is an official extenion lib maintained by Godot, either every node is shipped in every possible language, or i see cross language inheritance the only way that we can keep user friendliness.

[Zylann]

@vnen if you reference all assets by name then it will be a conflicting mess. That's more suited for scripts since only programmers have to come up with unique names for their classes to type and use them (or more rarely, resources). For the other stuff, if we get rid of paths, it would need this https://github.com/godotengine/godot/issues/15673, unless you want also artists, designers etc to have to name all their assets uniquely (which still comes with the issue of renaming).

[willnationsdev]

@Zylann @vnen The TypeDB I'm writing should allow people to optionally register a name for any asset, theoretically. I'm currently only using it for scripts, and plan to also do it for custom type scenes, but you could also modify it to work with resources. The logic currently just has a HashMap each for scripts and scenes, but maybe I could generalize the logic to handle any kind of asset, all in categorized HashMaps.

---

I can respect reduz's concerns with the system-wide stuff, and those are certainly concerns with plugins that would modify the engine's offerings, but when it comes to features for the Editor specifically, like vnen is describing, I can definitely see that being useful in a relatively safe way. If people are adding plugins to grant the Editor more features, those aren't things they will typically crack open and modify themselves. They'll just use it as-is, or if they do make modifications, it'll be in order to PR those changes to improve the tools probably. You aren't adapting them to a game after all. As such, if it's for a plugin like this, I think we should have a means of easily granting those features to the engine when you start a new project (somehow).

---

One possible method that could make things easier without needing a dependency manager would be to simply not share dependencies between plugins and instead have them duplicated within the project, that way versioning is a non-issue. It would make an individual project bigger potentially, but if it's for content that is game-relevant and people will be tinkering with it all anyway, as reduz says, then they can just move the dependencies. What we would need in that case is a way for plugins to declare WHAT external-plugin-dependencies they have and give users the option of customizing where they look for that dependent plugin. That way, if they DO choose to move a plugin and have other plugins look for it in the same location, they can customize that. But then, by default, everything would just work out of the box by managing its own dependencies. This case would probably also be easier to handle with the whole plugins-as-sub-projects proposal (#19178).

This would keep us from having global dependencies that require a dependency manager at all (reduz's issue number 3). And we could couple it with a feature that allows users to create a favorited list of assets from the Asset Library that they can choose to automatically install into their projects when a new project is created (so everything is still stored locally, but it happens automatically and they don't have to manually go out, find all their preferred plugins and manually download and install them). This would fix the usability issue that @karroffel wants to deal with in regards to easily extending the engine with identical assets when creating a new project.

For more editor-specific plugins that wouldn't need to be duplicated, perhaps people could just flag those "favorite" plugins and they would have symlinks made for them automatically, storing the plugins in a user folder of some sort. So, maybe have "Editor Favorites" and "Project Favorites" for "I want this in every project, store it in a shared location and make a symlink" and "I want this in every project, duplicate it", respectively.

Combine that with the TypeDB stuff for naming scripts / customizing custom types, and I believe it would properly address all of the mentioned problems except for the cross-language script-inheritance / multiscript issue which is something else entirely.

[starry-abyss]

So, maybe have "Editor Favorites" and "Project Favorites" for "I want this in every project, store it in a shared location and make a symlink" and "I want this in every project, duplicate it", respectively.

Maybe then it's more convenient to have "bundles" of plugins - user-created officially approved + user-created local. So instead of "Editor Favorites" and "Project Favorites" we have "First Person Shooter", "Voxel Block Game", or just "My stuff". Pretty much like e.g. Unity starter assets, but no need to bloat them with textures etc, different types of Godot mascot faces should be enough :D

This also works as a package manager, where package manager now is the human curating plugin bundles.

[willnationsdev]

@starry-abyss The plugins that people make now already have the potential do be "bundles" of other plugins (the user or plugin developer can just move them in themselves and merge them together). The goal here is to have a functional difference between the categories where the editor actually performs usability-improving operations on behalf of the user so the user doesn't have to do things manually.

To this extent, we already have the "humans curating plugin bundles" feature. The problem is with automating tedious tasks while keeping it customizable with efficient productivity in all use cases.

[vnen]

@Zylann

if you reference all assets by name then it will be a conflicting mess. That's more suited for scripts since only programmers have to come up with unique names for their classes to type and use them (or more rarely, resources).

Is it so much of a big deal to properly namespace stuff? But it could be UIDs, it doesn't matter much to me as long as it's not path dependent. Even the path is just like a name, since resources are remapped to their imported counterpart...

---

@QbieShay

the only problem i still see is the incopatibility between addons and projects. How would you address that problem?

If this is really a problem, then multi-script is likely the simplest solution. It was removed mostly because of the discussion in #8546, but I think this can be solved without much effort.

[kakoeimon]

I believe that the problem of multiple plugins doing the same thing is virtual. We already have a star system in the assetlib, this system is just this, an indicator of what is commonly used and preferred.

The thing that it is overlooked but in my opinion is one of the reasons people are asking for additions in the core engine is this : Who is going to compile this thing for all the platforms?

Of course for a gdscript plugin this is not a problem, but for a Cpp plugin this is a huge problem.

If we do not solve this problem then the discussion of what is going to be inside the core engine will transform to, which plugin is going to be marked as official, and by marking plugins as official will ruin the star system, cause people are going to just star a plugin cause it is just an official one.

So in my opinion we have to solve this problem first, and I have no idea how. Is it possible for example to compile for all the platforms through Linux? If so then we may use something like Ubuntu Imager : https://www.maketecheasier.com/create-linux-distro/ to create an already setted up linux environment for the plugin makers and GDNative users.

[Calinou]

Is it possible for example to compile for all the platforms through Linux? If so then we may use something like Ubuntu Imager : https://www.maketecheasier.com/create-linux-distro/ to create an already setted up linux environment for the plugin makers and GDNative users.

No (macOS and iOS builds can only really be compiled from macOS), and I don't think forcing people to boot a live Linux OS to be able to upload GDNative libraries to the asset library is a good idea.

However, it is possible to compile for all platforms using Travis CI and AppVeyor, which you can set up for free on public GitHub repositories.

[vnen]

If we do not solve this problem then the discussion of what is going to be inside the core engine will transform to, which plugin is going to be marked as official, and by marking plugins as official will ruin the star system, cause people are going to just star a plugin cause it is just an official one.

Well, the star system was never implemented, so you can't rate anything. For the official plugins, it's not that we're gonna pick one and mark as official, we're the ones creating and maintaining them.

[eon-s]

Built-in nodes for basic things are one of the best things of the editor, if important nodes or resources are going to be removed, the Project Manager will need templates to automatically download the common nodes.

This proposal also have risk to turn the engine on another "always online" and "junk store" engine, if done, many other things will need to get official support (bug fixing) and availability (version update).

And the store must offer official bundles for offline use (with offline store support in the editor).

[vnen]

if done, many other things will need to get official support (bug fixing) and availability (version update).

TBH, maintaining things in the engine or in the store is pretty much the same. The vehicle nodes, for instance, have few bug reports because almost no one uses them, and even the bugs that were reported never got a fix. I find outsourcing those kind nodes much better because it's easier to get people to try to fix things without having to mess with the engine source, so there's more potential contributors.

If an engine node has an issue, people will open a bug report. But if it is a plugin with GDScript code, it's quite easy for the person to change the code and test it, likely contributing the fix back to the community. I really see this as gain. But only if those external nodes can have an easy integration, which is not the case currently. That includes good visibility and integrated documentation.

And the store must offer official bundles for offline use (with offline store support in the editor).

Yes, and it should be easy to install offline packages as well.

[Two-Tone]

@reduz

I don't like the idea of a package manager and system wide install of add-ons. I think it's a terrible idea and nothing meaningful at all is gained from this. I've seen how traumatic is even for Unity developers to update a version of an asset in large projects and see everything break and spend days fixing it.

Okay, but why? What are the actual limitations and issues that you see from it? Saying "I don't like it" without saying why isn't going to convince people that it isn't a good idea; you should be giving at least some explanation other than the vague reasons mentioned.

[rminderhoud]

Okay, but why? What are the actual limitations and issues that you see from it? Saying "I don't like it" without saying why isn't going to convince people that it isn't a good idea; you should be giving at least some explanation other than the vague reasons mentioned.

Assuming global "addon" registry on system:

• Download addon v1.3 from online add-on system. Addon is installed to ~/.godot or what not
• Version gets associated with my project at ~/my-project in some kind of local store (file, sqlite, etc.)
• I upgrade to v1.4. Now my addon manager has to know that 1.3 is not used by any project and clean it up. So the manager has extra functionality to track dependencies and what projects have what dependencies. Alternatively nobody adds this functionality so my local machine gets polluted with multiple versions of same addon
• I decide I want to move my project to ~/big-project/game. Now I need a way to update the reference to my project in the addon registry. How do I do that? Do I have to run the addon manager separately? Manager register new_proj_location?
• Decision is now made to store dependencies locally ala requirements.txt python style. Great, problem solved. Except I'm back to not knowing which projects depend on what global packages, I have to scan system for godot projects or register each project still to avoid piling up versions.
• Fine, I just work around the issue. Manually clean up dead packages from time-to-time. So I'm working on VehicleBodyAddon for my project using v1.5. Ugh, It does F * M for Accleration calculation but I want to do F * M * 5. Let me quickly modify the code...Whoops, there goes all my other projects running that addon.
• Let me create a new addon that depends on the other addon. Hmmm now I need to register that my new custom addon depends on the existing one. Now I need to extend the manager to not only download addons from remote location but also to work with local paths or git repositories or something.

Assuming "vendored' addons:

• Download addon to addon directory
• Updating overrides addon in directory

[LikeLakers2]

So the manager has extra functionality to track dependencies and what projects have what dependencies. Alternatively nobody adds this functionality so my local machine gets polluted with multiple versions of same addon

This is exactly what other ruby's gem command does, and nobody's complaining there since they have a command to clean up old versions of gems (ruby's term for addons) that have no other gems depending on them. Plus, why do we need to keep the old version when, if we're updating, the new version's probably going to be mostly compatible anyways? Push come to shove, the user can get the old version again if their program absolutely depends on the old version, which it most likely will not.

I decide I want to move my project to ~/big-project/game. Now I need a way to update the reference to my project in the addon registry.

You don't, really. You're thinking way too deeply into this when the add-on manager could just... use the project list from the project manager, and ignore any paths that don't exist. Of course, that's only if it absolutely needs to keep track of projects and their requirements, which I don't see the need for when we could just keep the latest versions of whatever add-ons we have downloaded like literally every other add-on manager out there.

Decision is now made to store dependencies locally ala requirements.txt python style. Great, problem solved. Except I'm back to not knowing which projects depend on what global packages, I have to scan system for godot projects or register each project still to avoid piling up versions.

A Python-style requirements.txt would be a great option for this sort of add-on system. That way if the project absolutely really needs some old version (but again, you're often going to want the latest version anyways), we can specify that, and if the add-on manager ends up deleting that old version, it can know to re-download it later.

So I'm working on VehicleBodyAddon for my project using v1.5. Ugh, It does F * M for Accleration calculation but I want to do F * M * 5. Let me quickly modify the code...Whoops, there goes all my other projects running that addon.

Or use a copy of the add-on local to that project? If it really requires a change that only applies to one project, and you really really don't want to make a copy local to the project, just make a .gd file that extends it and makes the change(s), and then use that new GD file in place of the one from the addon.

Let me create a new addon that depends on the other addon. Hmmm now I need to register that my new custom addon depends on the existing one. Now I need to extend the manager to not only download addons from remote location but also to work with local paths or git repositories or something.

Git sub-modules. Another requirements.txt (like what python does). Or just tell the user that it requires Add-on X. It's not as hard as you're making it out to be. Most of Godot's users are not dumb. They know what "This add-on requires: Add-on X Core, Add-on Y, Add-on Z" means.

[willnationsdev]

@rminderhoud If your recommendation is then to store a copy of a project's addons in each project as needed (so it has only the version it needs and is free to modify it without causing issues), then perhaps my earlier suggestion could deal with the core problem surrounding this Issue? Namely, create a utility within the EditorSettings that allows the user to specify what addons they want to have automatically installed when they create a new project.

We could even create a similar utility that allows people to form hard links to a user:// location for storing certain ones (especially for addons that add editor features). If the user chooses to do that irresponsibly and screws over their project, it'd be their own decision. The default method of just copying all of the assets during project creation could be very useful.

[Ranoller]

Add a new plugin system should not be incompatible with respect res://addons like it is now... And this is because:

The actual system is great...

I repeat:

The actual system is great...

No previous complains about that point.

[chanon]

I'm late to this discussion, here's some thoughts about the proposal:

• I personally don't use many 'assets' yet and feel the addon folder as-is is easy to work with.
• I understand what cross language script inheritance would do and what problem it would solve (allow a C# developer to extend from a GDScript script to add custom functionality using C#) (not that I have any desire to use it) but I'm not sure what multiple script per node would do. Oh, so instead of the C# script extending from the GDScript script, it would be able to call it? Which script would be the main one? How would the C# code change what the GDScript script does?
• Allowing scripts to be referenced by class name / name space (TypeDB) may be useful. I am fine with res://paths, but it is true that inheriting from something in addons makes for very long path names. Not a major issue, but a nice-to-have - would probably be required (or preferred) though for cross language inheritance.

[chanon]

A simpler design for the asset management I would suggest is

• Having a dependencies.json (or just saving the data to project settings) is useful, just so I know which version of the asset I downloaded and if there is an updated version. I think that is needed info.
• No, do not let assets be able to define dependencies. I don't think Godot assets will have such high amount of dependency on each other like npm modules have. Let the asset developer state in documentation if it really depends on another asset and let user install manually. This will simplify things and we won't have to create another dependency validation thing which is code bloat / more code to maintain.
• No, do not save assets in a 'global' location. Keeping them in the project addons folder is already good. It allows them be committed with source control with the project and makes them easy to manage. Even npm node_modules saves a project's modules inside the project.
• Now, improve the asset browser so it can list which of the installed assets has been updated and let you update them if you wish.
• Now, it does make sense to be able to have 'global' addons for those addons that are just editor improvements. So normal addons still go into the project addons folder, but for editor related addons, you have the option to install them globally. This also moves that code outside of the project's addons folder so it doesn't go into release package. (This is similar to npm install -g for development tools)

So just a few improvements to help track updated assets and being able to have editor addons installed globally.

[Zylann]

Assuming that we don't handle dependencies, I feel like there could be a problem of discoverability / ease of use for users. Let's say a new user wants to try a plugin or "asset package" having a dependency. The package is shown in asset lib, downloading it, then it fails to work, likely with some "internal script failed to load" or something like that. That's cryptic.

• Either every developer who dares having at least one dependency will have to setup boilerplate code to show the user it needs to install a dependency (even if the package is just a demo for that plugin!)
• Or have the assetlib at least indicate that?

Also +1 for easy update of assets, no need to browse them again in the library.

[chanon]

It is true, however once the user sees it doesn't work, they will immediately look up the documentation and see that they need to install the other asset(s).

At this point it is only an imagined problem. And in the mindset of the project maintainers that I've seen, it makes sense not to implement it until we find out that it is really a problem. Ie. there are many assets that have dependencies on other assets and users are having a lot of trouble with it.

As a well engineered solution though, I agree that it would be good to have and may encourage the development of an ecosystem of assets building upon / depending upon other assets.

[eon-s]

Add on top breaking dependencies between assets due updates (which is common in 3rd party assets dependant engines), many versions of the same plugin should be kept on the library.

[willnationsdev]

@eon-s So, something like @bojidar-bg's suggestion?

[chanon]

I think the more important issue discussed here is about the topic of including things in the core engine vs. having them as extensions or official extensions.

I personally really like the idea of being able to customize the engine and removing everything that I am not actually using. [Can someone point me to how I would remove all 3D stuff for example? And can I just disable compilation of modules I don't use? How?]

The thing is, when someone proposes a new node type eg. the mentioned Spring Arm or new class eg. a better RNG class, or if some existing core nodes are to be removed eg. the Vehicle controller ... those are C++ code. If they are not included in core, then how will they be added as extensions? GDNative assets? That would be a lot of dlls and shared libraries to manage/build.

As someone who builds my own godot build, I would actually prefer to have a way to install them into my godot build folder. Eg., if they were modules, I would just git clone them into my modules folder, or maybe there would be a tool that automates that somehow. You would be able to select all the modules you wish to use from the list of official modules, official extension modules, and also from user submitted ones. Then download (git clone) them, and then you can go ahead and build your custom Godot. Also would be able to see which ones have been updated and update them and install new ones later. (This is referring to C++ code.)

[eon-s]

So, I guess that first we need a better and flexible asset library (asset installation, dependencies, versioning, etc.) and easier plugin integration/extension (editor tools, scriptable custom nodes, etc.) before trying to remove features from the engine.

[LikeLakers2]

Responding to @chanon here.

No, do not let assets be able to define dependencies. I don't think Godot assets will have such high amount of dependency on each other like npm modules have.

But why though? At that point, why not just let assets have this dependencies file anyways since it won't harm anything? The reason many npm modules have dependencies is because it saves everyone the hassle and reuses code from other modules that the user might even already have installed. Same goes for Ruby gems, Python via pip, even package managers on Linux distros allow it. I don't see why we shouldn't allow dependencies on assets just because they might not be used as often.

It is true, however once the user sees it doesn't work, they will immediately look up the documentation and see that they need to install the other asset(s).

I don't agree with the sentiment of this. There's going to be a lot of people who will look up documentation first, sure, but I'm fairly sure even they will become annoyed at having to search up each and every needed asset every single time on the asset library. But I think it's also worth mentioning that there's always that chunk of people who just never seem to google issues beforehand, and will just complain to the asset author(s) that their asset isn't working.

[Two-Tone]

I don't even agree with the idea of removing features simply because they can be done in GDScript. Simply put, it's antithetical to Godot's goal of "nodes for all your needs" if you're going to force a large swath of users to download or implement a tool/feature/etc that they need.

In fact, a massive swath of nodes fall under @reduz's second requirement for being included in the engine, simply because "It's difficult to do yourself" is completely arbitrary. What may be simple for some is hard for others.

[vnen]

I don't think having an official repository for extensions goes against the "nodes for all your needs". Just because they are not integrated in the executable download, it doesn't mean they're not being provided. Also, the idea is that the base needs are covered and the power to extend them is available for the game-specific needs.

It's nice to have a FPS character controller out-of-the-box, but it's useless to everyone not doing an FPS game. And by adding things like that, you make essentially every game containing many features that they don't need, increasing the final binary (which matters a lot on mobile and web targets). There should be a way to remove unused things from the final export, and providing them externally on a need basis is one way (at least partially).

I agree that difficulty is subjective. Making a spring arm node is easy only if you already know the fundamentals to implement it. Otherwise you'll spend quite some time on research, which defeats the purpose of using a full-feature engine. The better metric (though still somewhat subjective) is "how much low-level" is the feature. If it needs to add functions to the physics or visual servers, it likely should be provided in engine. If it's only an extension of an existing node, maybe it can be outsourced.

---

I still believe that is more beneficial to provide official nodes for download and trim the core. With this, we can:

• Have a small executable for editor and exports.
• Provide more functionality officially, with more game-specific helpers.
• Update and fix bugs in the extensions without releasing a new engine version. This means the outsourced nodes can have a much faster release cycle and release a patch as soon as a bug is fixed.

But for that we still need to make the addons a higher-class citizens than the user-land scripts. Using extension nodes should feel like using the core nodes, while still being clear that they are not.

[avencherus]

Interesting discussion. Many good proposals as well.

I must say first, that I do appreciate the compactness of the final binary, and am grateful that there is an engine made with that goal in mind. I’m glad the central vision here is to actively combat bloat. At least for me, I prefer to have only the tools I need for the job at hand in reach.

Just wanted to pass along a nod to some of these interesting ideas.

Especially the one regarding that more and more functionality get exported into something that takes on the form of something like official modules or plugins.

From that I imagine this being like a core and then a library of modules, with essential ones just installed by default. Perhaps many could be opted out through the editor settings. (I believe Zylann was mentioning excluding 3D for a 2D game.) Beyond the defaults the other modules could be easily located and included.

Though I wonder about the difficulties of implementation, because you'd have to consider what happens when one might want to remove a module that is actively in the project. It may have to warn the user to remove it from those scenes / scripts before it can be deactivated... or would they remain as tombstoned items in the project, that can revive when the module is added back?

It is probably no small thing to implement, but I am fond of being able to pick and choose the functionality provided, whenever possible, allowing the game export having only what is needed. More control over that sounds valuable to me.

The other thoughts from Vnen about having modules with some special classification between built-in and the scripts was a nice one as well. It would allow some other types of partitioning and editor behaviors and options without interfering with the good things about the existing design.

Some said yay and some nay on that idea of making addons appear more like Godot itself, but it can imagine it perhaps being situational. I could also imagine it being done by splitting the concept a little. Providing both.

I’d be also interested to see what could be done regarding keeping editor tools out of the final build. I have toiled much over trying to keep these things apart, because bugs and behavior intended for tools can creep into the game itself. Sometimes destructively if you’re doing something with modifying data that will be saved.

At the same time, I find it quite fantastic to easily modify the functionality inside the editor using tool script, so more attention there would also be fantastic.

Anyway, keep up the good work everyone.

[starry-abyss]

Just another crazy idea:

Maybe it would be possible to somehow auto-convert GDNative C++ to a module (ideally removing the indirection layers created by GDNative)? Then it might please both advocates of "everything in one static executable" approach and ones of "strip features without installing the compiler" approach

[AlansCodeLog]

For those against removing the nodes, if the nodes that are removed are implemented in GDScript as official plugins, wouldn't it be better than what we currently have? Because then beginners could actually peak inside the script to see how it works.

[Two-Tone]

I fear that would hurt performance as GDScript just isn't very fast.

On Fri, Jun 22, 2018 at 11:10 PM, Alan North [e-mail]
wrote:

For those against removing the nodes, if the nodes that are removed are
implemented in GDScript as official plugins, wouldn't it be better than
what we currently have? Because then beginners could actually peak inside
the script to see how it works.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/19486#issuecomment-399628884,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEJES-DWkzuyItQDPbOZgsuN_lA345z7ks5t_b_NgaJpZM4UhqDC
.

[LikeLakers2]

@AlansCodeLog If they absolutely must be removed and made into external extensions, it'd be nice to have a GDScript version at least, so that users don't need to recompile the engine (if it's the original C++ files) or install Mono (if it's converted to GDNative). However, part of my original response (See the section on an official extension repository) involved my concern that bringing them into extensions would just create this weird sort of market (for lack of a better word) if users stop remembering that there's a extension repo that contains what they need, regardless of how official or how up-to-date the official version is. New addons would come into the asset library that would have "Node X with Feature X", "Node X with Feature Y", and so on ("Node X" representing the node that was made into an extension) -- creating a mess of addons that all claim to help with vehicles but not having that level or scrutiny that more official nodes might get.

[chanon]

@LikeLakers2

But why though? At that point, why not just let assets have this dependencies file anyways since it won't harm anything?

I am not against it. But since reduz seems to not like it, I suggest doing a simple improvement like I suggested first. See how it goes, then could add ability for assets having dependencies later. Just a one step at a time suggestion.

However, I am against the idea of installing project addons at a location outside the project folder as it makes things harder to manage. Only 'global' editor addons should be outside the project folder.

[chanon]

After some more thought, actually if core features are going to be removed, and people want to have access to those core features [that would then be extensions] in all their projects, then it makes sense to be able to have those extensions installed at a global location with version management for them. But the option to download and install the extension directly into the project folder should also remain in that case. (Again, it is like npm option to install locally or globally with -g)

Like eon-s said above, it makes sense to have a better asset management system if things are to be removed from core engine and turned into assets. Otherwise it would amount to functionality being lost from the engine as users would not be able to discover the removed functionality easily.

I was thinking that ideal scenario for

• New users and people who don't compile the engine: On the engine download page they have the option to download the core engine, and also select which official or recommended extensions they want to include in their engine download. (Note that this would not be possible if extensions cannot be saved to a global folder. Users would have to find extensions after they create a project instead, which harms discoverability. ... In an ideal [impossible] world though, the website would build a custom C++ build from selected extensions/components on the spot.)
• People who compile the engine: A tool that manages extension C++ code downloads and versions and puts them in the correct spot in the build folder so that they are included in engine builds.

As someone who compiles the engine, I would like my extensions as modules rather than GDNative (or slower GDScript). I am imagining the level of modularity that I can choose specific Node types to include or not include in my build, or even drop all 3D stuff.

If C++ extensions are all going to be GDNative then that puts the burden on the extension creator to have to build for all platforms - which I think might discourage people from setting it up for simple small extensions that may be useful as it would feel overkill to do all that is required to publish it - and also discourages separating functionality into many small modules. I also don't like the fact that it will be so many dll/shared library files added to the output project.

On the other hand, for people who don't compile the engine, GDNative is the only solution for adding C++ extensions.

So like starry-abyss suggested, if GDNative could be auto adapted into a normal module then that would be great.

[swarnimarun]

Ok so I have changed my stand I like the idea of a bloat-free engine with capabilities in the same league as the powerhouses of the industry.

Having all the extra nodes separate will allow for lots of advantages especially the ability to choose what you want and what you don't.

Here are some of my suggestions of how to approach it,

1. First we can add the ability to the asset library to have assets which can be installed in parts, in git they can be separated by tags or branches.
   So now if I wanted only the 2D Feature Nodes I could download just that but still have only one place to look at to check for updates.
2. Second we can add the ability to add labels to updates of assets, like some can be core updates some feature and some backwards compatibility breaking.
   This can then be used by the Asset Manager automatically to update addons and ask update you on the matter. Like it will tell you that you have come core updates waiting, or your next update is gonna be backwards compatibility breaking. Do you want to update or not.
3. Third to allow for having multiple versions of a asset and then when a version get's too old the dev of the addon/asset can mark it as outdated which will be reflected when you open Godot and it will ask you to remove the older version or not.
4. And last an ability for addons to ask for other addons it will be like dependency but it will just tell you that you need this addon to run this addon and if you don't have it install it.
   And if ever the addon itself notices that the addon can no longer be found ask to re add it or just show an error message.

Using this method we won't have a need for dependency manager but the devs would be responsible for the work. If an addon get's old and is using an outdated asset/addon both the devs can then talk and the dev can decide what to do.
He might just port his addon quickly. If need be or in some cases the other addon can have outdated tag removed even.

And as @chanon said we will need to make the assets be downloaded as global which can then be imported into the project like Unity did it.
And a better asset manager than our current implementation with lot of flexibility into the installation of addon process should be allowed.
I recommend having the ability to add an addon as both GDNative and Module which will put some load on the addon devs but I think that's the only acceptable solution unless something radically changes here.
But we can still add some features to GDNative to ease the addon creation. Like being able to build for all platform at once would be much better.

[Two-Tone]

On the issue of dependencies and assets breaking due to plugins updating,
why not (and apologies if I missed someone already suggesting this)

1. have the server keep every version uploaded

2. allow plugins to specify a specific version of a specific plugin

3. try to use a sane versioning system for plugins (Godot seems to use the
   Major.Feature.BugFix version numbering, which I like)

4. And most importantly, don't auto-update plugins? (Or at least allow the
   user to select which type of updates they're okay with, eg bug fixes and
   feature updates, but no major updates).

[swarnimarun]

@Two-Tone Then why even make whole thing bloat free.??

We have this discussion to make the engine and the whole system as bloat free as possible not have needless versions flying around.

You are right about allowing users to decide to update but bug-fixes should be auto by default. Cause if I have hundred addons I am not gonna check them one by one for bug fix updates.
Rest can be upto the user we can have a setting to disable if the need be.

[efornara]

And the store must offer official bundles for offline use (with offline store support in the editor).

I would suggest one offline pack of a few 10s of MBs containing the most important demos and the most important add-ons that should be very easily discoverable. A smaller font link on the official download page, like the one for the old godot versions, would be great.

Probably, yes, but in a world where desktops don't seem to use anything less than 256BG SSD + 1TB HDD and even Phones have space starting from a minimum of 16GB all of which which is more than likely to increase, would it be that BAD?

In the portable and affordable laptop segment, eMMC is alive and well and 32GB size (with less than 10GB free) is not so uncommon.

By the way, on that class of laptops, I think that the current android export system is useable, while the planned one (https://github.com/godotengine/godot/issues/18865) is not. I do understand the merits of the new system and on balance I would have probably voted for it myself (when I saw the issue there was no contest already), but we should be aware that we are losing an important use case by the change.

[Two-Tone]

@swarnimarun There may be a misunderstanding. When I say "system" in that comment, I meant the server that serves the plugins, not the engine. Sorry if that is what tripped you up.

If not, then I don't see how trying to solve peoples issues with a dependency system for plugins is antithetical to trying to keep the engine bloat down. A plugin kept on a server as a just in case someone needs it doesn't contribute to bloat or maintenance overhead and most are so small that the cost to store them shouldn't be that high.

[swarnimarun]

@Two-Tone Yeah I thought you meant on the computer. But on server it's fine and it can already be done as Github allows hashs that can be saved even after changes have been made so in the current system even after you update the repo you must update the hash in the asset page.

Not only github even gitlab and others also allow it. If there are any others. Git supports hashes to be saved as commit ID and so things are done the way they are done.

[BastiaanOlij]

Just my few cents, pretty sure most of this has been said before so I doubt I'm bringing in anything new.

I don't believe in doing things as plugins for the sake of doing things as plugins. I understand the size issue but we're not living in the 80ies anymore. We're talking about the engine being 50Mb or 55Mb, not between the engine being 50Mb or 500Mb, the extra size increase of the feature to be moved into a plugin is negligible compared to the size of the assets of any serious game. If the price we pay is a more troublesome integration, it isn't worth it.

There are many places where plugins make more sense and with the architecture like GDNative maturing, we should definitely ask the question "is this better to be a plugin" and in many cases I would argue the answer is yes. One thing I like about plugins is that it allows you as a developer more freedom in many respects. But there are also limits.

My VR experience eludes to a few of these. There is a reason it is taking longer then I would like to get this up and running on mobile. Because of the demands, especially on mobile, of setting up the displays in various ways you have to change the way the core works and its proving difficult to capture it in a plugin.
OpenVR and OpenHMD so far have been the easy ones. Oculus I've gone through a workaround that is looked down upon but I'm happy with it.
ARKit has to be in the core, its not solvable as a module, at least not in our current structure (Apple doesn't even allow dynamic libraries on iOS so there is that too).
GearVR and Daydream really need to be in the core, an impossibility for GearVR due to it using proprietry libraries.
Hololens/Microsoft MR I am going down the route of seeing if I can make this a platform choice, Microsoft sees them that way because apps are supposed to run between the two and are installed on the Hololens as a holographic only app.

Now AR and VR are niche functionalities that very few people will use, so they are the type of things that should stay out of the core if at all possible and I'm on furious agreement with that standpoint. But I can't say the same for other things.

I saw vehicle body mentioned somewhere, seriously? The kilobyte of code you save putting that into a plugin? Physics are the types of things that I think should be fully supported in the core IMHO. I rather see a full suite of fully integrated physics objects inside of the core then have to hunt for those in plugins. While an interface approach similar to the ARVRInterface we wrote in GDNative would probably create a solid integration, I still feel we're just making life needlessly difficult.

Contrast that with say the terrain editor components, now there is something I see as ideally suited to be plugin based. There are various different ways to address terrains depending on the type of game you wish to make so having a choice between different addons makes sense, and being able to develop on these independent of development on the core, is even more of a reason.

To make a long story short, where plugins make sense, support plugins, but don't go overboard.

[eon-s]

Nodes that may worth turning into plugins are the Control nodes that are made by Control nodes (the composite ones), leaving only simple nodes to construct others.

Nodes with complex logic like Vehicle or even simple but useful like Position ones, are good in core.

[mhilbrunner]

IMO, none of these should be removed from the engine. Control nodes, physics nodes (like Vehicle), and even currently not existing Terrain support:

I use Godot because it is a fairly self-contained, integrated application.
If in the future I need to download Godot, and then look for Vehicle, Control nodes, Terrain support, a Shader editor, a decent Navmesh plugin and on and on and on - using the current plugin and AssetLib system - no thanks.

If we go this way, both the plugin infrastructure and the AssetLib need a usability overhaul.

And by the way, I'm way less bothered by UE4's large install size than I would be by it not coming with built-in support for Vehicles, Terrain or UI - which it has, all of it.

[PJB3005]

I'm not sure how much of this has been stated already, large threads like this can get hard to follow, but anyways:

I think we should focus on the best out of the box experience. Having to go into the asset lib, even for "officially endorsed" plugins is rather unwieldy.

Export size shouldn't make usability of the engine worse, IMO. If we need to cut down on export size, then make some feature to disable sections of the engine such as 3D at export. Especially because I personally do not value platforms like mobile and especially web, and it pains me to see the engine held back by them in discussions like this. (I can see the demand for them, don't get me wrong, though screw web)

Hell, if we're worried about download size of the editor then there should be separate "batteries included" and "minimal" editor downloads. How the separation is implemented is up for debate, but it should be nicely integrated, no matter what.

[akien-mga]

I haven't read the whole discussion thoroughly but I'd like to readjust it somewhat: we're not concerned about size. Sure, keeping export size small on some platforms is important, but it's not a deciding factor when implementing features in the engine. As mentioned, if some features need to be made conditional to allow small export sizes, it's trivial. Download size for the engine itself doesn't matter.

Now that's settled, the main factors encouraging us to consider keeping some stuff ("some" to be determined) out of the engine are:

• Technical debt. All code needs to be maintained. A broken VehicleBody like the one we have since 2014 is not particularly useful, as it's not good enough for real games. It recently got improved using Bullet so it's getting better, but for the last 4 years it was effectively dead code and unnecessary maintenance burden.
• Universality: Godot is an engine that lets you do any kind of game, and the features it provides should enable you to do any kind of game. It doesn't mean that it should be exclusive (you don't need both 2D and 3D at the same time in most games), but the features should cater to the common needs of most users. Which means that the features we provide should be low level, and that game-specific high level features are most of the time best provided externally.
• Flexibility: Having high level features maintained outside of the core enables a much faster iteration process to improve them, fork them if need be, etc.

If we go this way, both the plugin infrastructure and the AssetLib need a usability overhaul.

Well that's the raison d'être of this issue, no? :) The whole point is that we need a usability overhaul to allow for a better workflow instead of bundling all corner cases in the core engine.

[Ranoller]

Well... if someone need ideas to start the external features in addition to the so named VehicleBody i want to remember some nodes and classes lost from godot 2 to 3: ParticlesAttractor2D (node) and EventStreamChibi (this class is perfect for a external asset), i know that particlesattractor is incompatible with actual particles system, but... What about recover lost functionality of godot in form of external asset (Ej, old particles system coexisting with the new....). Maybe someone want to maintain that code in separate repositories...

[eon-s]

@Ranoller new systems always remove old stuff, is inevitable.
CPU particles are supposed to come back with GLES2 (because does not support CPU ones), but having atttractors on CPU but not on GPU won't be good.
EventStreamChibi was a module, I guess the new audio system can give better support for tracker files if gets more flexible.

[swarnimarun]

I don't think that even if we care about maintainability we should separate stuff out just yet.
We could ask people to help out on stuff getting older by posting about it on a special thread and then people can update features.
With the tremendous growth in Godot userbase I am certain we will find a lot of guys willing to update things and work on stuff getting old or essential features missing from parts of the engine eg, more joints.

If you think that it's easier to get people to work on separate code files then it's wrong most will just not know about those files as it will never get the same exposure, asset manager needs work yes but having features as plugins doesn't work well.
I have seen a few people switch from Unity to Godot because it's so well integrated and doesn't offer essential features as plugins.
And Unity realized that started working on making things more integrated and added the addon devs to the Unity dev team.

If we need to download a 30MB Engine and then waste time downloading separate features as plugins that I might forget to add into my project then I might not use them in the first place. Which is a problem for a beginner.

[Ace-Dragon]

Regarding technical debt, this is an issue that every project (FOSS or commercial) eventually has to deal with. The truth is that some developers eventually leave and an area ceases to get much attention (so it has to be placed into "maintenance" mode until someone is interested). As I mentioned, this even happens to big commercial apps. so it will inevitably happen with Godot, no matter how much trimming the engine gets.

The best solution in that area is simply to foster a development scene that encourages people to start contributing to the source (and the Godot scene has been doing a fine job when looking at all of the volunteer development, but it has to be kept up through things like making sure a patch contributor does not wait to long to see a review). However, the removal of features because the developer left should be avoided if possible (because a constantly fluctuating feature list would be a great way to convince people _not to use_ Godot at all). Rather, feature removal should mainly occur because it is being replaced by something better.

[akien-mga]

However, the removal of features because the developer left should be avoided if possible (because a constantly fluctuating feature list would be a great way to convince people not to use Godot at all). Rather, feature removal should mainly occur because it is being replaced by something better.

Again, the discussion is going in the wrong direction so I'll try to steer it back on the right track. The topic is not so much about removing stuff, it's about preventing the addition of new things that we might end up wanting to remove. Because adding stuff is very easy, but then we can't remove it without breaking compatibility.

As Godot grows, more and more new contributors come with additions which might not fit the architecture or design that we want the engine to follow. Reviewing all those changes takes a lot of time, and even more so deciding whether they fit what we want. Merging such new features by new contributors will result in code that the engine maintainers don't know well or may not fully approve of eventually, and the original contributors may often not stick around to refactor it and make sure it matches what we want (see e.g. the Tween node, which is a great feature but with sub-par design, and the original author is no longer around, so it's up to us to rethink and rewrite it, breaking compatibility on the way).

So the whole point of @reduz's proposal is that the ever-increasing number of features that new contributors will want to put into the core engine would be, most of the time, better suited as assets in the asset library. The point of this issue here is how to ensure this workflow can be as smooth as possible.

We're not trying to shrink Godot's codebase, we're thinking about how to protect us from having it explode and go out of control. (Because one-time contributors with a big PR are a lot more frequent than engine maintainers with a thorough understanding of the complete codebase and the vision we have for the engine's future).

[LikeLakers2]

So the whole point of @reduz's proposal is that the ever-increasing number of features that new contributors will want to put into the core engine would be, most of the time, better suited as assets in the asset library. The point of this issue here is how to ensure this workflow can be as smooth as possible.

I have to ask, what will making these into assets on the asset library do except move the problem back a step? You're going from code bundled with Godot that's no longer being maintained, to code in a Godot asset that's no longer being maintained. Either way, the original contributor is probably going to stop maintaining it at some point and it will eventually be broken again.

[willnationsdev]

@LikeLakers2 Well, the purpose of it is to create a smooth avenue for new contributions to be added for the engine (since most that are more asset-related are rejected for now). If some aspect of the engine isn't being maintained, then it shouldn't be bundled with the engine in the first place as its pure bloat at that point. Things of that nature are especially suited as externally included/updated plugins.

[AlansCodeLog]

If in the future I need to download Godot, and then look for Vehicle, Control nodes, Terrain support, a Shader editor, a decent Navmesh plugin and on and on and on - using the current plugin and AssetLib system - no thanks.

Exactly. The AssetLib/add-ons in general don't work well now for this, hence why the conversation tends to deviate to those. I don't think anybody wants to take stuff out right this minute, but as has been mentioned the point is about not adding too much new stuff.

I have to ask, what will making these into assets on the asset library do except move the problem back a step? You're going from code bundled with Godot that's no longer being maintained, to code in a Godot asset that's no longer being maintained. Either way, the original contributor is probably going to stop maintaining it at some point and it will eventually be broken again.

Well, I'm assuming it would be easier if they could be managed more separate from the engine. Blender does something like this. They have "official" (not really official, just pre-included) add-ons, but their code is kept in a separate git submodule, and when they become un-maintained by the people who submitted them (I think you have to resubmit with every big release), they are dropped. Though I honestly don't really like this approach, I wish they just had a built in asset browser for 90% of the stuff and just outright integrated the rest (things like exporters).

I think the idea though is that the engine would have just the basic building blocks included, then stuff like the vehicle node could be official add-ons, then everything else that's big and could be implemented in different ways should just be regular add-ons, right?

What I don't get is how the official add-ons would be implemented. I thought currently we couldn't write custom nodes that looks like they were built in, right, they had to have a script? Or is that possible with a c++ module or something (am c++ noob here).

And also eventually we get into the problem of how to have an add-on that depends on the separated official nodes. Without some sort of dependency system it sounds like hell.

[jonbonazza]

Late to the party, but given the scale of my current project, I have found myself developing _a lot_ of plugins for Godot 3 and see myself developing many more before the project is finished. With each plugin, I try to develop as many as I can in a game agnostic manner and open source them under the MIT license for others to use. I am not going to lie. The current addon system really isn't that bad in the grand scheme of things... HOWEVER there _are_ a couple missing pieces that make it borderline useless in many cases. Before I get to those, however, I want to say a couple things about @karroffel ideas.

First, thanks for putting this issue together. I think having this discussion is important--especially in the near term. That said, I do think each of the points of discussion you presented in the OP are mostly orthogonal with only tangential overlap. I think we can very much iterate on a solution here without having to address all of the concerns in one go.

This leads me to my _personal_ qualms with the current addon system and what I think would be two very good starting points to a better world from someone who actually develops plugins on a regular bases:

The single biggest problem with the current addons system is lack of dependency management.

This has been heavily discussed already, but most of the suggestions are rather heavy handed and aren't extremely compatible with current system. Now, I haven't familiarized myself with the assetlib module's code base yet, but I think that there _should_ be an easy win in this department. Currently, 3rd part addon repositories on Github abide by the following folder structure:

addons/
    addon_name/
        plugin.cfg
        ...other files

What I propose is to extend the plugin.cfg schema and add the ability to simply have a list of dependency versions with an option commit hash (or branch/tag name) to use. Something like the following would work:

[plugin]
name="foo"
description="foobar"
author="blah"
version="1.0.0"
script="plugin.gd"

[dependency]
name="dep1"
path="github.com/foo/dep1"
hash="7fc2367508c41cfab86eaf7d84e04cd122978fdb"

[dependency]
name="dep2"
path="github.com/foo/dep2"
tag="v3.1.2"

[dependency]
name="dep3"
path="github.com/foo/dep3"
branch="big-refactor"

Then, when the asset library goes to install the asset, it first downloads the project into the addons directory just as it currently does, then subsequently downloads each dependency (using the correct hash or branch defined) and puts it in the addons directory as well. As a first step, dependency conflicts can either be ignored or let the user choose which to use. This is not an _ideal_ solution, but I think it will work in most cases and at least hold us over until a more involved overhaul of the assetlib is performed. (unless someone can come up with a simple way of dealing with dependency conflicts off the top of their head).

As a real world example,
I have an asset for godot behavior trees. Part of this system is a UE4-like Blackboard node for storing arbitrary global or node-local data that is to be shared across nodes. This is used in the BT system for sharing state between behavior nodes. That's all fine and dandy, but then I am currently developing a dialog system and the accompanying tooling, and part of this system requires some sort of database of varibales to use for conditions. I would very much like to reuse the blackboard component from the BT system for the dialog system as well. In fact, the Blackboard component is about as generic as it can get, and global data is a fairly common idiom in gamedev, so even on it's own this could be a fairly useful addon, as it's one that is reimplemented in basically every project anyway.

With a proper dependency system, I could break the Blackboard component into its own addon and then reuse this component in all of the BT addon, the dialog system addon, and whatever other game-specific uses I have for it. Seem what I mean?

The need for multi-script nodes

This is another hot topic in this issue already, and honestly, I dont have a very good way of approaching this without changing the fundamental ideology of how Godot is intended to be used. This may or may not be okay, but yea... Anyway, this isn't so much a problem for the plugin _authors_ as it is for the plugin _users_ (often times the same person though ;).

In the end, if we can solve these two things, I think we'll be 90% there. The other 10% isn't as big of a deal in my opinion and should probably be approached from the perspective of overhauling the fundamental architecture of assets, which in my opinion, can wait after frying much larger fish.

[jonbonazza]

If there is no objection, I will probably try to implement my suggestion regarding dependency management in the assetlib code in the coming days

[volzhs]

the dependency management sounds cool to me. :smiley:

[Two-Tone]

The only thing I don't agree with is having the version of the dependency
be an optional listing. That will cause the problems reduz mentioned that
happens with other engine's plugin system.

unless someone can come up with a simple way of dealing with dependency conflicts off the top of their head

I think the solution for that would depend on how a plugin calls code from
a different plugin. If we set it up so that the plugin's scripts does
extends "dep2/folder/script.gd", at install we could just change dep2
to dep2-v3.1.2. For hashes we could just use the first 5 digits (unless
we want folders with names that are at least 42 characters long). At 5
digits the chance of collision is incredibly small.

I'm a bit tired right now, so I might be missing something, but I don't see
any issues.

[Zylann]

@jonbonazza if you do, please try to not hardcode Git usage in it. Although it sounds like granted around here, some people might prefer other VCS systems (or not using VCS at all), so perhaps that should be a detail to be handled on the asset-lib side.
Also, I wonder if we consider doing something for making it easier to have plugins as VCS sub-repos in a project? It was mentionned at some point but I don't remember if there was an agreement.

[chanon]

Yes, I think just the id of the asset and the version should be enough.

Specifying Git repository/branch/tag/hash might be an optional way to specify dependency, but the main way should be by asset id.

I'm not sure if the asset library has unique human readable ids (slugs). If not it should. For example npm modules have a text module name/id that you use when installing them.

Dependency specification would then maybe look like

[dependency]
asset_id="some_asset"
version="1.0.0"

[jonbonazza]

@Two-Tone I was trying not to get too crazy with this. Just wanted something simple to hold us over until a better approach is decided upon.

@Zylann I'd probably use the same approach that is currently used in the asset lib--downloading and extracting the zip file from github.

@chanon the asset lib wirks with github directly so the only identification we have for versions is commit hash, branch, and tag.

[Two-Tone]

What part is too crazy? Having dependency versions mandatory or my solution
for dependency conflicts? One is standard in most any place that has
dependency management and the other is a simple edit to text and folder
named. I don't think either of those things are crazy.

On Tue, Jul 3, 2018, 11:23 AM Jon Bonazza [e-mail] wrote:

@Two-Tone https://github.com/Two-Tone I was trying not to get too crazy
with this. Just wanted something simple to hold us over until a better
approach is decided upon.

@Zylann https://github.com/Zylann I'd probably use the same approach
that is currently used in the asset lib--downloading and extracting the zip
file from github.

@chanon https://github.com/chanon the asset lib wirks with github
directly so the only identification we have for versions is commit hash,
branch, and tag.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/19486#issuecomment-402214905,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEJES-_IJBjvj3ImvZnqdu-cIAunCp2gks5uC5qIgaJpZM4UhqDC
.

[jonbonazza]

@Two-Tone sorry, but "too crazy" I meant "too complicated" as this is just supposed to be quick fix to hold us over. I agree that in the long term, proper conflict resolution is necessary, but I was hoping we could just get away without it in the interim.

[jonbonazza]

Another option is to just forgo the assetlib all together and just git clone the deps directly into the project, though since the current de facto plugin project structure is rooted at the addons directory, this won't really work. We'd need the project root to be one level deeper (i.e. excluding the addons directory and rooted at the plugin name). I honestly think this would be a welcome change regardless, as then you could use git submodules for dependencies.

[chanon]

@jonbonazza I see. Your solution is something that could be implemented and work right away without needing to modify the Asset Library. Then I agree it is a good first step. (Not that I am a core maintainer who has any authority .. just that I think it is a good first step.)

I think if the Asset Library database has a text identifier (slug) plus version information added, then that could be added like my example above.

[Zylann]

you could use git submodules for dependencies.

@jonbonazza and that's where my latter question rises again, as it doesn't seem to be possible to use submodules with a plugin repro because that repo would need to include the full file hierarchy to be suitable for download, forcing .git folders to be at the same location... but even if that was a thing, remember it would force people to provide all their stuff through Git.

[jonbonazza]

@chanon exactly. It would just require a small amount of additional code in a couple files. Honestly, an addon could probably be developed that could do this for you instead of building it into the assetlib

[jonbonazza]

@Zylann yea that's what I was getting at in my comment about submodules. It's currently not possible due to current restrictions on project structure, but changing the asset lib tob be more flexible in this regard would be trivial.

EDIT: It could be done in a backwards compatible way as well.

[AlansCodeLog]

I actually made a proposal regarding the project structure a while back. That is actually my main complaint about how plugins work now. There is no nice way to package them. See #18131. That really should be changed first, imo, before anything, and the sooner the better, so we have less people having to transition their plugins.

[Two-Tone]

What's complicated about it? One is a requirement that can be enforced with
a simple check and the other is a simple match and replace or
append/prepend. This is simpler than actually adding in dependency support.

On Tue, Jul 3, 2018, 12:11 PM Jon Bonazza [e-mail] wrote:

@Two-Tone https://github.com/Two-Tone sorry, but "too crazy" I meant
"too complicated" as this is just supposed to be quick fix to hold us over.
I agree that in the long term, proper conflict resolution is necessary, but
I was hoping we could just get away without it in the interim.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/19486#issuecomment-402229111,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEJES7c8ziwhXZirsTsLY0zWfI-jrBzmks5uC6XUgaJpZM4UhqDC
.

[jonbonazza]

@Two-Tone you cant just rename folders because scripts in the plugin's code often use the path of the addon directly. E.g

extends "res://addons/foo/bar.gd"

[Zylann]

@jonbonazza I would add, it's not just scripts that do that, but any resource referencing another one.

[Two-Tone]

Then have the add-on folder structure be like I said but mandatory, then at
install time just use the find and replace functions that the String class
has on every script to look for

'extends "res://addonname/'

And replace it with

'extends "res://addonname-versionnumber/'

These are not hard problems in code. They're really simple thanks to c++.
These are mostly just user problems as to ensure these problems are fixed
we have to mandate some stuff. If they upload a plug-in and users find that
it's completely broken because the dev did not follow these incredibly
simple rules, then that is on the plug-in developer.

And we're going to have to mandate rules to begin with, so we might as well
include these or some form of these to make sure we're doing this right
from the beginning.

On Wed, Jul 4, 2018, 12:48 PM Marc [e-mail] wrote:

@jonbonazza https://github.com/jonbonazza I would add, it's not just
scripts that do that, but any asset referencing another one.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/19486#issuecomment-402533781,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEJESwoMKvVVXcllrQ4YQQO852oZ1v7rks5uDP_SgaJpZM4UhqDC
.

[Zylann]

I guess if the plugin breaks compatibility after an update, they should either provide deprecation fallbacks, or do as @Two-Tone says, intentionally breaking the path by either renaming the plugin or a subfolder of it (that would be worst case scenario though because it's not a nice situation). That's how the engine itself does, and others do. Writing a plugin comes with the whole different mindset that other people/plugins who you don't know will depend on it.

[Two-Tone]

What this solves isn't a dependency issue but a solution to dependency conflicts. One plug-in may depend on version 1 of a different add-on and another may depend on version 2, without this or something similar to it we'd either have conflicts or would have to restrict the user from being able to use plugins that have conflicting dependencies.

And the solution to preventing updates from breaking compatibility is to enforce that each update must have a unique version number. That, however, might start falling into the "too complex" category as you'd have to parse plugin.cfg then ask the assets server if there is already a plug-in with that name and version number. If there is, throw an error. If not, everything should be fine.

And I'm not sure if you're talking about scripts in the plug-in or scripts the user writes, but either way I do agree that it is a bit unwieldy but it shouldn't be that difficult of an issue.

The only solution I can think of is extending gdscript so that the user can specify in script what is version of the plugin they're using, then from that point on they just do extends "res://addonname/etc and have the engine itself handle using the correct path. That, however, definitely falls into the "too complex" issue that @jonbonazza mentioned. But for the short term it can be skipped.

There are other issues that will need to eventually be solved. EG the user should know what dependencies are being install, how to handle the multiple folders in the internal filesystem browser that comes from solving the dependencies conflict issue, how do we deal with plug-ins that are a dependency but have an update (do we apply the same settings that are used for manually installed plug-ins? eg auto install bug fixes or only install updates when the main plug-in updates its list of dependencies), and more I'm sure.

I can absolutely see why @reduz feel's that dependency management isn't important. There are so many variables that _need_ to be considered and solved in order for this to be worth inclusion in Godot or else we end up with a system that is no better than what other engines use.

[jonbonazza]

Honestly, I think the best course of action for a short term solution is to change the asset lib to support (though not _require_ because backwards compatibility) the ability to serve projects that are one layer under the addons dir. E.g

godot-tools.level-streamer/
    project.cfg

Instead of

addons/
    godot-tools.level-streamer/
        project.cfg

This way, we can just specify in the description of the addon that it requires some other addon.

It also would allow for us to easily use gitsubmodules in our project for addons instead if we want.

[Two-Tone]

What about doing that for the deps (eg dep/) then have the manually installed plug-ins in the root directory or a user defined folder? That way dependencies are not cluttering the root and addons can be in either or?

$UserSetLocation/
    godot-tools.level-streamer/
        project.cfg

deps/
    thing-level-streamer-uses/
        project.cfg

[willnationsdev]

@Two-Tone But then what if your dependency has a dependency? Shouldn't the deps folder instead be a subfolder of the plugin itself, that way you can go arbitrarily deep as necessary?

[LikeLakers2]

I feel like this whole dependency issue is being way over-thought about. Maybe we should come up with a few simple systems and have a discussion on each, or a vote? I don't foresee this conversation ending with the way it's heading currently.

That being said...

---

@willnationsdev That could work, but if many plugins rely on the same dependency, then you have multiple copies of that dependency where there might only need to be like... two.

[chanon]

Yes, dependency management isn't a simple thing. Otherwise npm wouldn't need to be as complex as it is and yarn wouldn't need to have been created.

I don't think putting dependencies as subfolder of assets is a good idea as it is leading to a path that will be very complex. There will be case when 2 or more assets have the same dependency and some kind of deduplication would be needed. Also, I think it is better for the game developer to see exactly what dependencies the project has, rather than having to go through subfolders of subfolders to find out which dependencies are being used and/or duplicated n times since it is a common dependency.

In node.js web dev, for example, it is somewhat ok to ignore what is inside the node_modules as long as it works as it will only be deployed on the server. For a game, you will be packaging all that code/assets into a final app package and having the same dependency duplicated n times is pretty bad.

These problems are why my initial suggestions said not to let assets define dependencies yet, but just have a way to list dependencies of the project, and let user manually install any dependencies of assets:
https://github.com/godotengine/godot/issues/19486#issuecomment-399559419

If we have assets define dependencies, then I think they should all be installed at the same level (in res://addons folder). For dependency version conflicts (two assets depend on the same asset but different versions of it) I think a simple resolution would be best, such as always going for the newer version.

On the otherhand, if we really want to implement dependency management with deduplication and version conflict resolution by allowing multiple versions of the same asset to be installed, then that's ok too! As if it is really robustly made, then it will be able to support expansion of assets ecosystem long term. Somehow I don't like the idea of having multiple versions of the same asset in my game project though. And it took npm so many versions until they got it right (I still prefer yarn though) so I don't think this task should be underestimated.

[Zylann]

I don't think putting dependencies as subfolder of assets is a good idea as it is leading to a path that will be very complex.
If we have assets define dependencies, then I think they should all be installed at the same level (in res://addons folder

I agree with that. I would also prefer addons to exist once in the projec in a standard location, not as many times as versions required by various other sources and in various folders. Dependencies are a thing, but we can find a middle-ground too instead of going maximum fine-grained, to keep things relatively simple.

These problems are why my initial suggestions said not to let assets define dependencies yet, but just have a way to list dependencies of the project, and let user manually install any dependencies of assets.

To me if assets define dependencies it would be primarily for the user to be quicker informed than having to look at docs or reviewing all plugins before installing. And eventually, have an option to download the dependencies at the same time, but just for convenience (especially when you install from scratch, you would need to install those anyways). We probably don't need to resolve dependency conflicts too, at least the install tool could inform the user if it can detect one.

[Two-Tone]

I feel like this whole dependency issue is being way over-thought about

@LikeLakers2 Proper dependency management and conflict avoidance is an important and hard issue. It's being discussed so much because of that

I don't foresee this conversation ending with the way it's heading currently.

Trying to just push this through without figuring out all the problems that dependency brings would be a very bad idea and just cause way more problems. This is a hard issue and it will take time.

[Two-Tone]

For dependency version conflicts (two assets depend on the same asset but different versions of it) I think a simple resolution would be best

@chanon My ideas are simple. What's complex about naming a folder and changing the relevant file paths in text documents? It's a one time cost for the user, makes it apparent what version of what you have installed, and is fairly easy to write.

such as always going for the newer version

That will lead to plug-ins breaking without warning as their dependencies update and change how they work. EG Plugin 1 and 2 both depend on Dep 1, but different versions. Plugin 2 depends on the current, latest version but plugin 1 depends on an older version that behaves differently than the latest version. Plugin 1 no longer works as intended or has bugs it didn't ship with.

[Two-Tone]

We probably don't need to resolve dependency conflicts too, at least the install tool could inform the user if it can detect one.

@Zylann Then we have the issue where the user can't install x plugin because y uses a different version of the same dependency. That will not be looked upon as a good thing by users

[rminderhoud]

I agree with @willnationsdev and @LikeLakers2

I feel like this whole dependency issue is being way over-thought about. Maybe we should come up with a few simple systems and have a discussion on each, or a vote? I don't foresee this conversation ending with the way it's heading currently.

Too much bike shedding. This is a game engine, not a game-engine-addon-delivery-platform. Just vendor dependencies in each of the addons' directories and call it a day. No stress about version conflict or management or w/e at the cost of some inefficiency. If the user of the addon is concerned about performance then they should be in a position to consider REAL performance optimizations like GDNative or modules. I also can't imagine why any addon would have a deep dependency graph. The only reason I can see an addon having a dependency on another is if it extends it in some way: SweetFpsController -> SweetFpsController w/ flying. In that case I would say it's up to the two projects or their users to reconcile their coupling rather than giving the responsibility to the engine.

[Zylann]

@Two-Tone if plugin A needs version 1 of X and plugin B needs version 2 of X, I would say Godot should not try to magically fix it. Here are simple solutions to this:

• Normally, version 2 of X should be backward-compatible with version 1, and that's how things should be to keep it simple (remember when I mentionned the mindset when developing plugins).
• If you can't use latest version of X, wait for plugin A to support it and download previous version (we need the assetlib API to handle getting different versions, not just the latest one, that shouldnt be hard to do). It's like mods, it doesn't have to be perfect.
• The plugin maker of X uploads it as a different plugin (because the API breaks compatibility), just like we had to when Godot 3 went out. So then you will have twice the same plugin, but it's fine because they are too different to be really the same ones anyways, and stuff depending on it will work too. It can even just be a folder rename by the plugin developer, not even the need to have two asset lib entries.

In the worst case scenario you would have to make this work manually (and PR compatibility to the plugin maker eventually).

If not these, that would involve two versions with different path being installed everytime (screw all references at each update), or core changes being added to the engine to alter how paths work (complicated).
I think if we can just get simple dependency management it will be quite nice already, since it would basically use the functionality and structure Godot already has (so it would be quite simple to implement as well), and do automatically things the user currently has to do manually anyways.

I'm not considering dependency management as a thing that should always work and be the entire responsibility of the engine (like npm), because that would be a lot of work. I consider it primarily as a convenience for things you currently have to do by hand with existing functionality, and plugin makers can release their work based on that simple system in mind.
If someone manages to implement nicely a system going way further than this, then it can be done later, because what I described above is workable before and is mostly simple improvements to the assetlib and the editor only.

[Two-Tone]

@Zylann How are any of those options better than what I've suggested? And there is nothing magical about searching for and appending text to a specific string.

Option 1 forces developers try and make sure their version updates don't break any other plugin that depends on it, which adds a lot of extra work on them. Who wants to develop plugins for something like that?

Option 2 forces users to not use the plugin they want because they have to either wait for the plugin to update, if it's even still being actively developed, or just not use it at all.

Option 3 once again makes things more difficult for the plugin developers.

Two versions with different paths isn't a hard problem. If they ever update, check to see what install packages have them as a dependency, rerun the search and replace (taking into account the old version number) on all the scripts for those packages, then finally unpack the new version.

I feel like people are trying to make this out to be more complicated than it needs to be.

[LikeLakers2]

Proper dependency management and conflict avoidance is an important and hard issue. It's being discussed so much because of that

@Two-Tone What I meant is that you're thinking way too hard on the issue even for it being a hard issue. A lot of the conversation (including my replies, admittedly, so please excuse me as this is going to sound hypocritical) have been about what's wrong with the ideas presented, not so much about what can be done to change it and make it a better plan. The ideas being suggested can't only have downsides, can they? It seems like everyone wants their way and their way only as well, which doesn't help matters.

Trying to just push this through without figuring out all the problems that dependency brings would be a very bad idea and just cause way more problems. This is a hard issue and it will take time.

I suggested coming up with a few simple ideas with the implication that we'd iron them out and then decide once we feel we've got a good system. I didn't think I had to explicitly point that out, but there you go.

---

I'm kinda annoyed at how long this conversation has gone on without any progress. Seriously.

[Zylann]

Two versions with different paths isn't a hard problem

That alone isn't, probably. But having two versions of the same plugin at a given time in Godot implies other things that become confusing to solve.

• Your paths no longer are what you think they are / changes need to be done to core, which is a concern for core devs mostly
• If the plugin exposes a custom type, which one does the editor displays?
• If the plugin is C#, how do you compile it without conflicting with others copies?
• If the plugin registers autoloads, how do you tell them apart?
• If the user creates data which uses the plugin, which one will it be using?
• Which one will show UI in the editor?

I think before we can even find simple solutions to those that fit the engine (and there may be more I didnt think about), we can already improve existing stuff in the editor and asset lib without breaking compatibility or introducing core changes.

[paulhocker]

+1 on everything in this thread - extending the plugin module and increase use of the asset lib is the way to go

[chanon]

After having trying out and learning Unity, I now have a clearer idea on what this should be like.

First is Unity now has a 'Package Manager' which they use to let users select which components of the engine they would like to install/uninstall. It looks like they may also use this for the Asset Store in the future.

Something like this would really help Godot to be able to "reduce bloat" while having "official addons". Unity is still super bloated even with their Package Manager while if Godot had one it could get even smaller. The "one exe" method of distribution might have to go though.

Also, normal "modules" maybe should be built as shared libraries (dll/.so files) so that they can be selected for install/uninstall in the package manager.

Projects and packages should have manifest files listing packages it depends on and Godot / the package manager would ensure that required packages are installed.

A public registry of packages would be needed.

This would enable the community to easily extend Godot's capabilities and easily build upon each other's works. I think it would promote community development a lot.

[mhilbrunner]

Not disagreeing with the overall merit of moving to the Package Manager approach, I have not made up my mind on that yet, but I want to point out that it has its weaknesses as well - one single executable can be way more integrated and well-tested, at least more easily.

With a Package manager, the testing surface grows exponentially for each plugin and version. This isn't that bad if the plugins are mostly small additions (like currently on the AssetLib) not hooking too deep into the engine core, but it quickly becomes so if you split up the core parts of the engine.

Unity already runs into issues, crashes and compatibility problems with different versions of their packages, their editors, or with having some specific package combinations installed (or not installed), or even with the order they were installed in.

This is a lot of additional complexity that needs to be managed well, lest we end up in dependency hell :)

[Eoin-ONeill-Yokai]

Regarding dependency hell, why not take a page from something like AppImage and try to avoid dependency problems by making plugin distributors responsible for managing asset / file dependencies? Here's how I see this problem...

Plug-ins should (ideally) be self contained elements of either logic (custom editors, custom nodes, dynamically linked libraries) or assets (models, materials, fonts, textures, etc) that should have their own sets of dependencies. Trying to figure out shared dependencies is a conundrum that, while people might like it, it means a lot more effort in order to maintain it. If you were to actually allow shared dependencies, you would have to approach this problem similar to the steam runtime and maintain a bunch of different libraries, plugins and assets as their own unique packages. You would also need to store multiple of the same packages on a server for multiple versions -- it would be a maintenance nightmare and it's no surprise that modern linux distros are more or less moving away from package managers.

A plugin could essentially be a single tar archive of all of the elements needed to for that plugin to function in isolation. They should work natively on their own with all dependencies archived inside as well. When godot sees this file, it checks for a manifest file inside of the root of the archived folder. This manifest could contain basic info (name, description, godot-version, icon, authors) but should also contain things like: update-info (link to download the latest version, maintained by plugin developers) which godot can use to automatically update the plugin (with users permission), project_website (link to repository or website to let users quickly find the source of a package), and class_names (list of classes within this addon that can be extended, used for scripting.) It could also point to the addons it uses as dependencies using relative path to an internal plugin file.

If we allow plugins within plugins, then the dependency problems will be (more or less) solved. Differing version of the same library will be in different directory paths and could be treated as different 'paths' on a given fork. The problems with this system, of course, is that you end up having a lot of duplicate information on a given disk. However, when packaging a game, godot could ideally go through all the plugin directories and merge file references that match via checksum (this can included tar-represented file structures?). This way, if two files are identital, godot will optimize the eventual output binary by merging the reference to a single file.

I'm sure this is far too naive of a solution for a problem this complicated, but I was thinking about the plugin system more recently due to my custom font patch. I think the best user experience is one where they don't have to worry about dependencies, and the developers are the ones who should be concerned with managing their own dependencies. This makes it easier for the user to manage: a project can have their addons represented as a single file in the file system dock, you can right click and get additional options (extract files (for development), update plugin (for easy updates), go to website (for easy community organization)) and also reference classes within a given 'container' (extends "[path-to-addon-tar]://ClassName")

Just my 2 cents on the matter, I'm sure there are some flaws with the way I'm thinking of it as well, but it could be an interesting solution and also be a decent way to cut down on the maintenance necessity for the Godot team.

[bojidar-bg]

Just a 2 cents based on the previous 2 cents (does this make it 4 milles?):
I guess .tar archives would be bit unfamiliar to most users (ahem, windows users) so it might be better to use .zip archives, as they have somewhat wider support.

That being said, what about distributing plugins in Godot's own .pck format? I does have some cons (such as being unrecognized by most applications), but that might be a chance to make it more standard and used.

[Eoin-ONeill-Yokai]

I guess .tar archives would be bit unfamiliar to most users (ahem, windows users) so it might be better to use .zip archives, as they have somewhat wider support.

That being said, what about distributing plugins in Godot's own .pck format? I does have some cons (such as being unrecognized by most applications), but that might be a chance to make it more standard and used.

Sure, I mostly meant .tar as an example of a generic packed archive type. It could and probably should use godot's .pck format.

[Calinou]

Why not use ZIP since it offers compression and is a standard format?

[willnationsdev]

@Calinou I agree. If we mandate that .pck files must be used, then people will have to use Godot Engine in order to pack things rather than just using whatever their preferred packing tool is to bundle up their plugin for usage. GitHub also downloads repositories as .zip files and it houses the largest collection of Godot plugins by far.

[paulhocker]

I agree here. I don't think another file type needs to be created. Zip files seem to work very well, and folks have the option of installing using the Godot interface thru the asset library, or they can download manually themselves. a .pck file does not allow that choice.

[golddotasksquestions]

In my ideal world, I would like to have two download buttons at https://godotengine.org/download:

• One sleek bare minimum download that meets reduz three requirements, and
• One bloat download with all approved and and regularly maintained plugins which have surpassed some quality control, so I know their API works at least as well with the core as the API from the minimum version.

Even more ideal:
Select and mix. Upon downloading the bare minimum, I can select minimum version + Plugin "Themes". Which are collections of common plugins needed for common usecases. (so for instance: 2DPixelart, 3D FirstPersonShooter, 3D ThirdPerson, RTS, RPG, 2.5D game, 2DCutout, Rougelike, ManagementSim, Sidescroller, Eventpackage, Shaderpackage, Scriptpackage, Assetpackace, ect). So when I start the engines essential plugins are already installed and ready to go and I know they work because they have been approved.
More advanced users could untick individual plugins within those Theme collections if they know they don't need them, or combine two or more Plugin Themes, or add additional single plugins to Themes from the pool of approved plugins to make a custom package.

Ideally, the result would be that https://godotengine.org/download creates from those choices _one download link on the fly_. A download link I could copy and share with whoever, who then could with one click download the _exact same Godot plugin configuration as me_.

Why? Because this would be a massive time saver and quality assurance for people who work in teams or groups, for educators and teachers in schools who have to deploy on multiple computers, for people who follow tutorials and want the same config as the tutor, for individuals who are happy there is one official approved plugin by the dev team for something that is commonly used, but not by everyone, and so not have to dig through Github and Reddit for days to find what they need, not knowing which is the most supported option out of the 6 they found.

A one-click bare minimum download should still exist for those who need the small size, as well as a max feature download who do not care about size but want maximum features to see what the engine can do.

[menip]

A simpler solution would be to ask to install additional [official only?] plugins when running editor for first time. Currently we have option to open AssetLib if project list is empty; why not have a dialogue popup to install official plugins if running Godot for first time?

[Calinou]

Closing in favor of https://github.com/godotengine/godot-proposals/issues/142, https://github.com/godotengine/godot-proposals/issues/577 and https://github.com/godotengine/godot-proposals/issues/607, as feature proposals are now tracked on the Godot proposals repository.

Edit from aaronfranke: Also see https://github.com/godotengine/godot-proposals/issues/554