Godot-proposals: Establish convention for sharing libraries

[nhydock]

Describe the project you are working on:
This applies to any project that would like to install dependency or assets developed by others.

Describe how this feature / enhancement will help your project:
Right now there is no established convention to enforce for managing packages that may contain source code, assets, or plugins. This applies both to how people are expected to create a repository for it to be used by a Godot project, as well as where a godot project should store its downloaded modules.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
An example of how a project could store dependencies, as well as how those dependencies can acceptably be organized to be recognized by Godot.

project/
├── godot_modules/
│   ├── actor-module/
│   │   ├── package.json 
│   │   ├── index.gd
│   │   ├── Actor.tscn
│   │   └── placeholder.png
│   └── simple-scene-manager/
│       ├── lib/
|       |   ├── SceneManager.tscn
|       |   └── SceneManager.gd
│       ├── assets/
|       |   └── transition.png
│       └── package.json
├── assets/
└── scripts/

Describe implementation detail for your proposal (in code), if possible:
This is more of a community pattern to describe and enforce than it is an actual code feature. However, if it's properly enforced, then the engine could also start to depend on the known convention and implement custom loaders for efficient coding, such as how npm modules can be required from root level namespace instead of relative paths.

Without a properly defined convention for sharing modules, what's checked out can become messy. My least favorite pattern I see in repositories is including a subfolder in the git repo with the same name as the repo. This makes paths in code look redundant when loading, ie.

const ActorClass = preload("res://godot_modules/mymodule/mymodule/index.gd").Actor

With the ability to define packages using with a meta file and recognize the convention with the engine you could simplify this to

const ActorClass = preload("dep://mymodule").Actor

Even resources in the file browser could be organized under a new tree for dependencies, and these files can be treated as Read-Only by the editor to prevent accidental tampering. Resource Loading could be referenced with appropriate abbreviated paths too in the TSCN and TRES files.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Best solution I've found, since not everyone publishes to the Godot Asset Library, is to use git submodules. This also allows for handling any modules that a company may keep internal for personal projects.

Is there a reason why this should be core and not an add-on in the asset library?:
If leveraging a defined convention to redefine how loading is perfromed, it must be implemented at a core level to perform lookups. Using something like npm's package.json to define where "sources" or an "index" exists is helpful as well.

[Calinou]

See also https://github.com/godotengine/godot-proposals/issues/1205 and https://github.com/godotengine/godot/issues/19486.

I'd be wary about using the package.json name, since many editors and programs will try to detect the add-on as a npm package. Also, we can reuse the ConfigFile class for reading and writing configuration files, so we can use a .cfg extension like we already do with plugin.cfg.

[fire]

Personally I use a different style of directory layout. https://gist.github.com/fire/f3981ad4a48ccae08224d3b82b9f95bc

It's incompatible with this layout.

If you force a layout, I don't know which one to pick..

One alternative that was mentioned was that each addon acts as a .gdignore and as an entire project.

[nhydock]

The layout itself of packages is not one I'd want to see enforced aside from each package gets its own folder installed inside a consistent location, in my example it being godot_modules. Packages should be left to their own on how they want to structure their contents, but encouraged to organize at the root level of their repo so it can be included as a git submodule or zipped up cleanly to be dropped in. Bad layouts that I've seen have been the expectation that you take the repo and overlap it with your own project, instead of nesting it somewhere safe.

Treating each package's structure as a standalone project would be pretty ideal, and matches the typical approach to packaging dependencies as found in other language ecosystems. The concept of "addons" feels fuzzy within Godot as the common interpretation of addons is more like plugins, however some packages may just be asset collections with no scripts. This is why I tend to refer to dependencies as packages or sometimes as modules.

[willnationsdev]

@nhydock

This is why I tend to refer to dependencies as packages or sometimes as modules.

Well, "modules" as a term is a Godot Engine module, as in, the godot/modules directory of the source code.

Packages should be... encouraged to organize at the root level of their repo so it can be included as a git submodule or zipped up cleanly to be dropped in.

I agree. I think establishing this as a convention for "packages", as you put them, would be a good idea. Assuming "package" refers to any redistributable directory containing assets/plugins or engine modules.

Bad layouts that I've seen have been the expectation that you take the repo and overlap it with your own project, instead of nesting it somewhere safe.

I'm definitely guilty of this, but mostly because I was under the impression that that was the convention already, up to this point.

The concept of "addons" feels fuzzy within Godot as the common interpretation of addons is more like plugins, however some packages may just be asset collections with no scripts.

I feel like project/addons is simply "the place for third-party code in a project", much like your godot_modules concept in the post. An addon doesn't necessarily include a plugin and I don't think there is an assumption that they are all plugins. This "common place for storing packages" already exists more or less imo. And if you renamed it to project/packages or something it would just 1) confuse people who were used to it being "addons", and 2) start an identical process of making people consider "packages" as an alternate term for plugins as the vast majority of addons still include a plugin, regardless of what you choose to call them (addon vs. package).

I'm pretty sure reduz still refers to them as addons too.

If anything, what I think would resolve this Issue is simply having an official "this is how you structure an addon repository" page in the documentation - separate from the "how to make a plugin" page, but perhaps with a note on that page that links to it or something.

[Calinou]

For the record, this is how I distribute my add-ons in a way that's compatible with Godot 3.2.x and doesn't require unchecking any files in the asset library's installation dialog:

Separate the add-on code from the demo

The most important concept here is to separate the add-on code from the demo. They must reside in separate repositories, with the add-on code being manually checked into the demo repository to ensure it works out of the box. Make sure to keep the add-on code inside the demo repository up-to-date; for instance, you can synchronize it on every tagged release. You can't use submodules to point to the add-on repository until https://github.com/godotengine/godot-proposals/issues/554 is implemented.

That said, even if the expected asset library structure is changed to accomodate for Git submodules, keep in mind ZIP downloads will not contain submodules (this is a GitHub limitation). This means downloading the demo from the project manager's Templates tab will no longer result in a functional demo out of the box. Therefore, I would recommend not using Git submodules even if they will be supported in the future.

Add-on

From godot-lod:

├── addons
│   └── lod
│       ├── LICENSE.md
│       ├── lod_cpu_particles.gd
│       ├── lod_cpu_particles.svg
│       ├── lod_cpu_particles.svg.import
│       ├── lod_omni_light.gd
│       ├── lod_omni_light.svg
│       ├── lod_omni_light.svg.import
│       ├── lod_particles.gd
│       ├── lod_particles.svg
│       ├── lod_particles.svg.import
│       ├── lod_spatial.gd
│       ├── lod_spatial.svg
│       ├── lod_spatial.svg.import
│       ├── lod_spot_light.gd
│       ├── lod_spot_light.svg
│       ├── lod_spot_light.svg.import
│       ├── plugin.cfg
│       └── plugin.gd
├── CHANGELOG.md
├── CONTRIBUTING.md
├── icon.png
├── icon.svg
├── LICENSE.md
├── README.md
└── TIPS.md

At the top level, there's a .gitattributes file with the following contents:

# Exclude all top-level files and directories (except addons) from ZIP downloads.
# This makes installing through the AssetLib easier, because no files and folders
# need to be unchecked.

/.editorconfig export-ignore
/.gitattributes export-ignore
/.gitignore export-ignore
/.pre-commit-config.yaml export-ignore
/README.md export-ignore
/LICENSE.md export-ignore
/CHANGELOG.md export-ignore
/TIPS.md export-ignore
/CONTRIBUTING.md export-ignore
/icon.png export-ignore
/icon.svg export-ignore

There is no .gitignore file.

Demo project

From godot-lod-demo:

├── addons
│   └── lod
│       ├── LICENSE.md
│       ├── lod_cpu_particles.gd
│       ├── lod_cpu_particles.svg
│       ├── lod_cpu_particles.svg.import
│       ├── lod_omni_light.gd
│       ├── lod_omni_light.svg
│       ├── lod_omni_light.svg.import
│       ├── lod_particles.gd
│       ├── lod_particles.svg
│       ├── lod_particles.svg.import
│       ├── lod_spatial.gd
│       ├── lod_spatial.svg
│       ├── lod_spatial.svg.import
│       ├── lod_spot_light.gd
│       ├── lod_spot_light.svg
│       ├── lod_spot_light.svg.import
│       ├── plugin.cfg
│       └── plugin.gd
├── default_env.tres
├── fps_label.gd
├── icon.png
├── icon.png.import
├── icon.svg
├── icon.svg.import
├── LICENSE.md
├── lod_mesh_instance.tscn
├── lod_particles.gd
├── project.godot
├── README.md
└── test.tscn

At the top level, there's a .gitattributes file to hide add-on diffs on GitHub. This is optional but recommended:

# The add-on code in this repository is copied from
# <https://github.com/godot-extended-libraries/godot-lod> regularly.
# Please open pull requests for the add-on code itself there, not in
# this demo repository.
/addons linguist-vendored

.gitignore file:

# Godot-specific ignores
.import/
export.cfg
export_presets.cfg

# Imported translations (automatically generated from CSV files)
*.translation

# Mono-specific ignores
.mono/
data_*/

---

Hopefully, this should provide you with a starting point to create easily reusable add-ons that can still be tested easily by anyone from the project manager's Templates tab.

[fire]

Personally I like the requirement Epic Games does on their demo projects to force the contents of the demo to be an easily removable and unique parent folder.

godot-lod-demo:

├── addons
│   └── lod
│       ├── LICENSE.md
│       ├── lod_cpu_particles.gd
│       ├── lod_cpu_particles.svg
│       ├── lod_cpu_particles.svg.import
│       ├── lod_omni_light.gd
│       ├── lod_omni_light.svg
│       ├── lod_omni_light.svg.import
│       ├── lod_particles.gd
│       ├── lod_particles.svg
│       ├── lod_particles.svg.import
│       ├── lod_spatial.gd
│       ├── lod_spatial.svg
│       ├── lod_spatial.svg.import
│       ├── lod_spot_light.gd
│       ├── lod_spot_light.svg
│       ├── lod_spot_light.svg.import
│       ├── plugin.cfg
│       └── plugin.gd
├── godot-lod-demo
│        ├── fps_label.gd
│        ├── lod_mesh_instance.tscn
│        ├── lod_particles.gd
│        └── test.tscn
|         icon.png
|         icon.png.import
│         icon.svg
│         icon.svg.import
│         LICENSE.md
│         project.godot
│         README.md
│         default_env.tres

[Calinou]

@fire The current asset library doesn't allow marking an asset as being both a demo and an add-on. This means you won't be able to see it both in the project manager's Templates tab and the editor's AssetLib tab.

[fire]

That is a defect and I'm going to assume that marking an asset as being both a demo and an add-on will be fixed some day.

I want to propose this structure for that future.

[Calinou]

That is a defect and I'm going to assume that marking an asset as being both a demo and an add-on will be fixed some day.

This is not handled by the asset library rewrite I'm working on yet, so it'd have to be implemented there. Also, it could be a bit confusing as the add-on and demo project would share the exact same description in the preview page.

[fire]

I recommend that the pure addons be made demo projects and all addons should have a demonstration project. However, I leave that up to you. The epic store is able to enforce minimal requirements that we don't have ability to do. So I don't know.

[Calinou]

I set up a template repository for Godot add-ons. You can use as a base for your new repositories: https://github.com/Calinou/godot-addon-template