Godot: Add optional data (data://) folder to be accessed in a similar way as user:// except located in parallel to the exe

[exts]

Godot version:
All

OS/device including version:
All

Why:
This would add a traditional option to breakup your application. Lets take fallout for instance, a lot of the modding options are stored in a local folder parallel to the executable for ease of access to be modified, while they move user settings, hotkeys, etc... to the shared user folder to prevent loss of data. Even steam does a similar thing, you can literally delete every file inside the steam folder except the exe and it'll attempt to regenerate the necessary files back parellel to the exe's location.

Now you could do this manually by using the OS class, but I think it would be cleaner if existing api's like File and Json, etc... worked directly with this path.

How would you access this?
Through data://. all it would do is read/write data from /Data/ parallel to the exe. Nothing less, nothing more.

[PetePete1984]

As a fan of portable, self-contained apps that keep all their stuff inside their installation directory: yes please.

[Zylann]

Isn't this possible by simply using File without root specifier? Paths without res:// or user:// access the working directory, which is next to the executable since File just wraps the C file API. Btw, that can't be portable (web and mobile), but I think it's doable already, right?

[Faless]

Well, the problem I see with it is that some platforms do not have writeable executable folders.
E.g. Android, (and I suspect iOS), plus in some cases the executable will be installed in a folder where you need superuser permission to write on other platforms too.
Additionally, you could get that folder in game via some tricks:
OS.get_executable_path().get_base_dir() (will return the executable path, but in editor, it will be the editor path)
You could try with something like:

    var path = ProjectSettings.globalize_path("res://")
    if path == "":
        # Exported, will return the folder where the executable is located.
        path = OS.get_executable_path().get_base_dir()
    else:
        # Editor, will return one level above the project folder
        path = path.get_base_dir().get_base_dir()

[TGRCdev]

I like this idea, but where would data:// point to in a non-exported project? Would it point back to res:// ?

[exts]

@TGRCdev yeah that'd be the idea. Thing is we'd need a way to tell the exporter to exclude data folder if one exists from being included in the PCK/Zip and copy the folder outside of the .PCK - then it comes the 'what do we call this folder? Data, .Data etc...

@Zylann not sure I haven't tested, but then there's still the issue of what I posted in my message to TGRCdev - how do we deal with the exporter and yeah the other messages regarding mobile & html5 - but does that matter if this is tailored towards desktop applications anyways? If mobile/html5 return the res:// data folder ?

[aaronfranke]

then it comes the 'what do we call this folder?

How about ProjectNameHere_Data? It's unique, descriptive, and will not conflict with name or casing differences on various platforms.

[Zylann]

how do we deal with the exporter

The importer/exporter is not bundled with exported projects so if there is a data folder, it's gonna have to be accessed without the importing fashion of load, probably... which makes it a regular folder you can access already with File, as I said. Was this what you meant about dealing with the exporter? Otherwise I still don't get what feature a special "data://" specifier would give to the engine that can't be done already (I don't think OS is even needed, just use relative path omitting the xx://?). If for modding purposes it's doable that way, but it can't have the import step only the editor can do. Perhaps I'm not completely understanding the issue?

[exts]

@Zylann alright, so I tested what you were explaining and you're right, you can already do this by leaving off the res & user root identifiers.

using System.Linq;
using Godot;
using Godot.Collections;

namespace DTAA
{
    public class Items : Control
    {
        public override void _Ready()
        {
            ItemLoader();
        }

        public void ItemLoader()
        {
            var data = string.Empty;
            var file = new File();
            if(file.Open("Data/items.json", (int) File.ModeFlags.Read) == Error.Ok)
            {
                data = file.GetAsText()?.Trim();
                file.Close();
            }

            if(data.Empty())
                return;

            var json = JSON.Parse(data);
            if(json.Result is Array jsonArray)
            {
                foreach(var item in jsonArray.ToList())
                {
                    if(item is Dictionary itemDict)
                    {
                        GD.Print($"Item: {itemDict["name"]} with weight: {itemDict["weight"]}");
                    }
                }
            }
        }
    }
}

win64-bin.zip

So maybe I should rename/recreate this issue to: Allow us to include folders outside of and exclude from the PCK file since you have to manually do this yourself. Check the example I used above.

[LikeLakers2]

Rather than have a data:// path, why not instead have an option to toggle user:// between where it links currently, and a folder next to the EXE? To me, the use cases of a data:// path would be the same as the use cases of a user:// path... unless there's something I'm missing, of course?

But if you really need to differ between the two for whatever reason, I imagine Faless's code would do just fine.

[neikeq]

I also think what is described here are use cases for the user:// path. If the objective is to have a portable self-contained game, then it would be better to have something like Godot editor's self-contained mode rather than a new data:// path that's not guaranteed to work.

[exts]

@LikeLakers2 it's unnecessary since you can negate the user/res root from the path completely and do just that.

I gave an example in the post right above yours.

[aaronfranke]

The way I see such a folder is a way to store files to be easily user-modifiable while not user-specific. So it shouldn't necessarily be treated as writable. User-specific save data would go in user:// and game data would go in res:// but let's say for example that you want people to be able to insert a custom sprite sheet or something, then it's best not to have that inside the executable nor in user://.

@exts I see that you can just manually type "ProjectNameHere_Data/whatever/etc" and it resolves to a local path, but this might not always be true (what if the game is launched from a terminal with a different working directory?) and it would be better to have an "official" data:// to do this.

[exts]

@aaronfranke yeah you're correct. I called the executable from a parent directory from CLI and it couldn't find the .json file which means no shortcuts would work I'd assume.

[LikeLakers2]

(remaking my comment as I completely rewrote my responses after some thought)

[@exts] it's unnecessary since you can negate the user/res root from the path completely and do just that.

Okay, then there's a way to access this path (seemingly reliably) already. Why are we still wanting to add data:// then?

[@aaronfranke] The way I see such a folder is a way to store files to be easily user-modifiable while not user-specific.
...
but let's say for example that you want people to be able to insert a custom sprite sheet or something, then it's best not to have that inside the executable nor in user://.

I'm not sure why those user modifications have to be stored next to the executable... If a game doesn't support modding, then the users will already be editing the game's files -- and if it does, I'm not sure multiple users using the same system would want to share such modifications between each other anyways.

[exts]

[@LikeLakers2] If a game doesn't support modding, then the users will already be editing the game's files -- and if it does, I'm not sure multiple users using the same system would want to share such modifications between each other anyways.

What? This whole thread is about allowing users outside of the shared folder to add/modify things in a traditional fashion. Look at fallout, civ, etc... the only data in shared folder is settings the rest is game code exposed to the user in the application directory open for modification.

[LikeLakers2]

[@exts] What? This whole thread is about allowing users outside of the shared folder to add/modify things in a traditional fashion. Look at fallout, civ, etc... the only data in shared folder is settings the rest is game code exposed to the user in the application directory open for modification.

...If that's what you're talking about, then just take my previous proposal and apply the idea to res://. Exported files would go alongside the executable, instead of in a .pck.

I still don't see a use case that requires a dedicated data:// path.

[aaronfranke]

@LikeLakers2 Your previous proposal isn't good because it makes sense to separate stuff like save games, video settings, controls, favorite color, etc, which go in user://, from actual game data, like textures, or maps, or models, etc, that users can modify, which would be data://, or would otherwise go in res:// if the users are not supposed to be able to modify them.

[LikeLakers2]

@aaronfranke I said to apply the idea to res:// instead of user://:

just take my previous proposal and apply the idea to res://

[neikeq]

Alright, so from what I understand, you don't want a directory you can write to at run time, right?
This looks a lot like the res:// path, only with the exception it also works with the file system and not only the PCK when exporting the game.
Not sure how important this is considering you can already access the file system, only there is no prefix for the executable directory. You can call OS.get_executable_path().get_base_dir() to get that directory (OS.GetExecutablePath().GetBaseDir() in Godot with StringExtensions, or Path.GetDirectoryName(OS.GetExecutablePath())).
Another thing to keep in mind is that Godot's way of doing this kind of stuff like modding is through packed containers (pck/zip). Is there any limitation with these that forces you to use the file system instead? The only I can think of are libraries that can only receive a path to something on the file system, or shared libraries that need to be loaded.

[exts]

@neikeq I guess it's about using paths directly through existing api's which you can already do if you omit res:// and user:// as shown in my example above. @aaronfranke does have a point that when calling the exe from a different path other than it's current directory, it'll look from the directory you're calling the exe which _is_ a problem and I could see why he would want a data:// but that limits the file system way of doing things.

I think there could be a solution to my suggestion since the main OP has changed as new information has been presented.

1. Allow direct file system calls to be locked to the original exe's directory, this could be a setting in the project settings. So when you call File.Open("Direct/Path") and you execute the exe from cli like ./SubFolder/Game.exe it won't be looking for Direct/path inside of ./, but rather the location of the exe instead.

2. Give us the option to exclude specific directories/files (which I think you might be able to do already?) from the .PCK folder in the export project settings page while allowing us to completely copy those same folders/files outside of the .PCK folder along side the exe to be exposed to an 'advanced' user.

[LikeLakers2]

1. Allow direct file system calls to be locked to the original exe's directory, this could be a setting in the project settings. So when you call File.Open("Direct/Path") and you execute the exe from cli like ./SubFolder/Game.exe it won't be looking for Direct/path inside of ./, but rather the location of the exe instead.

That... sounds like it's ripe for confusion. Honestly, I don't even know if I'm understanding your suggestion correctly. I reread it some, and understand it now. I disagree with this suggestion, and it's because I think some platforms (i.e. Android, macOS) require game files to be within the executable. However, your second suggestion might be better, if I'm understanding it correctly:

1. Give us the option to exclude specific directories/files (which I think you might be able to do already?) from the .PCK folder in the export project settings page while allowing us to completely copy those same folders/files outside of the .PCK folder along side the exe to be exposed to an 'advanced' user.

So basically an option to export all the game files to be alongside the executable, rather than into a .pck file? I could get behind that, but again, I believe some platforms require game files to be within the executable.

[exts]

@LikeLakers2 this wouldn't be used in mobile applications or html applications _ever_ because of how those projects are packaged. This is purely a change for desktop games.

1. yes

[LikeLakers2]

[@exts] this wouldn't be used in mobile applications or html applications ever because of how those projects are packaged. This is purely a change for desktop games.

Okay, but then at that point, why aren't you just using Faless' or neikeq's suggestions? If it's not going to be portable to all platforms, there's no reason we should be adding a new data path...

[exts]

@LikeLakers2 are you still stuck on data:// ? lol none of my posts talk about data:// after my C# example. I advise you to reread starting from my c# example. https://github.com/godotengine/godot/issues/22825#issuecomment-427816708

[LikeLakers2]

If you're complaining that you need access to a folder next to the executable, then use Faless' or neikeq's suggestions. So far that's what you're wanting, right? I've read and reread your comments and every one of them seem to imply you want access to a folder next to the executable.

Otherwise, I have exactly zero idea what you're talking about, and you'll need to explain to me what you're referring to because I cannot for the life of me figure out what this issue is about.

[exts]

@LikeLakers2 you're not really contributing to the discussion. I've been pretty clear on what I've been saying post my c# example now and you don't seem to understand anything I've written previously.

[LikeLakers2]

I'm sorry I don't understand what you've been saying. You seem to want me to understand, given how you told me to read from your C# comment, which I did. Yet I still don't understand. The only thing I can ask you to do is to try to explain it to me once more, because clearly I'm not understanding what you've said previously.

But that response makes me feel like you think I'm trying to misinterpret you, when I'm just trying to provide some feedback based on how I've interpreted this issue so far... and, y'know, maybe try to help you towards the goal you want.

So just forget it, I'm done trying to understand this issue if you're not going to help me, since I'm clearly unable to understand whatever this issue is about.

[exts]

I'll do you a solid and close this issue and create another one referencing this issue because my suggestion has clearly drifted from the original post about halfway through this comment chain after information provided by @Zylann.