Godot: Csv files are not included as resources when exporting project

[oskarious]

Godot version:
3.2.1

OS/device including version:
Windows 10 1909

Issue description:
https://godotengine.org/qa/71495/exporting-game-with-external-resources-csv-file
I have a project that parses a csv file. When exporting the project, the csv file is not included with the export. A workaround is to change the file extension to .txt (even renaming it to .tres worked when it came to parsing it, though the editor threw parsing errors when starting the project). Renaming it to .txt hides the file in the godot file tree, for whatever reason (I assume it is because it cannot be directly interacted with via the godot ui, but it is very counter intuitive to have a file tree not reflect it's actual contents, but that is a seperate discussion).

Steps to reproduce:

1. Open a project
2. Imprt a .csv file
3. Change import preset from "CSV Translation" to "CSV"
4. Open and parse the file content via script
   4a. Observe the file being parsed as expected
5. Export project and include *.csv in the include_filter in the export_presets.cfg file.
6. Run the exported project
   6a. Observe errors in the console stating something similar to

ERROR: File must be opened before use.
At: core/bind/core_bind.cpp:1956

Minimal reproduction project:
https://github.com/oskarious/godot-csv-problems

[charliewhitfield]

@oskarious , we do this successfully in our projects which work exported. See repository here. The .csv files are in ivoyager/data/solar_system.

Looking at your repository, I believe the problem is that you shouldn't be thinking of these as resources. If you look at ours, they are in a folder with .gdignore and without any .import files. They are not visible in editor browser, but are available in the export for File.open() (due to inclusion in export_presets.cfg as you have done).

Edit: On thinking about it, your approach seems more sensible, even if it doesn't work. Maybe text files should be resources? (I've been using an external editor for these without giving it much thought.)

[oskarious]

@charliewhitfield
Right, I hear what you are getting at, but even if it is not technically a resource being consumed directly by a node, it should still be included in the export no? My not very qualified guess (since I have only been working with godot actively for 2 or so weeks) is that it wants to treat the csv file as a translation file, since that seem to be the only way csv is used as a resource otherwise, and since I have not started with translations yet it is discarding it? It is of course also entirely possible that the exporter discards unused resources on export, but I have the export_filter set to all_resources.

Again, I have limited experience working with godot, so it is entirely possible I am misunderstanding the ideas behind resources and the file tree/file system.

And yeah, in terms of text editing I am also using an external text editor, so I have no issues interacting with the project, but since it seems like godot wants to be somewhat self contained it seems weird not to list stuff like .txt files. But again, that's not the core of this issue!

[charliewhitfield]

This jars my memory of other related problems I've had trying to read contents of directories in exports. Here are two that I remember:

• Image files (e.g., res://path/image.jpg) don't "exist" in the export directory. Yes, you can load it with load("res://path/image.jpg"). But it isn't there. The import file "res://path/image.jpg.import" is there. (This affected us because we search for and load images dynamically by file prefix name.)
• GDScript files are in the directory, but their extension is changed from ".gd" to ".gdc".

Well, maybe this does make sense after all. You don't want the original source image file in your export project taking up space. And I'm assuming the .gdc is a compressed version of the original. My guess is that imported .csv files are not there for the same reason that imported image files aren't there: they would take up unnecessary space in the export project.

Here is a function I use for debugging this issue. Just print in both editor run and export project to see differences:

# print(get_dir_files("my_directory_path"))

static func get_dir_files(dir_path: String) -> Array:
    # Use for debugging. Export removes files & changes file names!
    var dir := Directory.new()
    if dir.open(dir_path) != OK:
        print("Could not open dir: ", dir_path)
        return []
    var result := []
    dir.list_dir_begin()
    var file_name := dir.get_next()
    while file_name:
        if !dir.current_is_dir():
            result.append(file_name)
        file_name = dir.get_next()
    return result

[oskarious]

Right, that makes sense that it could be some compression/encryption on the resources messing with it. As for GDscript extensions, I would assume that has to do with the script_export_mode which can be Text/Compiled/Encrypted in the GUI, or 0/1/2 in the export_presets.cfg file. Still find it being strange behaviour in terms of imported csv files, but at least it seems to be intended behaviour.

[charliewhitfield]

I'll let an expert weigh in after this comment, but I think the bottom line is: Anything imported or with some special "export mode" will likely be missing or altered in an export project's directory.

[oskarious]

Yeah, I'd be interested in hearing what someone with more insight has to say as well, but until then I agree with your conclusion. I will also give your method of storing it in a folder with .gdignore a go later as well so I don't have to mess with different file extensions.

Thanks for your input so far though!

[KoBeWi]

I can confirm this, except it's a matter of "import" not "export". Imported files, like textures, audio or CSVs etc. are stored inside the .import directory. When you export project, the source textures/samples are not shipped with the game, but only their imported versions in specific formats (e.g. textures are kept as .stex). Each resource path is remapped to their imported file, so when you (pre)load something, you are actually loading the imported file.

This of course applies only to files with special importers, i.e. textures, audio, models and some other things. Text resources like scenes, scripts or materials are not imported this way (but they are converted to binary equivalents on export. Or most of them at least).

[oskarious]

Right, that makes sense that imported resources get's converted into, well, something other than it's source. I guess the confusion initially stems from the fact that one of the import presets for csv is just called "CSV" which sounds like it would be parsable as-is. I probably missed something in the docs, but until I got help from you guys in clearing it up, I had completely missed the distinctions between how the different file types are handled in the editor.

[fire]

It is possible to write a resource loader for csv, ini and json that converts them to dictionaries. There has been no proposal for it, but it is possible.

[Zireael07]

I am using a csv file that has the import set to "CSV", and from the sounds of what @KoBeWi wrote I will face the same issue once I actually get around to exporting the project... some solution is needed for data files, definitely.

[charliewhitfield]

@Zireael07 , I'm a little puzzled. What's the value of a "CSV data importer"? It's not needed for project export, as I described above. Is it just to sidestep the use of .gdignore? (That's good enough reason. Just asking in case I'm missing something.)

[Paulb23]

Similar kind of issue #38733