Godot: [Discussion] Storing md5's in the .import files causes issues with build systems and VCS's.

[sdfgeoff]

Currently, the file's md5 is stored along with the import settings in <filename>.import
Examples:

{snip}
[deps]
source_file="res://icon.png"
source_md5="40cd50be8aff465444de5539bc5f8ccf"        <---- MD5

[params]
compress/mode=0                              <----- Import settings
compress/lossy_quality=0.7
compress/hdr_mode=0
{/snip}

Now, many file contain timestamps, such as .dae files:

{snip}
<asset>
    <created>2018-02-28T08:58:31</created>
    <modified>2018-02-28T08:58:31</modified>
</asset>
{/snip}

Storing the md5 as part of the .import file means that if you have a build system that generates assets (eg converts blends into dae files), then every time the build system is run, the output has a different md5, and so every different developer ends up with their version control system notifying them that the .import files have changed.

_This problem stems from the fact that the .import files need to be version controlled because they contain import-relevant data, but the md5 of the file frequently changes irrespective of the import settings._

---

I'm happy enough to put in some development time to fix this, however the mechanism of storing the md5's will need to be decided.

I see two options (there may be more):

• Put the md5's in the .import folder with the other auto-generated data (not checked into the VCS)
• Create a new file <filename>.md5 for each resource (which can be ignored by the VCS)

Which mechanism would be better? Are there any that I haven't thought of?

[NathanWarden]

These really should be put in a folder that can be ignored by the version control system. These are very useful to have, but are really only relevant to individual machines.

Also, the default Line Ending installation settings for git on Windows, macOS, and Linux are different. So, if you're developing on multiple machines it starts to pointlessly update these files (for dae files in my case) since they'll have LF on one machine and CRLF on another machine, thus changing the hash. If these could be ignored per machine this issue would go away.

So, what happens for me is:

Linux checks out dae files with LF and thus creates HashA on import.
Then I go over to Windows which defaults to CRLF, so it checks out the dae files with CRLF creating HashB.

This can be resolved with a line ending file in the repo, but someone who doesn't know that is going to thing Godot is to blame, not their version control system setup.

I really think these should be stored somewhere like .import/hashes or something like that, or maybe even just make another folder like "res://.local/hashes".

As a side effect, this folder could also be used by plugins that need to have per project, per machine specific settings.

[akien-mga]

I agree that this is necessary, I had argued for that before 3.0 and failed to convince reduz, but now that more users have experience with this workflow, I think we can give it another try ;)

[reduz]

This is a very specific and corner use case that does not make it worth
changing the entire import system and making it more complex.
We could add a tag or option to import files to disable auto import, or
control how this proces happens better, so this is not a problem.

On Mar 1, 2018 10:16, "Rémi Verschelde" [e-mail] wrote:

I agree that this is necessary, I had argued for that before 3.0 and
failed to convince reduz, but now that more users have experience with this
workflow, I think we can give it another try ;)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/17103#issuecomment-369588288,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z28r_BoT0Rb61_hIAOX8DImrnLy_oks5tZ_S9gaJpZM4SWoeR
.

[sdfgeoff]

@reduz I would assume that having your project in a VCS is not a "corner use case." And currently, the .import files are difficult to version control because they contain information that is needed (import settings), but they also contain information that is too volatile and results in frequent merge conflicts (md5's). I would not consider disabling auto-import as a solution to this problem - it may make it occur less frequently, but it does nothing to prevent collisions between supposedly unchanged import configurations.

If you have any suggestions about other methods of keeping these pieces of data separate, then I'm completely happy to implement it - but it is a problem that has already caused some grief over the two weeks I (as part of a 3-man team) have been working with godot.

I do think I will change the path from <filename>.import.md5 to something in the .import folder. This will help avoid cluttering the users res:// directory.

[reduz]

I would assume that having your project in a VCS is not a "corner use case."

Of course not, but you specifically stated that your problem is:

Storing the md5 as part of the .import file means that if you have a build system that generates assets (eg converts blends into dae files), then every time the build system is run, the output has a different md5

Which does seem like a corner case to me. I don't see why the import system would not want to re-import this file, no matter the use case. It seems to me the only solution is disabling automatic re-import.

[vnen]

@reduz

I don't see why the import system would not want to re-import this file, no matter the use case. It seems to me the only solution is disabling automatic re-import.

I think that's not the problem. The problem is that the .import file will change and since those are also in the VCS, you generate a meaningless diff. If the MD5 is stored elsewhere, the asset will be re-imported anyway, but it won't generate any diff in the VCS.

[sdfgeoff]

I think this more an issue in 3D games (where you have artists working in separate formats with a defined export process) than in 2D games (where the artists work with the art more directly).

The question comes down to: What things should be version controlled?

• On one extreme, you could version control the .import folder, and tell your VCS to ignore the asset files, but then you need to transfer the asset files via some other method.
• Or you can track the asset files and let Godot handle the import on each developers computer.
• With a build pipeline, you take it one step further, and generate the assets. In our case, we version control the .blend files. We don't have the assets themselves as part of our VCS.

Following the "only track source files" thought, the md5 is itself a generated value, and thus shouldn't be VC'd. Unfortunately, the md5 (generated) is currently stored in a file along with other data that is not generated, and thus needs to be tracked.

This stems from godot assuming that the md5 is a good approximation of global asset state, and that a change to the asset files md5 indicates a change to the asset itself. However, due to differences in operating system (as in NathanWardens issue), or timestamps (as in the build system issue), this assumption can be invalid.
However, the md5 is a good approximation of local asset state. It can be used to identify if the file has changed for a specific user, but not if it has changed globally for all people. Changing the md5 from a global to a local state is as simple as removing the hash from the VC system.

[reduz]

The problem is that the .import file will change and since those are also in the VCS, you generate a meaningless diff. If the MD5 is stored elsewhere, the asset will be re-imported anyway, but it won't generate any diff in the VCS.

I see, so the solution would be to store, besides the .import file, just the text with the hash files in the .import folder?

That probably does make sense.

[NathanWarden]

@reduz I think that's a good idea so we don't clutter the main project directories with more files.

[reduz]

@sdfgeoff, @NathanWarden apologies, I misunderstood your use case, It definitely makes sense to move this to .import folder.

I can work on it when I have a bit of free time, otherwise if anyone wants to give it a try, code is in editor/editor_file_system.cpp

[sdfgeoff]

@reduz: I already have a PR open for a trivial implementation (#17138 filename.import.md5), but I agree with everyone here that a better place is the .import folder.

I'll give it a shot at moving it to the .import folder tonight, but will likely end up coming to you for advice on IRC.

[akien-mga]

I agree that reworking #17138 to actually keep the files in the .import/ folder would be best.

[sdfgeoff]

17138 was merged, and I've been using a branch with that PR in it for the past while, and it has filled my needs. So this issue is resolved.