Deno: Garbage collection for DENO_CACHE

[caijw]

change a ts file and recompile it , the prev xxxx.js cached file in ~/.deno/gen is still there.
I think it is some kind of files leak(deno creates files but does not clean them when they are out of date).

[ry]

Yes, there's no automatic cache purge. I've been putting it off until someone asked for it : ) so thanks for asking.

[bartlomieju]

Side question: you run program with Deno, then you update Deno to newer version. Should program still run under new version without problems using old gen files? I believe I hit problem with this once, unfortunately can't reproduce it now. Of course purging cache helped the issue.

[caijw]

Yes, there's no automatic cache purge. I've been putting it off until someone asked for it : ) so thanks for asking.

Do you hava any idea for this issue?
I am thinking about maintaining a file gen.cache to recode the origin filenames and their current hash filenames.
When code_cache a compiled file , update gen.cache for its hash filename.
When load_cache, read gen.cache to get its hash filename, and then read the cached file.
When deno's main thread existing, make a request to clear the cached file which is not recored in gen.cache .

[kevinkassimo]

@bartlomieju The problem last time was due to native ES module upgrade. The source files were not changed so the hash keeps the same, although the compiled version is no longer runnable under new deno release. A way to fix this is to include deno version (or the last compatible version) along with the source file content for hashing.

Regarding auto-purging-on-exit, we could possibly, instead of scanning the whole gen.cache to remove things not recorded in the last step, just remove ones that just got replaced in the code_cache step. However, storing a mapping from source file to compiled file, especially stored as a file, seems would not scale very well and adds penalty on each run as the list grows longer. On the other hand, there are some problems with how to clean compiled code for files that are deleted: the entry is still in the gen.cache, just the source file used as the key is deleted/moved to another location/renamed...

Alternatively, we can introduce an extra flag that cleans the out-of-date/invalid cache, like git clean. We can use both mechanism of hash comparison and last accessed date to determine which files to remove. In this case, we also don't need to maintain a dedicated gen.cache for the mapping, instead we can add a comment in the generated code that points to source location (so that we can check for the updated hash or just verify if the file still exists).

[caijw]

@bartlomieju The problem last time was due to native ES module upgrade. The source files were not changed so the hash keeps the same, although the compiled version is no longer runnable under new deno release. A way to fix this is to include deno version (or the last compatible version) along with the source file content for hashing.

Regarding auto-purging-on-exit, we could possibly, instead of scanning the whole gen.cache to remove things not recorded in the last step, just remove ones that just got replaced in the code_cache step. However, storing a mapping from source file to compiled file, especially stored as a file, seems would not scale very well and adds penalty on each run as the list grows longer. On the other hand, there are some problems with how to clean compiled code for files that are deleted: the entry is still in the gen.cache, just the source file used as the key is deleted/moved to another location/renamed...

Alternatively, we can introduce an extra flag that cleans the out-of-date/invalid cache, like git clean. We can use both mechanism of hash comparison and last accessed date to determine which files to remove. In this case, we also don't need to maintain a dedicated gen.cache for the mapping, instead we can add a comment in the generated code that points to source location (so that we can check for the updated hash or just verify if the file still exists).

Adding a comment in the generated code that points to source location is a great idea!
And i think cleaning should be transparent to users.deno can run cleaning job background and the users will never know about it.

[AbmSourav]

is that clear cash feature available on 1.0.0 ?
It's a pain "removing cash from .cash " :(