Rescript-compiler: clean staled artifacts in idle time or failed mode

[asolove]

(First, let me say: thank you so much for Bucklescript! I learned OCaml several years ago for fun, and at the time thought "too bad the ecosystem isn't ever going to catch up and be relevant on the web". I'm so glad I was wrong.)

(I'm very new to BuckleScript, so it's possible what I have reported as a bug is actually something well-known that experienced bucklers work around through configuration or workflow in some way I don't understand. I'd love to find out what that is!)

I noticed an odd behavior when working on refactoring my first bucklescript project and wanted to check if this is expected. It appears that the build cache in lib contains built versions even of files that have since been deleted or renamed, and that builds therefore succeed when they should fail.

The symptom

# Initialize bare bucklescript template
> bsb -init test-bsc-cache && cd test-bsc-cache
# make a module "A"
> echo "let a = 1" > src/a.ml
# use that from the main module
> echo "let () = Js.log A.a" > src/demo.ml
# build succeeds!
> npm run build
# now rename module A to B
> mv src/a.ml src/b.ml
# build
> npm run build
# ... succeeds ... lots of output
# but if we delete the cache
> rm -rf lib
# and then rebuild
> npm run build
# ... "Error: Unbound module A"

If you want to inspect it, I saved the state right before deleting the cache.

The problem seems to be that src no longer contains a file a.ml, but the corresponding a.js file still exists at lib/js/src.

The impact

The upshot of all this is that when refactoring, the interactive build and type tools sometimes report success when they should generate an error. And builds are somewhat inconsistent in one of two ways:

• If lib is included in the source control repo, then the build will work the same for everyone, but perhaps quite mysteriously since modules can be referenced despite not appearing in the source code.
• If lib is not included in the repo, then builds are nondeterministic across machines based on the state of their cache.

The fix

I _think_ that when doing incremental builds, files that have been deleted/renamed should be cleared out of the cache before changed files are rebuilt.

I would be happy to work on the PR if that change is desired and someone can point me in the right direction.

[bobzhang]

removing staled build artifact is a hard problem, ocaml made it worse since the staled build artifact could affect the compilation
One thing I did not get is "builds are somewhat inconsistent", for a clean build it should be consistent?

[asolove]

Ah, ok, if this is a known hard problem, I'm fine with just running clean builds when doing these kinds of changes. Sorry for the noise and thanks for the answer!

[bassjacob]

since bucklescript compile is so fast, and deletes are relatively rare, what about running clean if a file is deleted, inside the watch loop? so something like:

fs.watch: event
  if event is create:
    build file
  if event is modify:
    build file
  if event is delete:
    clean
    build

[bobzhang]

hi @bassjacob that's something I have in mind, when in idle or failed mode, do some cleaning, I re-open this issue to keep track of it

[bassjacob]

I'm happy to take a look at this. It should just be some behaviour mods to https://github.com/BuckleScript/bucklescript/blob/c1d63143d9589c522558f148c183801b52ac93ba/jscomp/bin/bsb_watcher.js correct?

If so, I'll have some time in the next day or so, and give it a shot.

[bobzhang]

it can be done, but I guess you need some help from bsb, for example bsb -query modules give you a list of modules, based on it, you can remove staled artifacts

[bobzhang]

@bassjacob I will add bsb -query modules next week

[bobzhang]

hi @bassjacob , see #1784 (cc @chenglou )

[bassjacob]

just saw the merge. this sounds good, will discuss on discord with you how to start.

[bobzhang]

"files that have been deleted/renamed should be cleared out of the cache before changed files are rebuilt."
Good news is that if you choose in-source : true, and suffix : ".bs.js", bsb will remove staled binary artifacts, let me know if you still have problems with such settings