Dune: Making hashing of values faster

[jeremiedimino]

Looking at the compiler C code, it seems that it should be easy to provide a C function that does the same as fun x -> Digest.to_string (Marshal.to_string x []) but with using constant memory. We should look at adding such a function in the stdlib, that should make Dune a bit faster.

[rgrinberg]

Note that this many not be enough for things like digesting maps properly. Since we'd like to digest things in the key order. Ideally, a new primitive would allow us to express that.

[jeremiedimino]

Currently the flow goes as follow:

ocaml value -> ocaml value in canonical form -> string -> digest

The proposed function would allow to get rid of the string step, but not the ocaml value in canonical form step. To get rid of this step as well, we would need an API to incrementally compute a digest. This API already exists in the C code but is not exposed to the user.

[nojb]

I could spare a bit of time to work on this suggestion, if it helps. I am very very interested in anything that can make dune faster.

[jeremiedimino]

@nojb that would be great! To be clear, I don't know exactly how much we'd win, but since dune computes quite a lot of digests, I believe the win wouldn't be negligible. At least it would reduce GC pressure.

[nojb]

Yes, I suspect the effect won't be dramatic, but it can't hurt. Also, it just seems a useful function to have.

I will give it a try when I can and report back here.