Dhall-haskell: dhall thaw

[FintanH]

I ran into a workflow issue while using dhall freeze with our configs.

1. Ran dhall freeze to freeze all files
2. Had to merge changes that were upstream
3. Now files didn't compile due to caching and difference in a record
   
   
   
   • A field was added (due to the merge) post-freeze which was not present in the cached version
   
   
4. Deleted cache to see that integrity checks were mismatching

Thus, I need to to thaw or refreeze all the files to get the new hash. This would also apply to when we want to develop on frozen files and then lock them again incorporating the new changes.

[jneira]

And what about if dhall freeze itself would update the existent hashes?

[FintanH]

I feel like I'd be happy with that, because you mostly want the compile to fail if the integrity check failed. You want the freeze to always freeze the current state.

[FintanH]

One workaround is via find . -type f | xargs sed -i -r 's/ sha256:[a-f0-9][a-f0-9]*//g'

[Gabriel439]

Yeah, I agree that the ideal behavior is that dhall freeze just ignores any existing hashes and updates all of them.

[joneshf]

Both sound useful! It seems like there's still a case for something like dhall thaw. Sometimes you do want to throw out the shas. It can be useful so other commands and executables can work. It can also be useful when going between dhall versions (as the algorithm changed recently).

I guess in both of those cases, there are different things that could happen: allow other commands/executables to ignore shas and backport refreeze behavior to older versions, respectively. Maybe I'm not supporting thaw enough, but those are two recent problems I ran into.

[sellout]

I don’t have strong feelings on this, so feel free to disregard, but it seems like one might want more fine-grained control. I.e., not have freeze just update all hashes. I could see this being like freeze adds hashes to un-hashed things (the current implementation, yeah?), so you can selectively remove shas, then freeze, updating only thase. If you want to re-hash everything, you need to either remove shas in advance (like the thaw proposed), or maybe have a freeze --force to overwrite shas.

Alternatively, I could see passing a prefix like freeze --prefix 'https://github.com/.../Prelude/' that would update the shas for any imports that match the prefix, so you can update a single repo or something.

[Gabriel439]

@joneshf: If the goal is to re-freeze then I think we can address that by having dhall freeze ignore existing SHAs when importing the values (which I think would be a fine default behavior).

@sellout: Note that having SHAs for multiple versions is currently a violation of the standard as the interpreter has to use one version consistently across the entire program.

The way I see it, then main use case for dhall thaw is stripping the SHAs with no intention of replacing them (i.e. you want to pick up changes to your dependencies as you iterate on them).

[sellout]

@sellout: Note that having SHAs for multiple versions is currently a violation of the standard as the interpreter has to use one version consistently across the entire program.

I’m not sure what that means. Versions of what?

I’m talking about wanting to update some dependency, but not other dependencies. Not that the version of the Dhall spec used for hashing should differ between hashes.

[Gabriel439]

@sellout: I think I misunderstood. I assumed that you meant selectively updating dependencies to support them being imported using different versions of the standard but now I think what you mean is not updating them all for efficiency reasons (to avoid having to refetch all of them)

[sellout]

Yeah, I’m actually having trouble coming up with a practical use case that exercises my point. So I’m going with what I’ll call “the bad steward” problem:

Person A (the bad steward) publishes a dhall library, person B adds imports pointing to it and runs freeze on their code. Person A then _replaces_ the files in the library at the same URLs with wildly incompatible versions. Person B still has the good versions in cache, so as long as the shas aren’t overwritten, person B’s code still works, loading from the cache. But now they want to update other imports … if freeze rewrites all shas, they have some hoops to jump through to maintain the old ones pointing to person A’s library.

Maybe the correct solution to this is for person B to first deserialize the files from the cache, and update their imports to point to the local copy that they can manage better themselves, rather than trying to preserve a cache that is expected to be pretty transient.

I don’t think this is a particularly good use case to try to preserve, so I guess I’m retracting my suggestion that we not re-freeze everything?

[Gabriel439]

@sellout: Yeah, in my view, caching a semantic integrity check is an optimization, but not one that a program should rely on for its own resilience. If the underlying import is broken then the program is broken and we shouldn't take great pains to keep it alive. Also, in general deleting the cache should be a safe operation. Similarly, relocating a referentially transparent program (i.e. a program with only remote imports) to another machine should also be safe.

I will go ahead and proceed with changing dhall freeze to update all imports, then.