Dhall-lang: Releasing Dhall expressions (was: import from tar)

[ocharles]

Just a random thought that popped into my head, but do we think we want to allow importing tarballs? Right now the easiest way to "release" Dhall work is probably to put it on Github and tag it, but this is a fairly limited distribution channel. A more conventional release process is to release a tarball. I could also imagine these release tarballs containing CBOR-encoded binaries, if this is something that would bring performance benefits.

This could also be significantly faster for things like the Dhall prelude. Right now, importing package.dhall incurs O(n) network requests. We don't pipeline afaik, so there's a lot of overhead. If we could import Prelude-1.0.tar.xz that's a single compressed network transfer, and then it's just local IO (which given the simplicity of the tar format could be just streaming stuff out of the tar - no need to actually create files).

[Gabriel439]

The solution I would suggest is that a client could add the headers Accept: text/plain application/x.dhall+cbor; Accept-Encoding: gzip or something like that and if the server supported it then it would serve the fully normalized binary representation compressed using gzip instead of the source code. That would be a much smaller transfer and only require a single network request

[ocharles]

That's an interesting idea, and would be even faster! I'm not sure if you can attach an arbitrary file like that as a GitHub release, but maybe you can.

[Gabriel439]

@ocharles: Oh, so if this is for serving things from GitHub specifically then I would suggest creating some sort of shared cache for Dhall expressions that an interpreter could optionally use (i.e. the Dhall equivalent of Cachix, for example). Then we would just need a way to instruct the interpreter to use that public shared cache instead of a local cache.

[ocharles]

It's not necessarily about serving from GitHub, that's just one place where people release things. I'm mildly thinking out loud about Dhallix - which is essentially using Dhall as a language for Nix derivations. You might want to share a package repository - which if it's anything like nixpkgs is a lot of files. I'm trying to think about the best way to share that. Of course it would be released frozen, so most users will only pay on the first evaluation of imports, but it'd be nice to avoid even that.

[Gabriel439]

Yeah, then if we would control the server serving the expressions then I would go with using the media type to optionally enable (A) server-side import resolution, (B) binary encoding or (C) both. Those should give most of the speedup we need. Server-side import resolution should be safe due to the fact that remote imports are designed to be referentially transparent.

[ocharles]

I'm no longer sure what I'm asking for here, so I'm going to close this :smile: