Dune: [RFC] allow to build extended dune binaries

[jeremiedimino]

In order to be able to use Dune inside Jane Street, we are going to need a customised version of Dune with additional Jane Street specific features. I'm thinking that if we do that, we might as well allow everybody to do it.

The goal is not to create a plugin system, but rather to allow one to build an extended version of Dune with additional private functionalities for their own personal needs. However, in order to maximise the reach of one's software package, it would be recommended that the said package can build without the private functionalities. In particular, requiring private functionalities would break the compositional aspect of Dune and would expose the package to regular breakages as we would provide no backward compatibility guarantee on the internal Dune libraries. It would however be expected that over time, better and more stable APIs would emerge, and we could potentially promote such APIs as stable ones, but not until we are absolutely happy with such APIs.

For instance, one application of this feature would be to implement custom testing frameworks, or building data that one is not interested in installing as part of the package.

How would it work?

We would turn the bin/ directory into a library and an executable that simply calls Dune_bin.main (). This new library would be public, so one would be able to link their own dune binary outside of the Dune project.

They could then use mechanisms such as the library sub-system mechanism or new ones we will develop to extend Dune.

Naming convention

Because one would need access to the internal libraries of Dune, we would need to publicise them. However, their naming should clearly reflect the fact that they are unstable. To that end, I suggest the following naming:

• dune.unstable.build
• dune.unstable.bin
• dune.unstable.stdune
• ...

Extending the language

Extensions should always require a (using <ext> <version>) field in the dune-project file, so that one can quickly understand what language is used inside dune files. All extension to the language, such as the addition of new stanza or fields should be named <ext>.<something>, so that we know what comes from the vanilla dune language and what comes from extensions.

Building without the extension

We would add a (using? <ext> <version>) form in dune-project fields. This would mean that when the extension is not available because the dune binary wasn't linked with it, Dune would ignore all the (<ext>.<something> ...) fields and stanzas.

Alternatively, we would also provide a command to strip all these fields and stanzas from the project, so that one could keep such things private.

[jeremiedimino]

Another way we could provide this feature is to make it easy to maintain a fork of dune. For instance, dune could try to link a dune-extensions library into the dune binary if it exists. Then one would just add to add a directory with the extension and rebasing would be easy.

[jeremiedimino]

Actually, this forking idea seems better to me: we'll be sure new releases of dune never break anything. It will be up to users who extend dune to rebase their fork

[avsm]

I'm wondering about some external use for this functionality. For example, we could build ctypes rules using this as a test, or perhaps even the Mirage CLI frontend could be a dune fork with extended commands to invoke functoria/etc. @TheLortex

[jeremiedimino]

The mirage case is compelling indeed :) Regarding ctypes, we should just put them in Dune proper since this is of general interest.

[avsm]

Been thinking about this more, and the idea of maintaining a fork of dune with the dune-extensions library providing the extension hooks is pretty compelling. It would let us put mirage-specific configuration into mirage.foo fields to map onto functoria. So happy to prototype this feature to ensure there is a public consumer for it other than Jane Street's internal use.

[jeremiedimino]

Thanks. So setting that up is pretty easy: we just need to add this to the libraries field in bin/main/dune:

(select extensions.ml from
 (dune-extensions -> extensions_real.ml)
 ( -> extensions_fake.ml))

where:

• extensions_real.ml contains: Dune_extensions.init ()
• extensions_fake.ml is empty

Then you can create an extended dune by forking dune and adding a new directory containing this dune-extensions library. The rest of the game is to add hooks at various places in dune to do what you want. There is already a sub-system mechanism for libraries which might already be enough for mirage.

[jeremiedimino]

BTW, @jordwalke you might be interested in this as well. It would allow you to easily maintain a fork of dune that would read json files.

[jordwalke]

Sounds interesting! Until now, I have been thinking more about building some pretty opinionated workflows around using Dune, and I think the easiest ways to support that would be to have Dune allow another pre-build stage to generate the dune files out of source without having to have any/many dune files in the actual repo itself. The reason I like that approach is because then these same tools can have an "eject" feature that generates the dune files in source for the sake of publishing. That way the projects could easily work without requiring a forked dune (monorepos, opam, etc). But if Dune was easier to maintain as a fork, then one could easily build that out of source dune file feature in the fork. On the other hand, if many possible forks could simply be avoided by allowing out-of-source dune files as a first class feature, it might be nice to just have dune support an out-of-source dune generation step.

[jeremiedimino]

I have a feeling that you'd get a better development experience out of an extended dune that can directly read the configuration files you want your users to read. In particular, using ejected dune files wouldn't give proper locations for error messages and editing them wouldn't work well (since they'd be generated).

[jordwalke]

@diml Are you suggesting that using an extended dune would allow us to intercept Dune error messages to redirect locations to the input config?

[jeremiedimino]

Even better: as long as your json parser keeps location, dune errors will point directly to the json files :)

[jordwalke]

Sounds pretty cool! Is there a sort of "source maps" support for Dune inputs currently?

[jeremiedimino]

I know the name source map but I don't know what it's about. Is a mapping between locations in two different files? Unless you'd want to generate dune files and commit them, that shouldn't be necessary.

[jordwalke]

Correct, that's what source maps means (informally). When transforming one AST into another AST, they are an extra data structure used to map back locations to the original source. I suppose if parsing json directly into dune AST, we would just copy over the locations from the json file into the dune AST and everything should work. No need for "source maps".

[rgrinberg]

Given that Jeremie wrote up a proposal to extract dune's build engine as an API, this proposal can be safely archived.