Dune: Use OCaml for configuration files

[bluddy]

Continuing the discussion from here, I'd like to suggest either a switch or additional support for OCaml as a Dune configuration language.

I think there are serious reasons to reconsider s-expressions as the language of choice for OCaml:

1. As stated by @agarwal, it's a very idiosyncratic choice that makes OCaml look 'weird'. This isn't very different from the previous issues we had with ocamlbuild's tag files. It's a choice that could scare away newcomers, which could bring us to weigh this issue again down the line when it's much harder to change.
2. s-expressions are generally hard to parse. They become messy when parentheses build up, and the fact that parentheses and spaces are the only syntactic elements makes it hard to interpret them.
3. s-expressions are not a well accepted convention in the industry with regard to tooling.
4. There appears to be an aim of adding language-like functionality to s-exp, which essentially means we're rebuilding lisp.

Particularly because of item 4, I think it makes more sense to consider OCaml as a replacement. OCaml is clearer due to more syntactic elements; the cost of learning dune syntax is already amortized into learning the language itself; and the tooling is already present.

From what I understand, dune already has rudimentary support for OCaml as a way to access more advanced functionality. All that would be required would be to make a simpler interface. It would also lead the way towards getting rid of all shellisms, including both actual calls to the shell and the esoteric conventions carried over from makefiles to refer to targets and such.

As an example, translating something out of the dune unit tests:

(alias
 ((name runtest)
  (deps (tests.mlt
         (glob_files ${SCOPE_ROOT}/src/.dune.objs/*.cmi)
         (files_recursively_in findlib-db)))
  (action (chdir ${SCOPE_ROOT}
           (progn
            (run ${exe:expect_test.exe} ${<})
            (diff? ${<} ${<}.corrected))))))

would roughly become

let config env = [
  Alias {
    name="runtest";
    deps=[
      Dep "test.mlt";
      Glob_files (env.scope_root ^ "/src/.dune.objs/*.cmi");
      Files_recurse "findlib-db"
    ]
  };
  Action(Chdir {
    dir=env.scope_root;
    prog= [
        Run (fun state -> cmd ["expect_test.exe"; state.filename]);
        Diff (fun state -> (state.filename, state.filename ^ ".corrected")
    ]
  });
]

This already looks to me much clearer than the s-expression version. It could become even better with a judicious use of functions with optional arguments to create the AST.

[lpw25]

additional support for OCaml as a Dune configuration language

On this front I should probably point to #56, which I think is the current long term plan for supporting using OCaml for specifying builds, as well as #414 which is the proposed OCaml API for this.

The idea in that plan is that you write interpretations of dune sexps as OCaml code. If you just wanted to use OCaml for the whole spec you would have a dune file like:

(my_build_rules)

and a dune project file including something like:

(plugin (my_build_rules 1.0))

and then you can specify your build rules as a plugin that interprets (my_build_rules).

Obviously this is designed to handle the more general case where you use OCaml to only specify parts of a build, or use different bits of OCaml for different parts of the build. Perhaps some additional hook to avoid needing to have a somewhat pointless dune file would be useful for the use case where you just want to use OCaml for everything. The lang construct in dune-project files is somewhat intended to allow this kind of forwards compatibility -- although I had previously envisaged it only supporting changes to how all the sexps were handled rather than a complete change of format (Akin to macros in Racket rather than to reader extensions).

[lpw25]

My personal experience with using OCaml to specify builds in Jenga is mixed. I like the idea in principle, and sometimes you really want the expressivity it provides, but you quickly find yourself with a bootstrapping issue.

Essentially, when you use OCaml as a build language you really want to all of OCaml as a build language -- in particular you especially want to have support for libraries. But then you have a problem -- you need to specify how to build the OCaml that you use to specify how to build the rest of your OCaml. This tends to mean having a simple bit of OCaml that uses no libraries that then bootstraps the rest of the OCaml, and it just ends up feeling like hard work.

So my current thinking is that it is nice to have a full-featured build language that knows how to build OCaml -- including linking in libraries -- and then you can use that to build your OCaml build specification. Since many simple packages can get by on just this build language, it might as well be a decent build langauge that people can use by default in the simple cases: i.e. something like the current jbuild language.

[bluddy]

I think the difference is that we'd still be aiming for simplicity here, as opposed to Jenga where you're able to do anything.

My thoughts on this were that the ocaml layer's purpose would only be to create the config object, just like the s-exp does. And we could curate a very limited set of libraries that would be baked in. The s-exp only has certain functionality, so there's nothing wrong with doing the same in a limited ocaml domain. In essence we'd be limiting the EDSL using the libraries provided. For example, we could include parts of BOS, which would allow doing all shell operations in ocaml instead. We'd have an escape hatch for calling the shell directly, but that would be frowned upon and maybe even eventually deprecated down the line.

In addition to this simple mechanism, we could also have a more extensive mechanism that would compile with other things, as currently envisioned for the plugins.

[jeremiedimino]

This PR would require a substantial of work and there is already quite a lot to do reach 1.0.0. @bluddy if you are interested in contributing to the project, I'm happy to point you to simple things you can do to get started, but you can't start by asking us to change everything.

[lpw25]

presumably we can add shell-like operations as needed

Yeah, you can. Actually I'm going to delete my comment (and then later this one) as it is clearly off-topic.

[lpw25]

I think the lang construct in dune project files makes us forward compatible with this change anyway, and should even allow it to be implemented outside of core dune, so I don't think there is anything that needs to be done right now anyway.

[agarwal]

See also #124, which also discussed use of a special syntax vs OCaml.

[jeremiedimino]

The official answer to the question of using OCaml as the configuration language is the following: let's see how good it is first. If one of you is interested in running the experiment, I'm happy to point to the place in the code you need to modify to make it happen, getting started is actually very simple. However, I expect that there will be several issues to solve before it could be viable.

BTW, I'm not a teacher and I don't know how hard it is for beginners to learn S-expressions, but I'm pretty sure that the example posted here would scare away any beginner. For instance, I have absolutely no clue why you wrote this config function or what this state variable is. These are just noisy bits that add no value and shouldn't have to be present in every single configuration file.

[Chris00]

If we go that route, may I suggest that one uses functions with optional arguments to specify the build rules? For example, the above example (correctly interpreted from the Dune file) would be:

open Dune.V1

let conf env =
  alias
    ~name:"runtest"
    ~deps:[
      File "test.mlt";
      Glob_files (env.scope_root ^ "/src/.dune.objs/*.cmi");
      Files_recurse "findlib-db" ]
    ~action:(fun state ->
      chdir ~dir:env.scope_root
        [run ["expect_test.exe"; state.deps.(0)];
         diff state.deps.(0) (state.deps.(0) ^ ".corrected")])

[yminsky]

One thing that one should be very careful about when using OCaml as a configuration language is thinking about managing versions. One of the nice things about having the code in a specialized format is that managing multiple versions in a single codebase and having upgrade/downgrade logic is all relatively easy.

Using the full OCaml language as your config format makes all of this harder. At Jane Street, after an initial burst of enthusiasm, we discovered a lot of tricky corners around versioning OCaml based configs. All of this makes me think that the current state, where there's a pure-data file with some very limited DSLs which is what the vast majority of user configs are written in, is a pretty good spot in the design space.

Eventually I expect Dune will need to have a more open API where plugins can be written in OCaml to add various kinds of build rules. But my conjecture is that these should be few and far between.

[rgrinberg]

Another possibility which here that is borderline off topic, but I can't resist to bring is the camlon configuration language:

Alias {
  name = "runtest";
  deps =
    [ File "tests.mlt"
    ; Glob "${SCOPE_ROOT}/src/.dune.objs/*.cmi"
    ; Files_recursively_in "findlib-db"
    ];
  action =
    Chdir (
      "${SCOPE_ROOT}",
      Progn [Run "${exe:expect_test.exe} ${<}"; Diff ("${<}", "${<}.corrected")]
    )
}

My suggestion isn't really serious of course. But if the goal is familiarity for OCaml programmers, then I think the above cannot be beat by json, s-exps, or yaml.
Of course, none of these alternative formats really justify the amount of work that would be required to transition into them, so I'm not suggesting anything.

[bluddy]

One thing that one should be very careful about when using OCaml as a configuration language is thinking about managing versions.

Clearly one of the issues in using OCaml here is figuring out the allowed scope for a basic configuration, as raised by @lpw25 above. It's not my goal to allow any module to be imported in the basic configuration files, precisely because I realize how problematic this is in terms of controlling the universe of possible actions in a dune file. I think that if we stick to functions that create an AST and other functions that live in the Build monad and manipulate files, we'll be able to get the familiarity of OCaml syntax without the kinds of side effects that make it impossible to do versioning. I don't think this will satisfy some of the wishes @agarwal raised in #124, for example.

In essence, we'll be trading the unfamiliarity of a new syntax for that of learning some OCaml libraries (as needed), which I still think is a good trade-off. What we get for free is clearly defined language semantics, and a syntax that's familiar and clear.

For instance, I have absolutely no clue why you wrote this config function or what this state variable is.

Right -- I didn't specify any required format. My thinking was that everything needs to be wrapped up in a config function. I don't want the configuration file to be the driver -- you're not creating the main function of an executable, but rather, providing a module for a driver to call. The state and env arguments are there to get rid of bash-isms like "${SCOPE_ROOT}". It makes more sense to have them passed in as arguments carried in record hierarchies, and the types of these arguments can be explored to find every available variable. These are things that I think will quickly make sense, even to beginners.

If one of you is interested in running the experiment, I'm happy to point to the place in the code you need to modify to make it happen, getting started is actually very simple.

I'm in the midst of an internship, but I'd love to use my spare cycles to try this. I've skimmed through the code, but any pointers would be appreciated.

[jeremiedimino]

It might not be obvious if you don't know the internals of Dune, but the S-expression syntax hides a ton of complexity. Exposing the Build arrow as the default way of describing projects is the complete opposite of making things simpler for users. Learning Dune would become a lot harder than learning S-expressions.

I'm in the midst of an internship, but I'd love to use my spare cycles to try this. I've skimmed through the code, but any pointers would be appreciated.

You could start with the OCaml syntax. It is implemented by generating a .ml file that includes some code at the beginning that implements the plugin/jbuild_plugin.mli API followed by the contents of the jbuild file. This code is then evaluated using the toplevel. This is done in src/jbuild_load.ml.

The first step is to change this to instead build a .cmxs and then dynlink it. This should be relatively easy and will allow you to create values of type Jbuild.Stanza.t directly. You won't be able to get rid of bash-isms as you call them, however it will be enough to expose the first serious issue to solve: all error messages are going to become absolutely terrible.

[bluddy]

Exposing the Build arrow as the default way of describing projects is the complete opposite of making things simpler for users. Learning Dune would become a lot harder than learning S-expressions.

Right -- I did see this when browsing through the code, but forgot about it. My intent is to have a similar layer to the s-expressions ie. a simplified interface, rather than one as powerful as the Build arrow.

Thanks for the tips! I'll dig in when I get a chance.

[agarwal]

I don't think this will satisfy some of the wishes @agarwal raised in #124, for example.

I should mention that it's been nearly a year since that issue. Back then I felt avoiding the full power of OCaml would be impossible, given some of the things I was doing in my build scripts. However, that turned out to be not true. We've been using dune's s-expression syntax for nearly a year, and we're building complicated projects just fine.

[jeremiedimino]

I have thought more about this question and there are two compromises that Dune will not make: syntax and error messages. For syntax, the criterion is simple. If we want to declare a library named mylib, then the three words we should write are: library name mylib. This rules out the record syntax. For error messages, it's simple as well: they should be easy to understand for anyone, including non-OCaml programmers. This is because Dune is already used for other languages such as Reason and I don't want to close the door to supporting others in the future. As a result the application syntax is not acceptable either.

One way to solve this issue would be to have default values for record fields. I discussed this idea with other OCaml developpers and unfortunately it appears that this is a complicated feature and the compiler is currently not ready to support it, so it's at least a few years away.

In summary, there would be too many compromises to do to switch to OCaml, so I'm saying no to this change. There is no point wasting time on it because it wouldn't be accepted.

To close the debate on other languages such as Json as well, we have a few EDSLs in Dune, and S-expressions offer the perfect support for them. Additionally, the learning curve for S-expressions is very small and I believe Dune is a good enough software to justify it. So we won't be switching to Json either.