Dune: Add an auto-formatter for dune files

[emillon]

As discussed before, an autoformatter for dune files would be great to remove some microdecisions for maintainers.

It seems that Jane Street has some code for this, but it's not released yet and has external dependencies, so it might not be ideal in a first step.

[rgrinberg]

As said in the previous thread, we have enough pretty printed code not to worry adding a little more here to properly format sexps. Thinking about the immediate problems posed here:

• Do we only stick to formatting dune files or also jbuild files?

• We need a parser that doesn't discard comments and preserves string wrapping. Since I assume our formatter will want to preserve those.

Here's another suggestion I have for this command: do not add any configurability of the formatting.

[emillon]

I agree about the non-configurability part.

[hcarty]

As a user, dune-only formatting seems acceptable given that jbuilder has a limited life ahead of it.

[rgrinberg]

Actually, I just remembered that dune might get a parser generator eventually. Maybe it makes sense for it to also include an auto formatter? Avoid this issue entirely.

[jeremiedimino]

I don't know when that will land though. Formatting s-expressions is simpler than the parsing part, we could start with a generic s-expression printer. The only issue if we format s-expressions without knowing the grammar is that we might not produce the best formatting possible.

We could also extract the grammar from the Of_sexp.t if we made it an applicative. However it's really painful to write applicative code with let%map.

[emillon]

I wonder how far we can go with just a "loose list of keywords" rather than a fully fledged parser. A nice side effect is that once we have that list, we can use it for syntax highlighting as well (I wrote this list by hand and it turns out that it's pretty incomplete).

However it's really painful to write applicative code with let%map.

I hope that most of these can be hidden behind Combinators.pair, Combinators.list etc and not have too much code using directly the applicative interface, but that's hard to tell (it's probably a good idea to use these combinators anyway).

[jeremiedimino]

I wonder how far we can go with just a "loose list of keywords" rather than a fully fledged parser. A nice side effect is that once we have that list, we can use it for syntax highlighting as well (I wrote this list by hand and it turns out that it's pretty incomplete).

I suppose we can go quite far. The only risk is that with s-expressions things that are keyword can also be plain data depending on the context. It's also more work to keep everything in sync.

I hope that most of these can be hidden behind Combinators.pair, Combinators.list etc and not have too much code using directly the applicative interface, but that's hard to tell (it's probably a good idea to use these combinators anyway).

The main issue is when you need to bind many values. For instance parsing record would look like:

record (field "a" ... @> field "b" ... @> field "c" @> nil) (fun a b c -> ...)

and you have to mentally bind the various field ... to the variables. For big records it's a huge pain.

I wanted to avoid using too high-techy stuff in Dune such as ppx rewriters, but for writing applicative code it might be justified.

[rgrinberg]

I'd prefer to avoid using ppx as well especially since it solves only one of many of the problems that are caused by our home grown parsing technology. A good parser generator would solve the syncing issues, formatting, and provide a much better basis for editor integration. So I think that resorting to some hacks to have a working formatter is fine for now. Certainly seems better than our own ppx stack in dune as well.

[jeremiedimino]

It's a trade-off. Code generators are always much more painful to work on than direct code. Both a ppx rewriter or a parser/printer generator are code generators, but with let%map you can limit much more the amount of code that goes into the generator itself. A ppx rewriter for let%map is easier to develop and maintain.

Regarding the editor integration, aren't ppx rewriters now relatively well integrated with the editor? If we have a custom parser/printer on the other hand, it will take a custom syntax as input that will have no editor support at all.

The ideal situation would be to add let%map to OCaml, then we would just need a compatibility ppx rewriter.

[rgrinberg]

Regarding the editor integration, aren't ppx rewriters now relatively well integrated with the editor? If we have a custom parser/printer on the other hand, it will take a custom syntax as input that will have no editor support at all.

I meant generating editor integration for the target language. E.g. syntax highlighting files, auto completion databases, etc for the target language. Of course if the parser generator is self hosting, then we'll have both variants editor integration :)

[jeremiedimino]

I see. We can do all that if we make Of_sexp.t be an applicative. It's the same idea as cmdliner.