Dune: other file extensions with preprocessor

[IwanKaramazow]

I'm trying to build a project written in Reason with jbuilder.
Basically all my project files have a .re or .rei extension. .re/.rei are the equivalent of .ml/.mli. Compilation happens with a refmt as preprocessor, e.g. ocamlopt -pp "refmt --print binary" -impl hello_world.re -o hello_world.exe". I tried to build some rules for this, but didn't manage to get them working. I'm not entirely sure, after reading the manual, if this is possible. Is it possible?

Example: a project with hello.ml which depends on a function exposed by someReasonFile.re.

(executable
  ((name hello)))
 (rule
     ((targets (someReasonFile.re))
      (action  (system "refmt --print binary ${<}"))))   

I'm getting Unbounded Module SomeReasonFile. What's wrong with this rule?
Can I generalize this rule to all files with .re extensions?

If hello.ml would be written in Reason (needing a preprocessor step), how would the executable look like?

[rgrinberg]

My first advice would be to just write OCaml ;)

My second advice would be to create a rule like this for every reason file:

(rule
 ((targets (file.ml))
  (deps    (file.re))
  (action  (run refmt --print binary ${<}))))

Kind of tedious, but perhaps @diml knows of a way to create rules with % patterns like make. Also, maybe jbuilder should support reason out of the box?

[jeremiedimino]

Currently you can't create rules with patterns. Maybe this could be supported, but there might be some complications, so I think this would go into the v2 of the spec.

Regarding supporting reason out of the box, possibly, depending on how complicated it is. However, I would only give directions and let someone motivated do the work.

@jordwalke is there already some recommended tooling for reason projects?

[jordwalke]

@jordwalke is there already some recommended tooling for reason projects?

@diml Well we're shopping for tooling for the native side - because we want to make an example project that forms the out-of-box experience. We were hoping that jBuilder would be flexible enough to run preprocessors based on file extension (which seems like a pretty common use case). Would you be opposed to adding that? It's pretty simple - ocamlbuild supports it and here are the rules we inject:

In short, we need to pass -pp $REFMT_CMD -intf-suffix .rei -impl file.re when compiling implementation files that end with .re, and -pp $REFMT_CMD -intf file.re when compiling interface files that end with .rei.

[rgrinberg]

That seems fairly simple. @diml I'll give it a try if you give me some pointers.

[jeremiedimino]

OK, let's decide how this will work first. A few questions:

1. how does reason interact with ocamllex/ocamlyacc?
2. is it desirable to have libraries that are a mix of .ml/.re files?
3. does reason support preprocessors/ppx rewriters?

I guess if the answer to (1) is that you are supposed to use the regular ocamllex, then the answer to (2) is yes.

From a configuration point of view, I can see two options:

1. the user has to add (reason) to a jbuild file to switch the directory to reason mode. .ml/.mli files will be ignored. This might make things complicated if you need ocamllex/ocamlyacc as well

2. the user has to add (reason) to a jbuild file to make jbuild recognize .re/.rei files as OCaml modules, in addition to .ml/.mli files

3. jbuilder always accepts .re/.rei file as OCaml modules. This is a breaking change as if someone has a .re file that is not an reason file for some reason, then it will be suddenly considered as an OCaml module

Depending on what the story is for ocamllex/ocamlyacc, I'd be in favor of solution (1) or (2).

Technically, the current rules are setup as follow:

%.ml --ppx/preprocessor--> %.pp.ml

for reason we can simply set them up as follow:

%.re --refmt--> %.pp.ml

and keep the rest unchanged. Additionally, if it makes sense to use ppx with reason, then we could add internal support for chains of predecessors, using temporary files in between:

%.re --refmt--ppx--> %.pp.ml

[jordwalke]

1. how does reason interact with ocamllex/ocamlyacc?

Ideally, just as ocaml would within a project, except for the fact that we don't use reason syntax inside of mll/mly.

1. is it desirable to have libraries that are a mix of .ml/.re files?

It is desirable. We can still try out jBuilder without being able to mix the two.

1. does reason support preprocessors/ppx rewriters?

Yes, it supports all the ppx processors that OCaml supports. It is capable of just generating an AST that is indistinguishable from one parsed by OCaml, except for the fact that the file extension is .re.

for reason we can simply set them up as follow:
%.re --refmt--> %.pp.ml

That will totally suffice. (Technically the command we need to invoke is refmt --print binary, which will print a marshaled AST, which the ocaml compiler always/already recocognizes as a plain text .ml file (such a convenient feature).

[rgrinberg]

I think option 3) should be considered more strongly. While it is a breaking change, it's only technically breaking. jbuilder doesn't have many users, and the .re,.rei extension pairs don't seem to be widely used. So I vote to minimize the hassle for users and make reason work out of the box for new users.

@diml While reason can work as a pure preprocessor (using $ refmt --print fmt), I think that approach will yield poor error messages. Since in this case, the errors would be printed for the post-processed ocaml code. AFAIK reason doesn't output line directives or anything of that sort to translate error messages in the post processed output.

But reason also supports outputting the marshaled parsetree (using $ refmt --print binary). This shold contain proper locations. So if it's possible, this approach should be preferred. Although we can settle for using it like a normal preprocessor for now

[jordwalke]

and the .re,.rei extension pairs don't seem to be widely used

We have .re/.rei reserved as Reason syntax on github, fwiw!

While reason can work as a pure preprocessor (using $ refmt --printer fmt), I think that approach will yield poor error messages

Just wanted to say, I have not confirmed that, but it could be true. If we use refmt --print binary, then the output binary AST does have locations preserved, it's just a matter of if the compiler errors use those exclusively as opposed to the file that that AST was pulled from.

[IwanKaramazow]

To give an example of what I'm doing now:
https://github.com/IwanKaramazow/ReasonNativeProject/blob/jbuilder/src/jbuild
The downside is that I have to add the whole re->ml/rei->mli conversion to every jbuild-config.

[jordwalke]

@IwanKaramazow That doesn't seem so bad!

[jeremiedimino]

Ok, let's go for option 3 and out-of-the box support for reason. I was indeed thinking of using --print binary. Currently, when you use ppx rewriters, the .pp.ml contains the binary AST. Errors are reported properly, so it should be fine for reason as well.

Technically, it should be quite simple:

• change Gen_rules.guess_modules to accept "re" and "rei" in addition to "ml" and "mli"
• adapt Gen_rules.pped_modules to add the refmt --print binary step when appropriate
• update the doc

And hopefully that should be enough

[jordwalke]

I was indeed thinking of using --print binary

Sounds great. I was thinking it could default to refmt --print binary, but could also be customized via env var or some other jBuilder config.

Errors are reported properly, so it should be fine for reason as well.

Fantastic news.

Note, that I do believe it requires passing -intf-suffix .rei to ocaml whenever compiling/ocamldep'ing .re files.

[jeremiedimino]

Well, if we save the output to a file called .pp.ml, we don't need to pass -intf-suffix

[jordwalke]

Ah, but of course.

[jeremiedimino]

Reason support was merged, so closing this issue

[jordwalke]

Thank you very much, @diml and @rgrinberg, I really appreciate your help.