Dune: jbuilder doesn't support menhir with --infer

[vsiles]

Hi ! I'm trying to move from ocamlbuild to jbuilder, but I can't invoke menhir with the --infer flag:

$ jbuilder build foobar.exe
      menhir parser.{ml,mli} (exit 1)
(cd _build/default && .../menhir --infer --explain parser.mly)
File "parser.mly", line 3, characters 13-19
Error: Unbound module Syntax

The following configuration

```(jbuild_version 1)
(menhir
((flags (--explain))
(modules (parser))))

(ocamllex (lexer))

(executable
((name foobar)
(libraries (core))))

works but 

```(jbuild_version 1)
(menhir
 ((flags (--infer --explain))
  (modules (parser))))

(ocamllex (lexer))
(executable
    ((name foobar)
     (libraries (core))))

doesn't.

The issue is described in menhir manual: by using --infer, some calls to ocamlc are done to typecheck the file, so some modules have to be compiled before the parser. It is advised to use --depend to get the correct dependencies.

For the record, my parser.mly starts with:

%{
  open Lexing
  module S = Syntax
%}

and I use S.Fobaar in the mly file (see here)
I'm using jbuilder 1.0+beta14 on ubuntu 16.04.3

[jeremiedimino]

We need to change a few things in jbuilder to support that. But before we look into this, I'd just like to understand better why menhir is calling the compiler. IIUC, we get better error locations when using the --infer flag, but for instance we get the right locations as well when using ocamlyacc or a ppx rewriter without calling the compiler. /cc @fpottier

[fpottier]

Hello! The initial motivation for menhir --infer was indeed to typecheck the semantic actions in isolation, so as to prevent the OCaml typechecker from reporting type errors at random places in the generated code. Then, later on, the so-called inspection API was added. When --inspection is turned on, Menhir generates additional code, which (among other things) includes a GADT of all nonterminal symbols indexed with their OCaml type. This requires the OCaml type of every symbol to be known to Menhir. The simplest way to achieve this is to use --infer, but it is also possible to manually annotate every symbol with an explicit %type declaration.

The reason why Menhir directly invokes ocamlc is that this was deemed simpler. One could conceivably instead imagine a scheme where Menhir is invoked in two stages; the first stage dumps an .ml file; then ocamlc -i is invoked (by the build system); then Menhir is invoked again and reads the types inferred by the OCaml compiler. This was suggested already by Fabrice Le Fessant but has not been implemented, by lack of demand. If there are strong technical arguments why that would be a good idea, I could look into it.

[jeremiedimino]

Thanks for the explanations! I understand better why we need this now. We'll add this to the TODO list.

The reason why Menhir directly invokes ocamlc is that this was deemed simpler. One could conceivably instead imagine a scheme where Menhir is invoked in two stages; the first stage dumps an .ml file; then ocamlc -i is invoked (by the build system); then Menhir is invoked again and reads the types inferred by the OCaml compiler. This was suggested already by Fabrice Le Fessant but has not been implemented, by lack of demand. If there are strong technical arguments why that would be a good idea, I could look into it.

The only advantage I can think of is that we would avoid quoting issues since we wouldn't have to construct a shell command for --ocamlc/--ocamldep. Apart from that it's the same complexity from the point of view of jbuilder, so I don't think it's worth changing menhir.

[vsiles]

Today, my opam updated to jbuilder.1.0+beta18 where the merge is located (if I'm correct), so I tried to see if the issue was addressed. My project is located here, and here is a copy of my jbuild file:

(menhir
 ((flags (--explain --infer))
  (modules (parser))))

(ocamllex (lexer))


(executable
    ((name tigercc)
     (modules_without_implementation (frame)) 
     (libraries (core))))

Upon executing jbuilder build tigercc.exe in the src directory, I get:

      menhir parser.{ml,mli} (exit 1)
(cd _build/default && /home/vsiles/.opam/system/bin/menhir --explain --infer parser.mly)

Did I forgot something to trigger the dependency computation ?

[jeremiedimino]

--infer is not yet supported. We only did some preliminary changes in order to be able to support it

[vsiles]

Ok ! Great to here you're working on it :)

[rgrinberg]

This is now in master and will be available in 1.4.0 with (using menhir 2.0). Please open new tickets if there are issues with this feature.