Dune: Avoid exhausting argument limit for windows cmd.exe

[jordwalke]

For very large build commands, dune can quickly exhaust the command line length supported by windows cmd.exe:

https://support.microsoft.com/en-us/help/830473/command-prompt-cmd-exe-command-line-string-limitation

I believe Dune could/should detect when hitting this limit and then either break it up into several smaller compile commands, or create a batch file then run that.

[rgrinberg]

@dra27 please advise.

[jberdine]

One option would be for dune to use/support @response-file arguments analogously to how ocamlc does. See https://github.com/ocaml/ocaml/pull/778.

[bryphe]

Thanks @jordwalke for logging the issue! Here's the log file where I encountered the error:
log.txt

[bryphe]

I like the idea of a @response-file (a file we can use to pass in the arguments). It looks like ocamloptc supports this via the -args and -arg0 arguments, so dune could leverage this in the case of long commands.

[jeremiedimino]

I like the idea of using a response file too.

[jordwalke]

We're pretty close to enabling "one click Dune + opam" projects on windows, but this is the main blocker. If you could fix this and push a release, I think we'll have something ready to go very quickly.

[jeremiedimino]

I had a quick look at this in this branch. I don't know if this is going to be enough because the compiler itself calls the C compiler to link executables for instance, and it will pass it the include flags as well.

[bryphe]

Awesome, thank you for the help @diml ! I'll test out the branch and report back with the results

[bryphe]

I hit this issue (unrelated) https://github.com/esy/esy/issues/425, but it's blocking my investigation - sorry for the delay.

[bryphe]

Alright, got to a point where we can test out this change - it looks like there is a build failure on Windows when we're trying to build the patched dune:

     "C:\Users\appveyor\.esy\3_\i\ocaml-4.6.5-ee7d6995\bin\ocamlc.opt.exe" -g
     -w -40 -o boot.exe unix.cma boot.ml
         # esy-build-package: running: "./boot.exe" "--release" "-j" "4"
         Multiple rules generated for
     _build/default/bin/.include-flags.response-file:
         - <internal location>
         - <internal location>
         -> required by _build/install/default/bin/dune.exe
         -> required by dune.install
               ocamlc src/.dune.objs/dune__Bin.{cmi,cmti} (exit 2)
         (cd _build/default &&
     C:\Users\appveyor\.esy\3_\i\ocaml-4.6.5-ee7d6995\bin\ocamlc.opt.exe -w
     -40 -g -bin-annot -I src/.dune.objs -args0
     src/.include-flags.response-file -no-alias-deps -open Dune -o
     src/.dune.objs/dune__Bin.cmi -c -intf src/bin.pp.mli)
         File "src/bin.mli", line 3, characters 6-12:
         Error: Unbound module Stdune
         -> required by src/.dune.objs/dune__Bin.cmi
         -> required by src/.dune.objs/dune__Bin.cmx
         -> required by src/dune.a
         -> required by bin/main_jbuilder.exe
         -> required by _build/install/default/bin/jbuilder.exe
         -> required by dune.install
               ocamlc src/.dune.objs/dune__Action_intf.{cmi,cmo,cmt} (exit
     2)

From our appveyor build: https://ci.appveyor.com/project/esy/esy/build/1.0.1123#L188

[jeremiedimino]

Thanks for testing. This looks like an obvious bug in the branch, I'll create a PR so that we can start debugging this issue with AppVeyor.

[jeremiedimino]

I created the PR: https://github.com/ocaml/dune/pull/1215

@bryphe you can give it another try, I fixed this particular issue.

[jeremiedimino]

Actually there is still another issue with the build of dune itself

[jeremiedimino]

I just wrote #1256 which is much simpler

[bryphe]

I did some more testing with the latest master - it's so great to have response file support! 👍

Unfortunately, I was still getting this error:

** Fatal error: Cannot run cygpath -m "libadvapi32" "libadvapi32.lib" "libadvapi32.dll.a" "libadvapi32.a" "C:/Users/bryph/.esy/3_/i/opam__slash__ocamlfind-1.8.0-12c842a0/lib/ocaml\libadvapi32" "C:/Users/bryph/.esy/3_/i/opam__slash__ocamlfind-1.8.0-12c842a0/lib/ocaml\libadvapi32.lib" 

I did some more digging, and found that this was actually the result of a call to flexlink.exe - there's some special handling to address this case here: https://github.com/alainfrisch/flexdll/blob/5f0011a3db1f3a499eeaaae2254e806cbe6b8035/reloc.ml#L71

So it looks like the flow is that dune is calling ocamlopt.exe (correctly!), but then ocamlopt.exe calls flexlink.exe (again, correctly within the limit!) - but then flexlink.exe calls a cygpath -m <lots of paths which surpasses the limit and crashes. The 'fatal exception' message comes from flexlink.

In other words... there's still a bug here blocking our build - but its a bug in flexlink and not dune - so we can close this out. Thanks again @diml for all your help!

[nojb]

@bryphe Could you please open an issue for this in the flexdll repository? Thanks!

[bryphe]

@nojb - definitely! Just logged this issue: https://github.com/alainfrisch/flexdll/issues/67

@jordwalke - we can close this out, as the dune issue has been addressed.