Dune: Library dependencies in aliases

[Julow]

In ocaml-mdx we currently use findlib to load libraries and we are having trouble to get it to work with dune.

(alias
 (name   runtest)
 (deps   (:x README.md))
 (action (progn
           (run ocaml-mdx test --direction=infer-timestamp %{x})
           (diff? %{x} %{x}.corrected)
)))

In this example, we want ocaml-mdx to load libraries required by README.md which can be external libraries or libraries in the current dune context.
I did not find any way to declare library dependencies in an alias.

Is there a way to do that in dune that I did not found ?

Otherwise, I can think of 2 solutions, that I'll be happy to implement:

1. Library deps: (alias (deps (library lib1) ...) ...)
   Dune will generate the META files so findlib can find them.
   Mdx need to properly configure findlib to search into _build/context, which is not possible at the moment but we can patch findlib.

2. Library paths: (action (env MDX_PATH "%{path:internal_lib}:%{path:external_lib}" (progn ...)))
   Dune output the same paths that would be given to ocamlopt -I.
   Mdx no longer need findlib. We can remove #require ... directives before executing using stock toplevel.

[bobot]

Library deps: (alias (deps (library lib1) ...) ...)

It could be an extension of #2417, %{dynlink:lib1} . Which would be expanded to all the files needed for dynlinking a library.

Mdx need to properly configure findlib to search into _build/context, which is not possible at the moment but we can patch findlib.

dune add the path to OCAMLPATH for public libraries so it should work. Do you consider using findlib.dynload and its support in dune?

[rgrinberg]

I would certainly lean towards a solution that doesn't require findlib at all. For 2., could you clarify what kind -I are you looking for:

1) The include path just for the library
2) The include path of the library and dependencies (used for compilation)
3) The include path of the library and transitive dependencies (used for linking)

[Julow]

I think we need the include path of the library and transitive dependencies. We need to execute the code in a toplevel.

@bobot We didn't try dynload because we are using a toplevel. I can give it a try. How will dune known which libraries to build and add to OCAMLPATH ?

[bobot]

Only public libraries are accessible through dynload, and they are by default built.

[NathanReb]

One thing I think is worth mentioning is that we'd like this to work with duniverse for dependency management. That means that most dependencies will be vendored and that nothing there is built by default, unless explicitly depended upon by dune rules in the main project.

That means that mdx will have to make the dependencies on libraries explicit in the dune rules.

Our current workaround involves:

1. Parsing code blocks in the .md file for #require statements and add (package ...) dependencies to the generated test alias for each package we find there.
2. Generate a findlib.conf with path pointing to %{project_root/_build/install/%{context_name}/lib
3. Add a (:conf findlib.conf) dependcy to the test alias
4. Wrap the test alias action in a (setenv OCAMLFIND_ROOT %{conf} (...))

This works but feels pretty hacky and fragile. It also makes it impossible to work simultaneously with opam and duniverse.

[bobot]

I don't understand why the step 1. is not sufficient. If the package is external, in opam, ocamlfind will find it. If it is local, dune will compile and copy it, and it will append to OCAMLPATH, %{project_root}/_build/install/%{context_name}/lib . I'm sorry, it seems that there is something I don't get. Why findlib is not able to find your packages in this case? Could you try to run ocamlfind query foo in your test?

[bobot]

By looking at realworldocaml/mdx#130, I have the impression that (deps (package ..) working for external library would solve one part of your problem. Is it true?

[Julow]

I think (deps (package ...)) working on external packages would solve our problem. But a (deps (library ...)) that accept private libraries or avoid building the entire package would be better.
Maybe we went too far with our findlib hacks and forgot how it really works.

[NathanReb]

The thing here is that we're also vendoring findlib and it might be mis-configured in our dune port. One issue we had is that it requires a findlib.conf no matter what since it self initialize before your program even starts thus preventing us from programmatically initializing it with the right parameters.

Having spent some time working with it in the past few days, I believe it would be nice to be able to rely on dune to figure out where to find libraries instead. I don't think having that repeated extra layer is doing any good especially since I believe dune already has a replacement for it internally.

Also note that because we have to rely on findlib, we have to depend on packages rather than on libraries which seems sub-optimal and prevents us from using private libraries for instance.

As you noticed, step 1 currently makes it impossible to rely on opam packages. Fixing this would be nice because it would make our current solution work with both duniverse and opam but the findlib issues would remain.

Thanks for your suggestions, we'll try to give it a go in the next few days!

[bobot]

One issue we had is that it requires a findlib.conf no matter what since it self initialize before your program even starts thus preventing us from programmatically initializing it with the right parameters.

Interesting, I never played with the toplevel and findlib with dynlink+dune. Usually I just use it for creating a plugin system in my tools, in that case I can run Findlib.init() or Findlib.init_manually() with all the parameters I need.

Having spent some time working with it in the past few days, I believe it would be nice to be able to rely on dune to figure out where to find libraries instead. I don't think having that repeated extra layer is doing any good especially since I believe dune already has a replacement for it internally.

Replacing findlib.dynload by dune.dynload simplifies some part, but I'm think Dune still doesn't support distribution like Debian without Findlib well enough.

[rgrinberg]

that accept private libraries or avoid building the entire package would be better.
Maybe we went too far with our findlib hacks and forgot how it really works.

I think this is basically why (package ..) can never be enough for mdx.

[rgrinberg]

I think we need the include path of the library and transitive dependencies. We need to execute the code in a toplevel.

Yes, but you really don't want to be using implicit transitive deps in your tests. Otherwise you'll run into the same problems they cause elsewhere.

[jeremiedimino]

IIUC, for any of this to work with private libraries, we would need to generate META files for private libraries?

[Julow]

IIUC, for any of this to work with private libraries, we would need to generate META files for private libraries?

Yes. The alternative would be to not to use findlib and pass the include paths (env variable or ${} substitution), like dune utop does.

[rgrinberg]

Not using findlib is absolutely preferred. We just need to clarify the issue with compilation vs linking include flags :)

[jeremiedimino]

I'm wondering if findlib itself couldn't be modified to cooperate well with Dune in such case. I'm not against exploring alternative solutions, but findlib has existed and served the OCaml community for many years before Dune even existed. It feels like we should at least have a chat with its author before doing our own thing (mostly thinking about dune.dynload here).

[rgrinberg]

I understand the sentiment but I think that our first priority should be hammering out all use cases without findlib, opam, anything else. To get a polished tool, we need something that we can iterate on quickly without co-ordinating with many different projects. Just as importantly, we usually want something that is at least functional for users out of the box.

So I'm not against co-operation, but I'd much rather prioritize user needs over general community needs. Of course, there's no harm in pursuing both goals.

[jeremiedimino]

That seems about right, and I guess we can split goals here.

[NathanReb]

Is there any update on this?

A quick update on our side: We cleaned up a bit the hack we needed to get RWO tests working with a duniverse. We have cleaned up the findlib dune port at https://github.com/dune-universe/lib-findlib/tree/duniverse-1.8.1 so it doesn't need a dummy findlib.conf anymore. All in all the local findlib cooperates with dune just fine!

We still have issues with dependencies in the mdx + duniverse world that require some hacks though:

1. In mdx we have a binary that depend on another binary and there's currently no way to express that in dune. That's an issue in the opam world as when you install mdx, you install both binaries and all goes well. When you try to run a local mdx with dune (e.g. in a duniverse or in the mdx repo) it is hard to express mdx rules properly. For instance in mdx end-to-end tests we have no way to make sure both binaries are built before running the tests, even using (alias install) dependencies leaves us with a race condition.
2. As @Julow mentioned, we have to give duniverse a special treatment by adding (package ...) dependencies based on the library you require in toplevel code blocks that you want mdx to execute. The same can be true for some executables some time. Given we can't add library or executable dependencies to aliases or rules, we have to give opam and duniverse different treatments, which isn't the case when you build a library for instance. In the case of libraries, dune knows whether it can build it locally or has to look for it with findlib/opam. Pretty much the same goes for executables.

I have a feeling that all of this is related and I kind of expect that the current work that is done on https://github.com/ocaml/ocaml-library-standard should help fixing this but I'd love to hear your opinion on that.

I'd also be delighted to contribute a change, say to add (library ...) and (executable ...) to the dependency specification language if you feel like that'd be a worthy addition!

[jeremiedimino]

There has been quite a few requests about mdx and it seems to me that mdx is a good tool to popularise. So let's get straight to the point and add a mdx stanza. Could one of you write a RFC for such a stanza? Once we agree on the high-level design, we can discuss the implementation.

[NathanReb]

That could work but we need to parse the markdown files to figure out the dependencies as the current dune + mdx workflow is the following:

1. ocaml-mdx rule file.md reads through file.md and output a runtest alias stanza with the right dependencies, targets, actions and promotions.
2. dune runtest executes those, usually an ocaml-mdx test ... invocation followed by a bunch of diff

[NathanReb]

We could replace the alias generated by 1 by an mdx stanza. That means you'd still need a rule to generate the (mdx ...) stanza. How would that sound?

There might be another way around that that I'm not thinking of. I also guess the action plugins might bring some novelty to that workflow, is it in a state that would allow me to experiment with it?

[jeremiedimino]

There is not enough stuff yet to do this via the action plugin. Dune allows dynamic dependencies, so there is need to generate the mdx stanza. Basically the user would write something like this:

(mdx (files *.md))

and Dune would do the rest.

[NathanReb]

Hmm interesting, I'll work on that RFC then! It might take some time as I'll be off for a couple weeks staring next tuesday though!

[rgrinberg]

Seems like the mdx stanza addressed all of these concerns. @Julow please create new issues if there are any problems.