Dune: Library variants poll: keep or drop support?

[jeremiedimino]

About a year ago, Dune got a new experimental feature called "library variants". This feature is an automated selection mechanism for virtual libraries. Because this feature has a high-cost in terms of code complexity and maintenance and also getting the design right turned out to be difficult, we decided to mark this feature as experimental and guard it under a (using library_variants 0.1) stanza in the dune-project file.

We would now like to move forward and either commit to long term support by promoting this feature to the vanilla language, or discard it entirely and simplify the Dune code base, making room for other exciting features.

From our point of view, it seems that this feature is unused. However, we wanted to get feedback from the community before making a decision.

So please react to this message with:

• :+1: if you are using library variants (please also post a link to the use case if public)
• :-1: if you are not using library variants
• :eyes: if you wanted to try library variants but didn't because they were experimental

The default choice will be to drop support for library variants.

Note that this only concerns library variants, not virtual libraries. Virtual libraries are part of the vanilla dune language and there is no plan to get rid of them at this point.

[dinosaure]

Hi,

Currently variants are used mainly by Mirage to be able to provide 2 implementations of libraries:

• checkseum
• digestif

Even if variants are not virtual modules, for some others projects like decompress, ocaml-git or irmin, the ability to choose a default implementation between a C implementation or an OCaml implementation fix a well-know linking error on these projects.

Currently, our fundamental use case is to properly build a project with dune and let it to choose our default implementation - but we want to keep the ability to choose which implementation we want when some people want a JavaScript support of these libraries (and to use the OCaml implementations of them).

[avsm]

I think there are two separate features here:

• support for multiple implementations across a common cmi (the so-called 'linking hack')
• support for link time selection of a variant based on string tags for dependencies

I use library variants in the ctypes dune port to support threaded/unthreaded variants, and it's also handy in checkseum and digestif and other Mirage libraries.

However, I think we can live without the link time selection feature, as users can explicitly select the dependency set in their executables pretty easily.

[samoht]

However, I think we can live without the link time selection feature, as users can explicitly select the dependency set in their executables pretty easily.

It's true in general, but without a default implementation we've seen many users very confused with the resulting error (as it can happen very deep in your dependency tree).

[rgrinberg]

I think we can keep the default implementations without variants. I would definitely prefer to do that

[jeremiedimino]

Keeping the default implementation seems good. Otherwise, turning a normal library into a virtual library cannot be done without a breaking change.

@dinosaure could you show us an example of what users of checkseum and digestif have to write in their dune file when they define an executable, both when using variants and when not using variants?

@avsm, given your message I'm considering that we can safely ignore the ctypes case in the decision process here.

Finally, just a remark: a few people replied that they wanted to try library variants but didn't because they were experimental. Next time, we should think whether we can do better. For instance, in this case we could have proceeded as follow: make the variant field on libraries part of the vanilla syntax and only guard the variants field on executables by (using library-variants 0.1). This way, we could still have easily dropped support for variants and authors of libraries might have been more keen to add variants to their libraries. THe only difference is that we would have had to keep the parser for the variant field on libraries if we decided to drop support, but that's not too bad. Anyway, just food for thoughts for next time.

[dinosaure]

@dinosaure could you show us an example of what users of checkseum and digestif have to write in their dune file when they define an executable, both when using variants and when not using variants?

Currently, due to the default mechanism provided by dune, end-user needs to do nothing to rightly link with our default implementation of these libraries. However, outside the scope of dune, the user must put at the front of any others libraries the right implementation. If the user use ocamlbuild (such as MirageOS), the user must fill a _tags file with:

true: package(digestif.c)

Otherwise, it will have a link-error (reproducible with utop -require digestif and any call of a Digestif function).

[jeremiedimino]

Indeed. That's something that could probably be improved.

So regarding the current topic, would you be sad if we dropped support for library variants? We would keep support for default implementation though. The only thing you would have to do on your side is re-release your package without (use library-variant 0.1) and with a bump of the (lang dune x.y)

[dinosaure]

So regarding the current topic, would you be sad if we dropped support for library variants? We would keep support for default implementation though.

From what I know about variants, the default implementation is the only _feature_ needed by this libraries, so if you keep this behavior, I will happy to give you a chance to delete variants :+1:.

[jeremiedimino]

Perfect, that's settled then. Let's promote default_implementation to the vanilla dune language and drop support for variants.

Thanks to all who participated in this poll/discussion!

[avsm]

Agreed, that would work great for all the usecases I can find. Thanks @jeremiedimino!

[jeremiedimino]

This poll is now closed, I opened #3483 to track the change