Dune: Options to disable implicit vendoring

[favonia]

Desired Behavior

I hope it is possible to disable implicit vendoring so that dune will skip subdirectories with dune-project. The git tool has a convenient new feature git worktree to checkout multiple branches at the same time as subdirectories. However, the dune tool will then search all the subdirectories, including the the ones within the extra git worktrees, and then complain that multiple dune files define the same library. If dune skips the subdirectories with dune-project, then it would have worked.

Example

In a typical dune project with dune-project at the root, I wish the following two steps would work:

git worktree add branch1 branch1
dune build

after I put the following line in dune-project:

(implicit_vendoring false)

[hcarty]

A downside to the requestrd change is that it would break simple vendoring of other repos. It's very easy now to clone or symlink dune libraries under a parent project and have them "just work." If directories with dune-project files were ignored then this very useful dune feature would be lost.

For what it's worth, I use git worktrees as well but with a slightly different organization. All the branches are at the same directory depth. For example:

myproject/branch1/
myproject/branch2/

So new worktrees would be created with git worktree add ../newbranch

[favonia]

@hcarty Thank you. I have updated the description accordingly. I am curious about your usage of git worktrees, though---how did you remove the main working tree?

[nojb]

"Implicit vendoring" is at the very heart of the dune model, so we would need a very good reason to consider changing the way it works.

Forgetting a bit about git worktrees (for the question at hand, it doesn't matter if you are using git worktrees or just plain old git clones), it seems to me the solution is not to have several clones of the same repository within the same tree, but rather clone them somewhere else where they would not interfere with dune. What do you think?

[favonia]

I see. Thank you. How about a new option to disable implicit vendoring? Something like

(implicit_vendoring false)

in the dune-project file. By default, it's true so that dune will keep the current model. When it's false, dune will only look at the directories explicitly listed in vendored_dirs. Do you feel this is acceptable?

[nojb]

I see. Thank you. How about a new option to disable implicit vendoring?

I will let others chime in as well, but personally I still don't see a good justification for having this option: if I understand correctly your problem can be fixed simply by putting your clone/worktree outside your dune tree. Do you agree that this would suffice to solve your problem?

[favonia]

~Unfortunately, no, because you cannot move worktrees outside the repository. There is a fundamental difference between git clones and worktrees---git will add additional worktrees under subdirectories and leave the main working tree at the root, which means you will have a dune project within itself the very moment you start to use worktrees. As far as I understand, one has to either avoid git worktrees altogether or leave the main working tree unused. You would not have this problem with git clones because they are independent of each other and can be placed anywhere, but there are reasons to use git worktrees. That said, @hcarty seems to have a way to work around this, and I am interested in learning it.~ EDIT: the entire reasoning was based on the wrong assumption that worktrees must be under subdirectories and should be discarded.

[hcarty]

how did you remove the main working tree?

@favonia I don't remove the initial clone/working tree. Rather, I make the initial clone with a name that matches the project's default branch and then create worktrees outside of the source tree. Git can create worktrees at arbitrary locations on the filesystem, so they don't need to be nested under the original clone. For example:

$ mkdir dune
$ cd dune
$ git clone https://github.com/ocaml/dune.git primary
$ cd primary
$ git worktree add ../newbranch
$ cd ../newbranch
$ # hack hack hack

Also, if you have a local opam switch that you create under primary then you can share it across worktrees by using opam switch link ../primary from the new worktree directory.

I don't want to take this discussion too far off topic for the dune issue tracker but would be happy to follow up on discuss.ocaml.org if you'd like to go over this further!

[favonia]

Git can create worktrees at arbitrary locations on the filesystem

Somehow I failed to make this work in my previous experiments. Sorry for the bothering and thank you for your help. @hcarty @nojb

[hcarty]

@favonia No problem at all - I updated my comment with a bit more info and the offer stands to follow up on the discuss forum or on the OCaml discord server.