Dune: Assert failure upgrading Revery to [email protected]

[bryphe]

Hi,

We were stuck on [email protected] for a while on our project revery, so we're trying to get to the latest dune (thanks @Et7f3 for all the help)!

We're running into this error after upgrading:

File "src/sub_dirs.ml", line 48, characters 23-29: Assertion failed
Backtrace:
Raised at file "src/sub_dirs.ml", line 48, characters 23-35
Called from file "src/file_tree.ml", line 310, characters 14-46
Called from file "list.ml", line 117, characters 24-34
Called from file "src/file_tree.ml", line 304, characters 8-1023
Called from file "camlinternalLazy.ml", line 27, characters 17-27
Re-raised at file "camlinternalLazy.ml", line 34, characters 4-11
Called from file "src/file_tree.ml" (inlined), line 121, characters 20-32
Called from file "src/file_tree.ml", line 143, characters 22-34
Called from file "src/dune_load.ml", line 263, characters 4-319
Called from file "src/main.ml", line 43, characters 4-35
Called from file "bin/import.ml", line 49, characters 4-30
Called from file "bin/main.ml", line 7, characters 17-39
Called from file "src/fiber/fiber.ml", line 112, characters 7-12

The assert seems to be here https://github.com/ocaml/dune/blob/10a9ea4969732bc4839a89f33dc9fcf42dd4c525/src/dune/sub_dirs.ml#L48

This is reproducing on all platforms we test on (Windows, Linux, Mac)

And we're working on this PR:
https://github.com/revery-ui/revery/pull/546

I imagine we must have something strange or non-standard in our directory layout that is causing problems - any pointers on where we can look?

Thank you!

[bryphe]

A little more info:

• Our build command is: "dune build -p Revery -j4" (https://github.com/revery-ui/revery/blob/7903bbb9ec34b9bd6552c775bbc71025bb8968a7/package.json#L20)

Running with --verbose just gives:

# Workspace root: "D:\\revery"
File "src/sub_dirs.ml", line 48, characters 23-29: Assertion failed
Backtrace:
Raised at file "src/sub_dirs.ml", line 48, characters 23-35
Called from file "src/file_tree.ml", line 310, characters 14-46
Called from file "list.ml", line 117, characters 24-34
Called from file "src/file_tree.ml", line 304, characters 8-1023
Called from file "camlinternalLazy.ml", line 27, characters 17-27
Re-raised at file "camlinternalLazy.ml", line 34, characters 4-11
Called from file "src/file_tree.ml" (inlined), line 121, characters 20-32
Called from file "src/file_tree.ml", line 143, characters 22-34
Called from file "src/dune_load.ml", line 263, characters 4-319
Called from file "src/main.ml", line 43, characters 4-35
Called from file "bin/import.ml", line 49, characters 4-30
Called from file "bin/main.ml", line 7, characters 17-39
Called from file "src/fiber/fiber.ml", line 112, characters 7-12

I must not segfault.  Uncertainty is the mind-killer.  Exceptions are
the little-death that brings total obliteration.  I will fully express
my cases.  Execution will pass over me and through me.  And when it
has gone past, I will unwind the stack along its path.  Where the
cases are handled there will be nothing.  Only I will remain.
error: command failed: "dune" "build" "--verbose" "-p" "Revery" "-j4" (exited with 1)
esy-build-package: exiting with errors above...
error: build failed with exit code: 1

esy: exiting due to errors above

[bryphe]

Seems like, if I remove the ignored_subdirs from the root - it builds successfully. Must be related to the deprecation of ignored_subdirs -> data_only_dirs.

[bryphe]

I'll close this out since we can resolve this with ignored_subdirs. I'll explore the new data_only_dirs/vendored_dirs options to see if we can speed up our build with those 👍

[jordwalke]

Multiple other people were hitting this, myself included and it was very difficult to know what was going on. Is it worth reopening this issue so that someone could perhaps add a more helpful error message? My understanding was that Dune would like to never completely crash without a descriptive error message.

[jordwalke]

In my case, I had (data_only_dirs node_modules _esy _release) in my dune file, and when I removed it, it stopped crashing - so it seemed to be the culprit. But isn't data_only_dirs a supported feature?

[rgrinberg]

@jordwalke. I'll take a look. Is there an easy to reproduce example somewhere?

[aalekseyev]

It looks that to trigger this bug it's enough to have a dirs set that's not a superset of data_only_dirs:

cat >dune <<EOF
(dirs foo)
(data_only_dirs bar)
EOF

mkdir bar
touch bar/x

[rgrinberg]

This is now fixed in master. Shall I backport this to 1.11?

[Et7f3]

So to avoid this bug in the future, should we update our dependencies with dune > 1.11.2 or ^1.7.3 is fine.

[rgrinberg]

There is no released version of dune with a fix for this bug, only master is fixed. I was just wondering if you need the fix in 1.11.4 or 2.0

[Et7f3]

We have found a workaround be listing only source folder. So the fix can be in 2.0 except if what we made is a dirty hack. Thanks.

[jordwalke]

Would it be possible to get a release for this fix? The reason is that (I believe) this will cause even dependent packages to fail when there's nothing you can do in your own project to fix it. The danger is that people will start putting constraints like dune < 1.11.2 in their packages, which can have a fragmenting/hard-to-correct effect on the ecosystem. Pre-2.0 would be great as well for similar reasons.

[aantron]

I will add that I have also observed and worked around this, just under three weeks ago, during development of my own package. The issue could have easily crept into a release, if I had been developing with an earlier version of Dune. As I recall, it was when I used (ignored_subdirs _esy).

[jeremiedimino]

Sure, I started preparing a 1.11.4 release

[jeremiedimino]

I didn't manage to do the release today. I'll do it next week, or someone else should feel free to do one before. Let's include #2693 as well in 1.11.4.

[rgrinberg]

@diml I can take over 1.11.4 unless you're on this already.

[jeremiedimino]

@rgrinberg that'd be welcome, thanks! :)