Elixir: ExUnit output incomplete file paths when ran from umbrella root.

[hickscorp]

I was able to reproduce the same issue with Erlang 19.2 up to 19.3 with Elixir 1.4.2 up to 1.4.4.

If a test fails while running ExUnit at the root of an umbrella app, the filename and stacktrace filenames are incorrect, and lack the app/app_name prefix.

I'm writting regexps to allow Sublime to play nice with ExUnit and open the files where the test failed + highlight the line. This ExUnit behavior makes it impossible.
For instance, a test failing in apps/bar_app/test/bar_app_test.exs is showing a file of test/bar_app_test.exs.

I have assembled a how-to reproduce the issue here.

[josevalim]

Thank you @hickscorp. I believe this has been reported in the past and marked as "won't fix". That's because it is impossible to run an individual test from the umbrella root, since the umbrella project works as a meta application that runs a common on every child. So even if we showed the proper path, you would be unable to run a particular path from the umbrella and you would need to CD into the proper application. And if you change directory to a given application then the paths we emit today are already in the proper format.

[hickscorp]

@josevalim Thank you for investigating,

I understand, it's too bad tho, because simple integration in IDE is rendered impossible without a more complex system (Eg for Sublime, writting a plugin because regexp wouldn't work).

Appologies for reporting it again, I have asked in Slack and havn't had a conclusive answer so I thought it was not known.

[josevalim]

If you can configure sublime to go into each directory in apps and run the
command, then the behavior is maintained. I am unsure how easy or hard this

is though.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[cdegroot]

@josevalim I'm always intrigued when people use the term "impossible" in software, and usually I take that as a challenge ;-). Would you care to elaborate what stands squarely in the way of this functionality? If someone proved it possible, would you consider accepting the patch?

[josevalim]

The feature he asked is impossible given the current design. If you change the design, that would make it possible, but that would come with large code changes and breaking changes. All projects assume that tests run in the current working directory and to fix this issue, we would need to run tests from the umbrella root, which would cause test suites to break.

[cdegroot]

Thanks for the scope indicator. I'll give it a shot regardless ;-). Not in the least because this is encouraging:

cees@OldMac:~/mine/palapa$ mix test_all
.........................................................

  1) test Current term returns 0 on an empty database (Erix.LevelDBTest)
     apps/erix/test/leveldb_test.exs:29
     Assertion with == failed
     code:  42 == PersistentState.current_term(state)
     left:  42
     right: 0
     stacktrace:
       apps/erix/test/leveldb_test.exs:32: (test)

................................................................................

Finished in 3.0 seconds
138 tests, 1 failure

Randomized with seed 528639

(using this project file and artificially breaking a test) - that _is_ what we're talking about, not?

[wojtekmach]

I think a possible solution is to write a custom "umbrella-aware" ExUnit formatter that rewrites the paths and stacktraces accordingly; I took a stab at it one day but gave up :) An advantage of trying this approach is that it should fit into the current design and as a matter of fact would not require any changes to Elixir itself at all.

Some pointers:

• https://github.com/elixir-lang/elixir/blob/v1.4.4/lib/ex_unit/lib/ex_unit/cli_formatter.ex#L70
• https://github.com/elixir-lang/elixir/blob/v1.4.4/lib/ex_unit/lib/ex_unit/formatter.ex#L113

[josevalim]

The problem though is that you still cannot run "mix test
apps/foo/test/foo_bar.exs" as that will go into each application, try to
find a file with that name, which will fail. this would still require a

large change on how Mix test works.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[wojtekmach]

Right, I only had test output in mind. Fwiw for anyone interested, for running tests I have a small extension [1] that basically turns mix test apps/foo/test/bar_test.exs into (cd apps/foo && mix test test/bar_test.exs) but I can see that it's annoying that every editor would have to potentially reinvent something similar.

[1] https://github.com/wojtekmach/dotfiles/blob/eb013eb14f186abd6023dc3f05711987d26d9097/vim/.vimrc#L146:L156

[hickscorp]

@josevalim I'm reading the explaination:

The feature he asked is impossible given the current design. If you change the design, that would make it possible, but that would come with large code changes and breaking changes. All projects assume that tests run in the current working directory and to fix this issue, we would need to run tests from the umbrella root, which would cause test suites to break.

How about adding a "cosmetic parameter" to ExUnit (Eg --prefix_umbrella_apps_filenames) that would simply add app/#{app_name}/ in the output?

So when running mix test inside of an umbrella app, the task which (according to what I understand is happening) in turn calls mix test in the subdirectories would pass the cosmetic argument?

[josevalim]

@hickscorp please see my replies above. The problem is not only showing the path but also computing the proper path to run when you do mix test apps/foo/test/bar_test in your umbrella.

[cdegroot]

we have :apps_path to indicate an umbrella; I'm sure we could invent some heuristics around that per the above? That's the route I'd want to pursue.