Elixir: Mix Test Runner Cannot Run Specific Test In Umbrella Application

[ybur-yug]

Environment

• Elixir & Erlang versions (elixir -v): 1.4.0 (Though I've also verified it exists in 1.3.4)
• Operating system: Mac OSX

Update

It seems that just running mix test and the path that is in the app-folder of apps works wonderfully as @sabondano showed. But, this still seems off as from the directory that it is run its a path that doesnt truly exist to the current context.

I don't know if this is the intended purpose. If so feel free to close.

Current Behaviour

To replicate:

mkdir scratch
cd scratch
mix new replicate_error --umbrella
cd replicate_error/apps
mix new foobar
cd ..

Now, if we try to run specifically the one test that is there (beginning on line 5 of apps/foobar/test/foobar_test.exs), we get an error from mix:

➜  replicate_issue mix test apps/foobar/test/foobar_test.exs:5
==> foobar
Test patterns did not match any file: apps/foobar/test/foobar_test.exs:5
➜  replicate_issue mix test --only line:5 apps/foobar/test/foobar_test.exs
==> foobar
Test patterns did not match any file: apps/foobar/test/foobar_test.exs

However, if we just run them all:

➜  replicate_issue mix test
==> foobar
..

Finished in 0.03 seconds
2 tests, 0 failures

Randomized with seed 511803

If we cd in to the directory, this works:

cd apps/foobar
mix test test/foobar_test.exs:5
Including tags: [line: "5"]
Excluding tags: [:test]

.

Finished in 0.03 seconds
2 tests, 0 failures, 1 skipped

Randomized with seed 616907

This is reasonable, but in certain projects (almost all of my employer's and my own applications have been umbrellas for a while), specifically things built using the new Phoenix umbrella architecture that will be introduced in 1.3, cd'ing in and then doing this gives other errors.

Discussing that seems beyond the scope of the basis of this issue, though.

Expected behavior

This should allow you to run the single test, per the documentation listed here.

Source code is here it seems.

Example Repository

You can find an example here.
Simple clone, then (cd'd to the rood directory), run:

mix do deps.get, compile && mix test apps/foobar/test/foobar_test.exs:5

But if we remove the apps prefix, it works even though we are giving a directory that doesnt exist to our current relative position.

[sabondano]

Try mix test test/foobar_test.exs:5 from the umbrella's root.

[ybur-yug]

If you run the command from the Example Repository section it confirms that does not run @sabondano. Gives this output (same as shown prior) after a fresh clone + cd to root of project

➜  replicate_error mix do deps.get, compile && mix test apps/foobar/test/foobar_test.exs:5
==> foobar
Compiling 1 file (.ex)
Generated foobar app
==> foobar
Compiling 1 file (.ex)
Generated foobar app
==> foobar
Test patterns did not match any file: apps/foobar/test/foobar_test.exs:5

I have tried this with elixir 1.3.4 and 1.4.0 both, but not tested back to any versions prior to those.

I'm going to update the original post to be more clear now, thank you.

[sabondano]

Try mix test test/foobar_test.exs:5 not mix test apps/foobar/test/foobar_test.exs:5. From the umbrella's root directory.

[ybur-yug]

That directory does not exist from the root of the project. Its an umbrella application. I already know I can run the tests from inside, but the expected behaviour per the documentation would be from the root level running it, not having to cd into apps/foobar. And in other cases (such as a phoenix app) it can sometimes not work even then if you've done certain things (but thats outside the scope of this issue at the moment, as the example repo clearly shows the basis of the problem and I've shown where its explained in the docs as how it should behave/examined the source code of that exact function).

[sabondano]

Right, I guess I don't understand the issue. Just trying to help. My recommendation works without having to cd into apps/foobar by the way. Maybe that's obvious and I missed the point but just incase. Good luck! :)

[ybur-yug]

@sabondano its much appreciated and I just realized that I wasn't actually trying to run that because its technically pointing at a directory that doesnt exist (so I didnt paste it in my own foolishness to really try) BUT IT WORKS!!

That is not the expected behaviour still it seems I dont think, I'm going to update the original to expand and show this behaviour and see if it is in fact whats inteded. Theoretically generic modules might end up with the same name, and this could error out.

It seems as if there isnt really a way that I've found to break this in the tests I've done, but it doesnt seem like what would be the expected behaviour since the path doesnt exist from where you are currently at. Strange to me but could be intentional.

[wojtekmach]

Try mix test test/foobar_test.exs:5 from the umbrella's root.

Please note that the command above works, perhaps counterintuitively as ./test_foobar_test.exs does not exist in the umbrella root, because mix test is a @recursive true task, which means it's not actually running from umbrella root, but under the hood it cds into each app. This is why when you run mix test from umbrella root it runs tests for all your apps.

I wouldn't recommend going with mix test test/foobar_test.exs:5 from umbrella root though, because it would break as soon as there's another app (again, since mix test is recursive it will attempt to run the command in all apps, and will fail if one on the apps does not have the file test/foobar_test.exs).
The closest thing I can think of in Elixir is in v1.4 you can do: mix cmd --app foobar mix test test/foobar_test.exs:5 (you can specify --app multiple times) which would limit the command to run only in the apps you specify.

Btw, in my projects the way I run tests from umbrella root is my editor allows me to run test for the current file, and when it sees the file in umbrella instead of doing mix test apps/foo/test/foo_test.exs it transforms the command to (cd apps/foo && mix test test/foo_test.exs) and it works pretty well. If you're using vim and vim-test it might be helpful. Otherwise, I would consider writing a bash script, e.g. t apps/foo/test/foo_test.exs does (cd apps/foo && ...).

[ybur-yug]

Thanks a ton @wojtekmach! I didnt realize Elixir 1.4 allowed that. It seems theres quite a few ways to do this, which is awesome. I just wonder if it could be made more clear what the proper path is. I may not have read the docs etc clearly enough. I will peek more into them tmrw and see.

Also I'm a vim user so that looks awesome. I try not to customize my editor too much and stay minimal, but considering these days 60-75% of my time in an editor is spent writing Elixir I'd say thats worth toying with.

[josevalim]

Oops, I forgot to clarify that you need to give a path from the child project and not the umbrella. So it seems everything is working as expected.

EDIT: Apparently my initial comment was never sent, glad that everything was solved regardless.

[eidge]

Hey guys, sorry for reopening this.

I'm facing the same issue, in my case the biggest problem is that it breaks my editor test setup as that is based on the current path. There's probably some workaround I could do to fix it, but it just seems like an hack.

Is this something you would be willing to accept as a PR?

[deiwin]

@eidge, if you're able to configure your editor to use absolute paths (a la this), then that would also work.

[eidge]

Yup, that does work @deiwin !

Thanks :)