Chapel: Mason test UnitTest integration

[ben-albrecht]

We plan on integrating the unit test launcher into mason test. There are some details to iron out in doing this:

1. Should mason test be be usable outside of a mason package?
2. Should the existing mason test behavior be preserved for backwards compatibility?

I would say yes to (1), allowing something like mason test MyProgram.chpl. It would be nice if mason test had similar behavior regardless of whether in a mason package or not, rather than having 2 completely different code paths when in a mason package vs. not.

For (2), I would say maybe depending on how challenging it is to maintain. With mason being pre-1.0, we are able to change the interface. However, having a fallback behavior of exit-codes if no unit tests are found sounds appealing.

[Spartee]

I think we should support mason test outside of a mason package. A user could be required to list out the file names such as mason test test.chpl or mason test testdir/ which would ensure that mason could find the test files.

I think it would be advantageous to take this opportunity to refactor MasonTest.chpl. As @ben-albrecht points out Mason hasn't reached a major version yet. Mason(including mason test for within a project) and a separate binary for mason test outside of a project could be compiled whenever Mason is built and share the same path. The files could use the same refactored MasonTest.chpl module as a base but be compiled as two different executable.

[krishnadey30]

I don't know much about the (2) but I think (1) should be given. The main of a UnitTest Framework is to allow users to test their code whether it is a mason package or a normal code.

[ben-albrecht]

Given the feedback, I'd propose the following:

mason test is both usable inside and outside mason packages.

mason test outside of a mason package acts as a test runner (similar to python -m unittest) and requires positional arguments of Chapel files or a path to search for Chapel files.

mason test inside a mason package will run on tests within test/ directory (similar to cargo test). The test runner will not require/assume UnitTest usage, i.e. if the test does not emit the expected output that would be emitted if it were using UnitTest module, then it will default to relying on the exit code of the program, maintaining the previous behavior.

[ben-albrecht]

Closing now that https://github.com/chapel-lang/chapel/pull/13866 is merged