Chapel: Develop Chapel Unit and Integration Testing Harness

[buddha314]

According to extremely reliable sources, the start_test command is designed to test the Chapel language itself, not applications built upon Chapel. Because of this there is no clear way to test an application. For instance, start_test does not allow the user to specify the location of cblas.h or libzermoq.

A great example of usability is Python's Unittest framework.

This should probably come with a Mason test command as well that looks in test/. And documentation. Nah, I'm just kidding about the documentation...

[bradcray]

I'm not sure I'd say start_test is designed for the language rather than apps; but I would say that it was designed for developers rather than end-users. I'm not clear in what way start_test doesn't permit users to specify the locations of files. But I don't disagree that a more end-user-facing official way to write tests would be attractive.

[buddha314]

Suppose I have dir structure

src/
  myproj.chpl
test/
  mytest.chpl

and myproj.chpl relies on cblas.h for instance. When I compile myproj.chpl I have to do -L/path/to/blas It does not appear I can specify the path in code. Typically I call this out in the Makefile. But when running start_test it won't read the Makefile and my attempts to put the path in mytest.compopts have failed. So it's unclear to me how to set these.

[bradcray]

I think a non-executable myproj.compopts containing the line:

-L/path/to/blas

ought to do it.

[ben-albrecht]

My Reflections-on-methods branch adds the ability to iterate over and execute methods on a class or record (with some caveats), so that we can eventually do something similar to Python's unittest:

use UnitTest;

class TestInts: TestBase {
  proc testAdd() {
    this.assertEqual(1+1, 2);
  }

  proc testSquare() {
    this.assertEqual(10*10, squareHelper(10));
  }

  proc squareHelper(x) {
    return x**2;
  }
}

// Could this be encapsulated into a UnitTest.runTests() procedure?
var testInts = new TestInts();
testInts.run(); // Calls every method starting with "test"

I also like the simplicity in D's unittest feature. We could alternatively pursue something like that at the language level.

Whichever design we pursue, it should definitely be integrated with mason through mason test.

[buddha314]

Is there a way to do variable substitution? The Makefile for one of my projects starts with

CC=chpl  // not needed for start_test
ZMQ_BASE=/usr/local/Cellar/zeromq/4.2.2
NUMSUCH_HOME=/Users/buddha/github/buddha314/numsuch
BLAS_HOME= /usr/local/Cellar/openblas/0.2.20
INCLUDES=-I${ZMQ_BASE}/include -I$(BLAS_HOME)/include
LIBS=-L${ZMQ_BASE}/lib -lzmq -L${BLAS_HOME}/lib -lblas
MODULES=-M../src -M${NUMSUCH_HOME}/src
FLAGS=--fast  // not needed for start_test

and that's a lot to put on a single line.

[ronawho]

You could do that with an executable .compopts file. Just give the .compopts file execute permissions, and you can run any arbitrary python/bash/whatever script and just have it print your compopts to stdout. Something like:

#!/usr/bin/env bash

ZMQ_BASE="/usr/local/Cellar/zeromq/4.2.2"
NUMSUCH_HOME="/Users/buddha/github/buddha314/numsuch"
BLAS_HOME="/usr/local/Cellar/openblas/0.2.20"
INCLUDES="-I${ZMQ_BASE}/include -I${BLAS_HOME}/include"
LIBS="-L${ZMQ_BASE}/lib -lzmq -L${BLAS_HOME}/lib -lblas"
MODULES="-M../src -M${NUMSUCH_HOME}/src"

echo "${INCLUDES} ${LIBS} ${MODULES}"

edit: By the way I _think_ you could do something like brew --prefix openblas or brew --prefix zmq if you don't want to hardcode brew paths or library packages numbers

[buddha314]

I'll say it for you: Users suck. I'm trying out start_test in this repo and I'm not sure how to get Testy to run and take the message. My next step will be to make testy run with ZeroMQ as a dependency.

[ronawho]

msg is a config const not a param so I don't think you can specify it at compile time. If you move the --msg='I Love You' into a testTesty.execopts it should work.

[ben-albrecht]

msg is a config const not a param so I don't think you specify it at compile time.

but if you wanted to set it at compile-time (which would just set the new default value of the config const), you could use do it like this: -s msg=\"Do you really mean it?\"

[bradcray]

@ben-albrecht beat me to it, but since I've already typed it: You actually can also set the defaults for config consts and vars at compile-time (though execution time is more common), though due to compiler flag parsing challenges, you have to use the -s flag to set them (which can also be used on executables). Whatever follows the variable name will effectively be placed into the source of the program in place of the initializer expression that's in the code. If it's a string, you may need to do some escaping of quote characters to get them past your shell. For me, something like `-smsg=\"I Love You\" works from the command-line. I haven't tried putting it into an executable .compopts file to see how the handling of quotes might differ or not.

[buddha314]

Well, I'm getting closer. This thinking is new to to me. I've checked in a working version and next I want to be able to set dependencies for both the test class (which may need to know about BLAS) and the tested class. Have to run, but please keep those cards and letters coming!

[buddha314]

How did writeln + .good become the standard? Why not use assert()? Just curious.

[bradcray]

Personal style—programs that just assert are boring to run. And we needed a testing system that could ensure that features like writeln() work as expected, and which work with general benchmarks and applications whose output needs to be verified. But nothing prevents one from writing tests with assert()s (they'd just need empty .goods if they had no expected output). You can find tests that do this if you poke around the test system (I don't write that style so don't know where to look... :) But git grep would).

[buddha314]

I was convinced at this line:

programs that just assert are boring to run.

[bradcray]

I have to admit that when programming with asserts, I always have to remember things like "Are they removed if optimizations are turned on? Will my test still work then?" So I tend to avoid them for things that I want to make sure always behave in a certain way. But I'm old and superstitious that way.

[buddha314]

I can relate, the only debugging tool I really know how to use is printf

[buddha314]

@ben-albrecht so we need the reflections package to know if the proc begins with test, right?

[buddha314]

Is it worth the time to support multi-line command in COMPOPTS? Things like

-M../src \\
-L/usr/local/Cellar/openblas/0.2.20/lib -lblas

[bradcray]

Is it worth the time to support multi-line command in COMPOPTS?

I think that'd be a reasonable feature request

[buddha314]

Some routines don't return data in a predictable order. For instance in Chingon testing we have

var vn: [1..0] string;
vn.push_back("star lord");
vn.push_back("gamora");
vn.push_back("groot");
vn.push_back("drax");
vn.push_back("rocket");
vn.push_back("mantis");
vn.push_back("yondu");
vn.push_back("nebula");

Which gets inserted into an object, but read back out as

for v in g3.vnames {
  writeln("g3 vids[", v, "]: ", g3.vids[v]);
}

g3 vids[yondu]: 7
g3 vids[star lord]: 1
g3 vids[rocket]: 5
g3 vids[drax]: 4
g3 vids[nebula]: 8
g3 vids[groot]: 3
g3 vids[gamora]: 2
g3 vids[mantis]: 6

because vids is indexed by a string domain. I don't see a way to use the .GOOD format to handle testing this.

[ronawho]

For output that has a unpredictable order, we typically just use a prediff to sort the output. Something like test/release/examples/PREDIFF

[buddha314]

Honestly, that seems a bit Draconic.

[bradcray]

If you want to propose a change in the testing system, feel free to make a counterproposal. (Turing-complete .good files?)

Another approach is to make your test deterministic -- e.g., using a sort or the "sorterator" to iterate over the strings in a deterministic order:

var names: domain(string);

names += "Brad";
names += "Bruce";
names += "Brian";

for n in names.sorted() do
  writeln(n);

[buddha314]

Oh, I want to make a change all right... TO YOUR FACE!!!

Really, I'm not qualified to tell you how to build this, I'm just here to complain in a marginally constructive way. I ended up using a sorted loop.

[bradcray]

I'm not asking you to tell us how to build it, I'm asking you to tell you what you're actually asking for. So far I've got "a magic test system that automatically knows whether what a program generates is correct or not." Help fill in the gaps for me.

[buddha314]

Well, that seems clear to me! Users, they suck. (I know, I have a case of them myself.)

So, the things that I'm used to are things like Java, Spring and Python unit and integration testing frameworks. I'm not claiming they are superior, but I'm used to the thinking. When I start using the Chapel test framework, there are few things that confuse and / or irritate me.

• The .good, .execopts thinking is pretty cool, but it creates a lot of files and it makes it harder to look through the directory. Also, it is very orthogonal to the JUnit workflow.
• Wrapping your tests in a single test class gives you a nice one-liner at the end, saying "5 tests passed and 6 tests failed".
• Python allows you to send a specific error when a test fails. Like "test to see if Brad's attitude is set to BAD... passed". "test to see if Brad's attitude is set to amicable... failed"
• Right now Chapel does not allow initializers to throw errors. I don't know if it needs to, but I do want to test class validation. If int1 must be <= int2, I want to send an error on new IntMeBaby(5,3)
• Testing dictionaries are set correctly means doing a lot of assert citizens[fairestInTheLand] = 'Brian' tests. That is hard to map onto .good files because looping over a string domain will return in non-deterministic order. Like above. You have to pull tricks like sorted.

On to the world of integration testing, like in Spring apps. If I need my process to pull from a database, I'll need a mock database to test against. In reality, integration tests are always under-developed so maybe Chapel doesn't need them for a while. However, I'm looking into have a REST client built and that will require minimal integration testing. The one I know best is Mockito.

We talked a bit multi-line compopts files for more complicated builds. And what's with that last comment? HILARIOUS!

I _think_ I am answering your question. Let me know!

[buddha314]

I also mentioned we are using Charcoal until this gets closed. The API is like Python. Also referenced as a mason target: https://github.com/chapel-lang/chapel/issues/8714

[russel]

Just in case it matters, most of the cool Python folk use pytest not unittest. Except for unittest.mock which is cool and nothing to do with unittest. https://docs.pytest.org/en/latest/ Python is though a dynamic language so has great runtime reflection.

The D test framework unit-threaded is also cool. https://github.com/atilaneves/unit-threaded. It builds on the built-in unittest infrastructure. It uses a program to build the test suite as there is not sufficient runtime reflection to do that. Also currently it requires "compile all at once" to work since it requires the compiler ADT.

Then there is Groovy's Spock Framework: http://spockframework.org/, but Groovy is dynamic so perhaps a bad example.

It seems Charcoal requires an explicit run method with explicit registration of tests. This is so 1990s and a huge turn off.

Trying to emulate the sUnit architecture (unit testing in Smalltalk) in a statically compiled language is arguably a bad idea. JUnit got away with it because Java has some run time reflection, but it wasn't till JUnit5 that they actually got something right. Python unittest got away with copying sUnit because Python is a dynamic language. However pytest is more Pythonic. C++Unit showed how not to copy the sUnit architecture in a statically compiled language as did Aeryn and many other frameworks. For an idiomatic C++ test framework look at Catch2 https://github.com/catchorg/Catch2, I ignore Boost Test and Google Test.

I really would suggest D with it's built in unittest system with unit-threaded on top is a nice paradigm for a module-based statically compiled language.

To close, separating unit tests and integration tests is increasingly not seen as a good thing. TDD (using sUnit) was initially about testing behaviours of the system, not features of the code. The separation of unit and integration testing is an artificial split probably relating to roles and responsibilities in corporate culture of the 2000s, i.e. rigid departmental structures. With the rise of DevOps an more integrated view is in the ascendant. Back to testing as black box feature checking. Obviously this is about application development, libraries and compilers need a somewhat different approach.

[russel]

On a slightly different topic: example-based testing vs. property-based testing. Testing used to be about examples. This then became big tables of examples. However Haskell brought us QuickCheck and the world of testing changed. No longer are examples the be all and end all, now it is about testing relationships and randomly generated values from the domain. In the Python world Hypothesis is the market leader. It is also moving into the Java and Ruby worlds. C++, Rust, etc. all now have variants on the QuickCheck version of testing. Which is very cool.

[ben-albrecht]

@russel - As always, thanks for your comments.

Would you mind migrating these comments over to the Chapel Test Framework (https://github.com/chapel-lang/chapel/issues/9730) issue where we are discussing design in more depth?

Relatedly, I'm going to close this issue as it is now redundant with that one. Someone speak up if they disagree.