Chapel: Paratest feature requests

[ronawho]

Misc list of feature requests/improvements for paratest (our parallel testing script)

• [ ] Have paratest.chapcs halt early if you're not running in the test/ directory (i.e., generate 'Error: log directory Logs does not exist' first thing, not later)
• [x] Automatically kill jobs that are still running by the time nightly testing wants to start
• [ ] Automatically forward all chapel and third-party env vars
  
  
  
  • (e.g. CHPL_RT_CALL_STACK_SIZE, GASNET_ROUTE_OUTPUT, QT_AFFINITY, etc)
  
  
• [ ] test futures that have a .skipif file by default (this is what nightly does)
• [ ] Make it easier to test different configurations (gasnet, valgrind, other nightly configs)
• [ ] add capability to more easily cancel a paratest while it's running
• [ ] add a fail-after argument that tells paratest to stop after seeing N failures
• [ ] avoid using the "login" node as part of parallel testing
• [ ] request each node individually, then free (opens more nodes when paratest is waiting to finish)
• [ ] Do some kind of smoke-test before to catch dumb errors like not having built the runtime for the current chplenv
• [ ] Allow paratest to work with a module. Right now it forces the use of whatever compiler is above the test directory.
• [ ] Have paratest show passing futures, not just the futures count
• [ ] Have paratest only run futures with -futuresonly, instead of running everything and futures (which is the same thing as the -futures flag)

[benharsh]

I added a couple of items involving termination of a paratest run.

[ronawho]

Added item for not running on the login node

[bradcray]

I added an item relating to generating the 'Error: log directory Logs does not exist' early and not after a bunch of other high latency things. Currently, a bunch of slurm negotation and generation of spec tests happens first which gives the impression things are plugging along fine. You stand up, go to your meeting, come back to leaf through the results and find... it didn't do anything because you were in $CHPL_HOME not $CHPL_HOME/test. :'(

[ronawho]

And I think generally we want paratest to be able to work from non test/ dirs anyways

[bradcray]

That'd be OK too, though I don't mind needing to run it from test/ If it could be run from arbitrary directories, it gets into questions of "Does it switch to $CHPL_HOME/test and run itself from there?" vs. "Does it run itself from wherever it is?" (i.e., could I be set up to use one Chapel installation but run it on a second clone's test hierarchy?).

(I didn't want those philosophical questions to bog down what seemed like a simpler fix that would've saved me 20 minutes).

[ronawho]

Agreed, we can/should implement the simpler fix first

Note that I don't think there's a big philosophical question. IMO paratest should behave exactly like start_test WRT to what's tested.

[dmk42]

I added the bit about paratest under salloc on avalon, and making paratest work on a module.

[dmk42]

Never mind. The salloc problem isn't paratest-specific. It happens with start_test too. I'm removing it from this list.

[ronawho]

@dmk42 had a thought on the way home -- paratest actually uses the compiler relative to paratest.client not the test dir. You might be able to use the paratest that's in the module.

e.g.

module load chapel
cd <my lustre or css test dir>
$CHPL_HOME/util/test/partest.client 

[ronawho]

Adding an item to test futures that have a .skipif by default to avoid noise in nightly configurations

[bradcray]

(Or even all futures by default? The amount of time seems increasingly negligible...)

[ronawho]

I could be talked into it, but all futures by default seems unnecessary to me. It'll add a non-trivial amount of time for gasnet/numa/no-local configurations (and probably at least 3-5 minutes for the standard run), without adding value for 99% (complete guess/fabrication) of paratest runs

[lydia-duncan]

I added a couple of items related to paratest and futures