Assemblyscript: [Tests] Rewrite test runner in parallel & async way

[MaxGraey]

Currently most test routines run synchronously. But number of tests and files continiusly growing and this could be a problem in future.

Proposed steps:
1) glob.sync -> await glob after util.promisify or use tiny-glob / globby
2) Switch from WebAssembly.Instance +WebAssembly.Module to WebAssembly.instantiateStreaming for instantiation.
3) fs.readFileSync -> fs.readFile, fs.writeFileSync -> fs.writeFile
4) spawn/fork process for each separate test (?)

[dcodeIO]

Most time is spent in compiling and executing the modules, which is always synchronous, so using asynchronous APIs alone won't help much. What would work though is to spawn one process per CPU core and split up tests among these.

For instance, spawn X child processes and make them request tests from the master when idle, run the test until idle again and request a new test, until all tests are done. Individual tests can still be synchronous then.

[MaxGraey]

Or use "worker_threads" which should be better I guess

[MaxGraey]

Or use isolation process for each test like in ava

[JamesLMilner]

So I had a little go at writing some of the tests in paralell using cluster. I started with the compiler tests as they were pretty accessible.

Findings were that it saves off ~5 seconds when run in parallel across 12 cores, which was a bit disapointing (around 27 seconds instead of 32).

Before:

real    0m31.898s
user    0m50.598s
sys     0m0.585s

After:

real    0m27.547s
user    2m41.578s
sys     0m2.291s

I doubt doing this for the parser tests would be any quicker, and actually be somewhat slower because of the overhead of setting up processes and interprocess communication. I didn't have enough time to figure out the entry point and workings of the packages tests, but if you think it might help there I can explore further.

One thing I did notice is that their is no abstractions for writing tests. It seems it might be useful to have some shared code that makes adding / removing tests simpler, which could be implemented in a way that allowed for an optional process oriented approach.

Update:

Bringing the number of workers down to 6 from 12 brought running town down to ~22 seconds :

real    0m21.961s
user    1m39.059s
sys     0m1.408s

10 seconds off the initial running time :tada:

[JamesLMilner]

I've taken this a step further and rewritten the three major test suites (parser, compiler and packages) to allow them to be run in parallel.

Before:

real    0m48.464s
user    1m18.247s
sys     0m1.415s

After:

real    0m32.126s
user    2m57.957s
sys     0m3.364s

I'll see if there's any more savings to be made here or better approaches to improve this further.

[MaxGraey]

Hmm, I see after slower then before. Is it correct?

[JamesLMilner]

Not exactly (you can see the meaning of the times here: https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1). The real amount of time is about 16 seconds faster after.

[MaxGraey]

I see. Great!

[dcodeIO]

Interesting that it becomes just a bit faster, though, even though it does significantly more work overall. May this have something to do with synchronous I/O blocking the entire cluster? Or does it do another spawn/fork per test it didn't do before or something like that?

[dcodeIO]

Closing this issue as part of 2020 vacuum because it seems to be sufficiently covered by the --parallel option meanwhile added to the compiler test suite, distributing tests over multiple workers.